Port to Python 3

[dotlambda]

Python 2 has been EOL since January 1, 2020 and it is insecure. That's relevant when parsing other people's code. See https://github.com/NixOS/nixpkgs/pull/201859.

[abathur]

• @dotlambda py3 is a no-go because oil is in 27
• @figsoda I'm not quite sure what you mean by opy, but if you mean the python bytecode compiler inside of the oil repo... I don't think this is a good idea. This code is pretty old. It might be under-test and therefore working, but there've been notes in Oil for a while that it was basically a dead-end experiment that will eventually be completely replaced with the mycpp work they've been doing (leaving resholve stuck at that version of Oil). Going down this route entails building the vendored python27 inside the oil repo; I gave this a try on macOS and it fails on:

  /private/tmp/nix-build-python2.7-oildev-unstable-2021-07-14.drv-0/source/Python-2.7.13/configure: line 8397: gcc: command not found
  /private/tmp/nix-build-python2.7-oildev-unstable-2021-07-14.drv-0/source/Python-2.7.13/configure: line 8398: test: <: unary operator expected
  /private/tmp/nix-build-python2.7-oildev-unstable-2021-07-14.drv-0/source/Python-2.7.13/configure: line 8444: /usr/bin/arch: Operation not permitted
  configure: error: Unexpected output of 'arch' on OSX

  This almost certainly converges on re-creating the existing py27 expression.

• @7c6f434c After wasting a couple hours clearing hurdles I ran into an error that reminded me that I have tried this before, but the issue is that Oil leans on multiple C extensions

[7c6f434c]

Ohhh OK I was too naively reading the claim that OPy is a small subset of Python2… Sorry.

[abathur]

I won't pretend to have a really deep understanding of it. IIUC it's a subset in that it builds the py27 and then uses some tree-shaking process to try and track what parts of that python are actually exercised by the oil codebase, and then do (magic? crimes?) to get discard parts that it didn't need for whatever load/memory performance it can claw back by doing so.

[SamLukeYes]

Can xonsh be used to replace oil?

[figsoda]

xonsh's syntax is probably too incompatible with bash to be useful here

[abathur]

My understanding aligns with figsoda's. xonsh is Python + some bits of Shell. As they put it on the landing page:

The language is a superset of Python 3.6+ with additional shell primitives that you are used to from Bash and IPython.

Pretty trivial to find valid bash that it can't parse:

$ xonsh <<EOF
{ declare -gA bashup_ev bashup_ev_r
} 
EOF

  File "<stdin>", line 1
    { declare -gA bashup_ev bashup_ev_r
                 ^
SyntaxError: ('code: bashup_ev',)

Oil's OSH parser is almost-entirely-Bash-compatible.

[SamLukeYes]

Well, I see some discussions about xonsh's bash-compatibility in https://github.com/xonsh/xonsh/issues/4015, and the issue author has mentioned bashlex - Python parser for bash. Would it be helpful here?

[abathur]

resholve uses the OSH parser (and, thus, python2) because it forms the foundation of a working shell that can actually execute nearly all of Bash and is tested against a massive corpus of wild scripts, which is generally what we want for a tool that we can trust to be able to parse, rewrite, and emit the vast majority of Bash/POSIX shell found in the wild without breaking it.

bashlex (or any other bash parser) might work, but it isn't battle-tested like OSH is. It exists to power explainshell.com, which generally just needs to handle a single line of shell at a time. It doesn't have to worry about deploying broken scripts if the parser doesn't recognize a structure and subtly changes it on the way back out. If you look through the issues, it's common to see syntax that doesn't work and the maintainer acknowledges that it doesn't work because he didn't need it for his case.