MacOS: Python shebang issue

[bergkvist]

Describe the bug On MacOS, nested shebangs/shebangs pointing at scripts are not allowed. When using pkgs.python39.withPackages(...) and installing packages with pip into a sandbox environment, binary executables get a shebang that makes them unable to execute.

To Reproduce

1. Create shell.nix, and run nix-shell:

   let
   pkgs = import <nixpkgs> {};
   python = pkgs.python39.withPackages(ps: [ ps.pip ]);
   in pkgs.mkShell {
   buildInputs = [ python ];
   shellHook = ''
   export PIP_DISABLE_PIP_VERSION_CHECK=1
   export PIP_PREFIX="$(pwd)/_build/pip"
   export PYTHONPATH="$PIP_PREFIX/lib/python3.9/site-packages:$PYTHONPATH"
   export PATH="$PIP_PREFIX/bin:$PATH"
   '';
   }

2. Install ipython with pip, and try to execute it.

   
   [nix-shell]$ pip install ipython
   [nix-shell]$ ipython

/Users/tobias/nix-python-issue/_build/pip/bin/ipython: line 3: import: command not found /Users/tobias/nix-python-issue/_build/pip/bin/ipython: line 4: import: command not found from: can't read /var/mail/IPython /Users/tobias//nix-python-issue/_build/pip/bin/ipython: line 7: syntax error near unexpected token (' /Users/tobias//nix-python-issue/_build/pip/bin/ipython: line 7: sys.argv[0] = re.sub(r'(-script.pyw|.exe)?$', '', sys.argv[0])'

**Expected behavior**
It should be possible to do `$ pip install ipython`, followed by `$ ipython` - and have ipython actually execute successfully.

### Observations

The shebang in _build/pip/bin/ipython points to a script, which is not allowed on MacOS (only Linux):
```sh
[nix-shell]$ cat $PIP_PREFIX/bin/ipython

#!/nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/bin/python3.9
# -*- coding: utf-8 -*-
import re
import sys
from IPython import start_ipython
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(start_ipython())

If /nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/bin/python3.9 was an executable, everything would be fine. But since this is a script, with its own shebang, this causes issues:

[nix-shell]$ cat /nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/bin/python3.9

#! /nix/store/ra8yvijdfjcs5f66b99gdjn86gparrbz-bash-4.4-p23/bin/bash -e
export NIX_PYTHONPREFIX='/nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env'
export NIX_PYTHONEXECUTABLE='/nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/bin/python3.9'
export NIX_PYTHONPATH='/nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/lib/python3.9/site-packages'
export PYTHONNOUSERSITE='true'
exec "/nix/store/7pjbbmnrch7frgyp7gz19ay0z1173c7y-python3-3.9.2/bin/python3.9"  "$@"

Metadata

• system: "x86_64-darwin"
• host os: Darwin 20.3.0, macOS 10.16
• multi-user?: no
• sandbox: no
• version: nix-env (Nix) 2.3.10
• channels(tobias): "darwin, nixpkgs-21.05pre284563.ab6943a7450"
• nixpkgs: /Users/tobias/.nix-defexpr/channels/nixpkgs

[FRidh]

Duplicate of https://github.com/NixOS/nixpkgs/issues/65351

[bergkvist]

Some more investegation:

[nix-shell]$ which pip

/nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/bin/pip

Looking at our pip executable (which is also a wrapped shell script):

[nix-shell]$ cat /nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/bin/pip

#! /nix/store/ra8yvijdfjcs5f66b99gdjn86gparrbz-bash-4.4-p23/bin/bash -e
export NIX_PYTHONPREFIX='/nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env'
export NIX_PYTHONEXECUTABLE='/nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/bin/python3.9'
export NIX_PYTHONPATH='/nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/lib/python3.9/site-packages'
export PYTHONNOUSERSITE='true'
exec "/nix/store/ccm6jcg1il8wdshiavrbc65p3s6i8rbl-python3.9-pip-21.0.1/bin/pip"  "$@"

---

What happens if we try to modify NIX_PYTHONEXECUTABLE?

# ...
export NIX_PYTHONEXECUTABLE='this-is-a-test'
# ...

... and then reinstall ipython:

[nix-shell]$ rm -rf _build && pip install -q ipython && cat _build/pip/bin/ipython

#!this-is-a-test
# -*- coding: utf-8 -*-
import re
import sys
from IPython import start_ipython
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(start_ipython())

So it seems like NIX_PYTHONEXECUTABLE decides what the shebang will look like when pip generates executable python scripts.

[bergkvist]

I'm guessing a solution (on MacOS) here would be to set NIX_PYTHONEXECUTABLE to something like

# ...
export NIX_PYTHONEXECUTABLE='/nix/store/ra8yvijdfjcs5f66b99gdjn86gparrbz-bash-4.4-p23/bin/bash /nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/bin/python3.9'
# ...

But it seems like NIX_PYTHONEXECUTABLE changes behaviour when containing spaces:

[nix-shell]$ rm -rf _build && pip install -q ipython && cat _build/pip/bin/ipython

#!/bin/sh
'''exec' "/nix/store/ra8yvijdfjcs5f66b99gdjn86gparrbz-bash-4.4-p23/bin/bash /nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/bin/python3.9" "$0" "$@"
' '''
# -*- coding: utf-8 -*-
import re
import sys
from IPython import start_ipython
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(start_ipython())

Which doesn't work

[nix-shell]$ ipython
/Users/tobias/nix-python-issue/_build/pip/bin/ipython: line 2: /nix/store/ra8yvijdfjcs5f66b99gdjn86gparrbz-bash-4.4-p23/bin/bash /nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/bin/python3.9: No such file or directory
/Users/tobias/nix-python-issue/_build/pip/bin/ipython: line 2: exec: /nix/store/ra8yvijdfjcs5f66b99gdjn86gparrbz-bash-4.4-p23/bin/bash /nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/bin/python3.9: cannot execute: No such file or directory

Manually modifying the ipython shebang (_build/pip/bin/ipython) to this makes it work:

#!/nix/store/ra8yvijdfjcs5f66b99gdjn86gparrbz-bash-4.4-p23/bin/bash /nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/bin/python3.9
# -*- coding: utf-8 -*-
import re
import sys
from IPython import start_ipython
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(start_ipython())

[nix-shell]$ ipython
Python 3.9.2 (default, Apr  8 2021, 19:41:15)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.23.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]:

[bergkvist]

Okay, through some experimentation, I discovered a hack for working around the issue with spaces in NIX_PYTHONEXECUTABLE.

# ...
export NIX_PYTHONEXECUTABLE='/nix/store/ra8yvijdfjcs5f66b99gdjn86gparrbz-bash-4.4-p23/bin/bash" "/nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/bin/python3.9'
# ...

Notice the double quotes in the middle.

This causes the following shebang expression to be generated instead:

[nix-shell]$ rm -rf _build && pip install -q ipython && cat _build/pip/bin/ipython

#!/bin/sh
'''exec' "/nix/store/ra8yvijdfjcs5f66b99gdjn86gparrbz-bash-4.4-p23/bin/bash" "/nix/store/i46k148mi830riq4wxh49ki8qmq0731k-python3-3.9.2-env/bin/python3.9" "$0" "$@"
' '''
# -*- coding: utf-8 -*-
import re
import sys
from IPython import start_ipython
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(start_ipython())

Which actually seems to work:

[nix-shell]$ ipython
Python 3.9.2 (default, Apr  8 2021, 19:41:15)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.23.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]:

[bergkvist]

Since this also works around all the shebang-limitations (like maximum length, number of arguments, pointing to a script etc), I imagine this should be fairly polymorphic with respect to different unix-platforms.

One would need to check whether the python interpreter is a binary file or a text file before determining what to use for NIX_PYTHONEXECUTABLE though.

The reference to /bin/sh could be a problem for reproducibility since it is outside of /nix/store/... (if the user has linked a non-POSIX compliant shell or something else here).

[bergkvist]

NIX_PYTHONEXECUTABLE was introduced in this PR: https://github.com/NixOS/nixpkgs/pull/65454

It works by setting sys.executable in Python. https://github.com/NixOS/nixpkgs/blob/3280de8b8c4f1f1f93a6fde62ad16b92be1c03f4/pkgs/development/interpreters/python/sitecustomize.py#L31-L35

---

pip uses sys.executable for deciding on what shebang to use when creating _build/pip/bin/ipython.

ScriptMaker._get_shebang() https://github.com/pypa/pip/blob/e6414d6db6db37951988f6f2b11ec530ed0b191d/src/pip/_vendor/distlib/scripts.py#L164

get_executable() https://github.com/pypa/pip/blob/e6414d6db6db37951988f6f2b11ec530ed0b191d/src/pip/_vendor/distlib/util.py#L312

This code puts double quotes around our executable if it contains spaces, and doesn't start with a double quote: enquote_executable(executable) https://github.com/pypa/pip/blob/e6414d6db6db37951988f6f2b11ec530ed0b191d/src/pip/_vendor/distlib/scripts.py#L63

This code creates the exec-multiline shebang if it detects any spaces in the shebang (or we exceed the maximum shebang-length): ScriptMaker._build_shebang(): https://github.com/pypa/pip/blob/e6414d6db6db37951988f6f2b11ec530ed0b191d/src/pip/_vendor/distlib/scripts.py#L147-L155

[bergkvist]

So turns out (from looking at pip source code) that we can put double quotes at the start/end of NIX_PYTHONEXECUTABLE, to make it slightly more explicit:

[nix-shell]$ cat $(which pip)

#! /nix/store/jdi2v7ir1sr6vp7pc5x0nhb6lpcmg6xg-bash-4.4-p23/bin/bash -e
export NIX_PYTHONPREFIX='/nix/store/qpqw3macz4iv66a0rxvlajpnifhxxw4c-python3-3.9.2-env'
export NIX_PYTHONEXECUTABLE='"/nix/store/jdi2v7ir1sr6vp7pc5x0nhb6lpcmg6xg-bash-4.4-p23/bin/bash" "/nix/store/qpqw3macz4iv66a0rxvlajpnifhxxw4c-python3-3.9.2-env/bin/python3.9"'
export NIX_PYTHONPATH='/nix/store/qpqw3macz4iv66a0rxvlajpnifhxxw4c-python3-3.9.2-env/lib/python3.9/site-packages'
export PYTHONNOUSERSITE='true'
exec "/nix/store/vr4yqqmx0s999xspgprnw7r3d1609hac-python3.9-pip-21.0.1/bin/pip"  "$@"

[bergkvist]

Counterexample to where the NIX_PYTHONEXECUTABLE from above doesn't work:

[nix-shell]$ pip install live-server

Collecting live-server
  Using cached live_server-0.9.9-py3-none-any.whl (5.9 kB)
Collecting beautifulsoup4==4.6.3
  Using cached beautifulsoup4-4.6.3-py3-none-any.whl (90 kB)
Collecting Click==7.0
  Using cached Click-7.0-py2.py3-none-any.whl (81 kB)
Collecting tornado==5.1.1
  Using cached tornado-5.1.1.tar.gz (516 kB)
    ERROR: Error [Errno 2] No such file or directory: '"/nix/store/jdi2v7ir1sr6vp7pc5x0nhb6lpcmg6xg-bash-4.4-p23/bin/bash" "/nix/store/qpqw3macz4iv66a0rxvlajpnifhxxw4c-python3-3.9.2-env/bin/python3.9"' while executing command python setup.py egg_info
ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: '"/nix/store/jdi2v7ir1sr6vp7pc5x0nhb6lpcmg6xg-bash-4.4-p23/bin/bash" "/nix/store/qpqw3macz4iv66a0rxvlajpnifhxxw4c-python3-3.9.2-env/bin/python3.9"'

It seems like sys.executable is expected to be a valid path, meaning we can't use the bash-hack. Although we get a working shebang - something else ends up failing as a result.

[stale[bot]]

I marked this as stale due to inactivity. → More info

[NilsIrl]

I'm having the same issue with ruby and the cewl package.

[bergkvist]

@NilsIrl I'm assuming you are using ruby.withPackages(...) to set up ruby if you are getting this issue. Now that makeBinaryWrapper has been merged into nixpkgs, you can try this out to see if it solves the problem:

# shell.nix
{ pkgs ? import <nixpkgs> {} }:
let
  ruby = pkgs.ruby.override { makeWrapper = pkgs.makeBinaryWrapper; };
  rubyPkgs = ruby.withPackages(ps: []);
in
pkgs.mkShell {
  buildInputs = [ rubyPkgs ];
}

If it does, I can make a PR to make this the default wrapper for ruby

[NilsIrl]

That doesn't seem to be the case: https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/security/cewl/default.nix

[bergkvist]

@NilsIrl What nix expression are you using?

[bergkvist]

Oh, looking at pkgs.cewl I see the following:

nix-shell -p cewl
[nix-shell]$ head -n4 $(which cewl)

#!/nix/store/44mm7f7rsp5vc8dflkkn3pfbp8ghrjlf-wrapped-ruby-cewl-ruby-env/bin/ruby
#encoding: UTF-8

# == CeWL: Custom Word List Generator

[nix-shell]$ cat /nix/store/44mm7f7rsp5vc8dflkkn3pfbp8ghrjlf-wrapped-ruby-cewl-ruby-env/bin/ruby

#! /nix/store/a54wrar1jym1d8yvlijq0l2gghmy8szz-bash-5.1-p12/bin/bash -e
export BUNDLE_GEMFILE='/nix/store/s56719qzccbym8mbyfax6rsi2jx26zv3-gemfile-and-lockfile/Gemfile'
unset BUNDLE_PATH
export BUNDLE_FROZEN='1'
export GEM_HOME='/nix/store/yxapcqnbrfhgi7sl69hbqp18j0vs61sm-cewl-ruby-env/lib/ruby/gems/2.7.0'
export GEM_PATH='/nix/store/yxapcqnbrfhgi7sl69hbqp18j0vs61sm-cewl-ruby-env/lib/ruby/gems/2.7.0'
exec "/nix/store/ia70ss13m22znbl8khrf2hq72qmh5drr-ruby-2.7.5/bin/ruby"  "$@"

I'm on Linux right now though, but I can imagine that this wrapper looks the same on macOS as well - so we need to use makeBinaryWrapper for wrapped-ruby-cewl-ruby-env - since nested shebangs are not allowed on macOS.

[bergkvist]

@NilsIrl Kind of a "shotgun"-solution, but can you verify whether this works for you?

# shell.nix
let
  pkgs = import <nixpkgs> {
    overlays = [ (final: prev: { makeWrapper = prev.makeBinaryWrapper; }) ];
  };
in
pkgs.mkShell {
  buildInputs = [ pkgs.cewl ];
}

nix-shell shell.nix
[nix-shell]$ head -n4 $(which cewl)

#!/nix/store/d9shnlmjh51av5hsych4r4fpl0m9ydbf-wrapped-ruby-cewl-ruby-env/bin/ruby
#encoding: UTF-8

# == CeWL: Custom Word List Generator

Now, the shebang will be a binary wrapper instead of a shell script, which should make this work on macOS:

[nix-shell]$ cat /nix/store/d9shnlmjh51av5hsych4r4fpl0m9ydbf-wrapped-ruby-cewl-ruby-env/bin/ruby

...binary data...

# ------------------------------------------------------------------------------------
# The C-code for this binary wrapper has been generated using the following command:

makeCWrapper /nix/store/ia70ss13m22znbl8khrf2hq72qmh5drr-ruby-2.7.5/bin/ruby \
    --set 'BUNDLE_GEMFILE' '/nix/store/s56719qzccbym8mbyfax6rsi2jx26zv3-gemfile-and-lockfile/Gemfile' \
    --unset 'BUNDLE_PATH' \
    --set 'BUNDLE_FROZEN' '1' \
    --set 'GEM_HOME' '/nix/store/fal997gsyzyjrzcpx6cg2qirpyw1n0cn-cewl-ruby-env/lib/ruby/gems/2.7.0' \
    --set 'GEM_PATH' '/nix/store/fal997gsyzyjrzcpx6cg2qirpyw1n0cn-cewl-ruby-env/lib/ruby/gems/2.7.0'

# (Use `nix-shell -p makeBinaryWrapper` to get access to makeCWrapper in your shell)
# ------------------------------------------------------------------------------------

...binary-data...

[NilsIrl]

I can happily say that after waiting a few hours for stuff to build I have finally landed in a shell in which I can run cewl without issue.