Bad interpreter when running scripts from setuptools setup scripts

[hodeinavarro]

Issue

Executing a bin installed from "scripts" tag at setup from setuptools errors

❯ my_package
zsh: /Users/hodeinavarro/Developer/astral.sh/uv-bin-scripts-mre/with-uv/bin/my_package: bad interpreter: /Users/hodeinavarro/Library/Caches/uv/.tmp7Y2T4Y/.venv/bin/python: no such file or directory

MRE

See reproduction and reproduction steps on repo.

Platform

macOS Sonoma 14.4.1 M1 Max zsh & oh-my-zsh & iTerm2

Version

❯ uv --version
uv 0.1.24 (a5cae0292 2024-03-22)

Also confirmed with newest

❯ uv --version
uv 0.1.26 (7b685a815 2024-03-28)

[charliermarsh]

Reproduced, thanks!

[charliermarsh]

(Only with an editable install.)

[charliermarsh]

Huh, I guess this comes from a behavior whereby if the shebang contains python in it, Distutils will replace it with the current Python interpreter: https://docs.python.org/3.11/distutils/setupscript.html#installing-scripts. That doesn't work here, because we build the wheel in a temporary virtual environment.

[charliermarsh]

When we build as a non-editable, the shebang is correctly #!python when we go to install the wheel, and so we replace it with the interpreter path -- all good.

When we build the wheel as editable, by the time we go to replace the shebang, it's already set to #!/private/var/folders/nt/6gf2v7_s3k13zq_t3944rwz40000gn/T/.tmpDZLVal/.tmpebUcQR/.venv/bin/python or whatever the temporary environment path is.

[charliermarsh]

I have no clue why the build system is changing the shebang when building an editable, nor how to stop it from doing so, nor why pip doesn't suffer from the same issue.

[charliermarsh]

@jaraco -- Sorry to bother, but do you know why we would be seeing a shebang rewrite for the attached script when we build via the PEP 660 editable hooks, but not the PEP 517 build hooks (which seem to preserve #!python, allowing us to rewrite it when we install the wheel later)?

[charliermarsh]

My guess is that pip isn't affected by this because it builds in the same environment, but uses --prefix (I think?) to isolate the installed packages.

[jaraco]

Here is the code that Setuptools (and also distutils) uses to rewrite shebangs. It looks like it constructs the executable from sysconfig variables BIN_DIR, VERSION, and EXE. It's conceivable pip alters those configuration values (BIN_DIR in particular) to affect the rewrite. I traced through the PEP 660 code and it appears to be reaching the build_script command through the build, but that doesn't explain why the behavior is different for non-editable installs.

Aha, I think I see why. The PEP 517 build uses bdist_wheel to produce the build, which is implemented in the wheel project and specifically sets the executable to "python" when "building" scripts.

That almost certainly explains the disparity. And it sort-of makes sense in that a pure "wheel" is portable and doesn't know where it's going to be installed but an "editable wheel" is not portable and presumed to be running from the current environment. That's consistent with "pip isn't affected because it builds in the same environment."

I'm not sure the right thing to do here. My instinct is that the most expedient thing to do would be for UV to rewrite the shebangs in these scripts when moving them. The proper solution is probably more involved and would entail updating the spec to make scripts portable and require installers to rewrite them the same way they do regular. It's possible the spec already does that and Setuptools is violating it by not re-using the wheel building logic across PEP 517 and PEP 660 builds.

[charliermarsh]

Thanks!

I suppose another option would be to try and rewrite the shebangs before invoking setuptools... But that would require that we introspect the project contents (e.g., we can't really know where the scripts are stored without making assumptions about the build backend and configuration, I think).

[ripatrick]

That almost certainly explains the disparity. And it sort-of makes sense in that a pure "wheel" is portable and doesn't know where it's going to be installed but an "editable wheel" is not portable and presumed to be running from the current environment. That's consistent with "pip isn't affected because it builds in the same environment."

On the other hand, if an "editable wheel" is presumed to be running in the current environment, why would they rewrite the shebang at all? 🤔 If they were trying to cover that corner case presumably they could leave it at what the package developer has hard coded.

I've been trying to switch our infra to uv and ran into this issue with one of our local packages. If there is a substring that matches python in the shebang, the whole line gets rewritten. In my particular case (macos 14.4.1) it's a reference to a temp cache #!/Users/patrick/Library/Caches/uv/.tmp87qDc8/.venv/bin/python that does not exist after uv pip install -e is finished.

Hopefully you can find a reasonable solution, my vote, not that I get one, is to rewrite it to #!python like the wheel project.

[sbidoul]

Aha, I think I see why. The PEP 517 build uses bdist_wheel to produce the build, which is implemented in the wheel project and specifically sets the executable to "python" when "building" scripts.

@jaraco FYI it seems the problem persists with setuptools 74.1.2 which now does not dependend on wheel, I think.

[jaraco]

Confirmed, now setuptools specifically sets the executable to "python". That could make it easier to possibly converge the behavior, although it may still prove impractical to know what's the best shebang to put there, as it's only doing a build and not an install. Probably the installers need to take responsibility to rewrite the shebangs.

[sbidoul]

@jaraco Now I'm confused. I tried running the build_editable hook manually with setuptools 74.1.2 on setup.py with a scripts key referring to a script with #!/usr/bin/env python3 as shebang. In the resulting wheel there is a *.data/scripts directory where the script has had the shebang replaced with the full path of the python used to run build_editable.

When I run build_wheel, however, the shebang in the wheel is #!python, as expected.

So I tend to think the problem is still present in setuptools for editable wheels?

[jaraco]

Sorry. I didn't mean to confuse. I don't believe anything changed with setuptools, except where the code for setting the executable for non-editable builds is found (formerly wheel, now embedded in setuptools).

For PEP 660 build_editable installs, the behavior remains unchanged, relying on the old distutils behavior to rewrite the shebang.

What I don't yet understand is why pip isn't subject to this issue, or what setuptools should possibly do differently.

[sbidoul]

why pip isn't subject to this issue

I'll try to find out

what setuptools should possibly do differently

My instinct says build_editable should emit the same #!python shebang as build_wheel. From the point of view of installers, a PEP 660 wheel is no different than any other wheel, so the rewrite #!python part of the wheel spec applies.