Closed tie closed 1 week ago
Force-pushed to fix issues reported by ruff
.
Not a fan of this. Why not just create or symlink to /usr/bin/env
?
Why not just create or symlink to
/usr/bin/env
?
In general, only build directory is writable in sandbox, and a limited set of host directories is readable.
While it should be possible to pass /usr/bin/env as an impure derivation dependency, as far as I know, this is usually used only for bootstrapping stdenv, and not in the packages themselves. Note that for Node.js, /usr/bin/env
would also need to be a propagated impure dependency since some packages (e.g. iosevka) depend on vendored node-gyp (in Node.js, gyp exists in deps/npm/node_modules/node-gyp/gyp
and tools/gyp
, the latter is used to build Node.js itself).
Alternatively, Nixpkgs can patch shebang lines in pylib/*_tool.py
to absolute Python executable path (rewrite #!/usr/bin/env python3
→ #!/nix/store/g5cm6iik6p4k39cj9k7a6sg2p09hl7wf-python3-3.10.12/bin/python3
), e.g. https://github.com/NixOS/nixpkgs/issues/261820#issuecomment-1772518861 and https://github.com/NixOS/nixpkgs/pull/248708, but I don’t really like doing that since technically it is a change to the upstream source and not necessary with this PR.
@cclauss, hi, are there any updates on this issue? I’d like to move forward with https://github.com/NixOS/nixpkgs/issues/261820 and https://github.com/NixOS/nixpkgs/pull/262124, and ideally I don’t want to apply this patch without getting it merged upstream (see also my previous comment).
Why not set up /usr/bin/env
in the sandbox macOS environment? The default macOS distribution comes with it.
Sandbox in the Nix term means that only packages built by Nix (which are in /nix/store
) are available in the build environment. Access to /usr/bin/env
is hence prohibited.
If a sandbox is going to provide a macOS-like environment, it should provide /usr/bin/env
as a default macOS does.
It depends on what you think a build environment should look like.
/usr/bin
and pre-installed toolchains in /Library/Developer/CommandLineTools
./usr/bin
or /Library/Developer/CommandLineTools
, but to LLVM Clang, GNU Core Utilities, and other open-source components, all built from source and isolated in /nix/store
.Both models are valid, tested in practice (165,000 PRs in Homebrew Core, 270,000 PRs in Nixpkgs), and can be used to build a multitude of things.
@cclauss @legendecas, thanks for taking a look at this, can you please explain what concerns do you have with this change? I understand that we can disable/relax sandbox to a certain extent, but that is really the last resort unless there is some technical reason why this change is not sound.
Gyp already uses sys.executable
on Windows hosts, so it’s not like this change introduces some new behavior, but rather makes it consistent across platforms. Am I missing something here?
E.g. (%(python)s gyp-win-tool
)
https://github.com/nodejs/gyp-next/blob/4a8e328641dac7d6416d5061345fded81ba6fa1f/pylib/gyp/generator/ninja.py#L2147-L2157
Yes, but I don't think the existing code base handles that well too. Some of the formatting arguments should be using shlex.quote
, but they are not. I think it would be rather confusing to correctly escape in this case but not in others.
I hesitate because the assumption is correct for the vast majority of our Mac users and the others can use the --python
option. NixOS should provide a pervasive solution instead of requiring all tools to make Nix-specific patches.
IMO as long as NixOS community keeps sending patches and having support for their use case is not something that adds maintenance burden for us, there are no downsides for us. AFAICT this patch does not break the common use-case we actively support, it does not make the code less maintainable, so I think we should land it.
I hesitate because the assumption is correct for the vast majority of our Mac users and the others can use the --python option. […]
I don’t think node-gyp currently rewrites the shebang line when generating tools (see permalink below), so it’d still be using #!/usr/bin/env python3
instead of the Python interpreter set via --python
option.
On that note, would it make sense to change source[0]
to #!{sys.executable}
instead of formatting tools command invocations with sys.executable? The only downside I can think of in this case is that POSIX shells have weird shebang handling when an executable is a text file and there is something wrong with the interpreter (see https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_01_01) and I usually prefer to avoid shebangs if possible for this reason (though that shouldn’t be an issue in this case, and we already use shebangs).
That could make the change in the PR smaller and hopefully easier to review/maintain. What do you think?
I.e. https://github.com/nodejs/gyp-next/commit/6d804d929013e22a6adc28646970470b80477021
Using #!/usr/bin/env python3
discourages the creation of a venv
which results in polluting the system Python with dependencies from multiple projects.
We want to encourage the creation of venv
s in which case #!/usr/bin/env python
(no 3
) will work just fine.
I’m confused, I don’t understand how venv
’s are related here.
Just to be clear, I’m referring to the shebang lines of the tool scripts, e.g. https://github.com/nodejs/gyp-next/blob/4a8e328641dac7d6416d5061345fded81ba6fa1f/pylib/gyp/mac_tool.py#L1
You’ve said that we can use --python
flag to change the Python installation being used. I assume you mean node-gyp flag,
When gyp runs, it copies these scripts to the Makefile directory using CopyTool
.
https://github.com/nodejs/gyp-next/blob/4a8e328641dac7d6416d5061345fded81ba6fa1f/pylib/gyp/common.py#L513
The code adds “Generated by” header, but doesn’t change the hash bang from #!/usr/bin/env python3
to sys.executable
. When the build is run, these files are executed and the OS uses the interpreter from hash bang line to execute the script, /usr/bin/env
with python3
argument in this case, so essentially PATH
lookup for python3
.
However, if, for example, PATH is /path/to/python-3-incompatible/bin:/path/to/python-3.12-good/bin
where both directories contain python3 executable, and --python
option is set to the lower /path/to/python-3.12-good/bin/python3
, we’d still use python-3-incompatible
because /usr/bin/env
causes PATH lookup.
So, I’m suggesting that we can do https://github.com/nodejs/gyp-next/commit/6d804d929013e22a6adc28646970470b80477021 which should also fix the Nix sandbox issue on macOS. This is just another approach for the same issue, but may be better than the PR in its current state if we want to continue using shebangs.
Again, I don’t see how venvs change anything. They are not movable/copyable, so we can safely refer to sys.executable
of the virtual environment when generating the project.
@cclauss, gentle ping 👀
This changes fixes build error when
/usr/bin/env
is not available (for example, when building in Nix sandbox). It makes gyp-generated build configurations invoke Python directly on the gyp tools (similar togyp-win-tool
that uses sys.executable on Windows).See also https://github.com/NixOS/nixpkgs/issues/261820 and https://github.com/NixOS/nixpkgs/pull/248708