Open nerosnm opened 11 months ago
If I read the log correctly, the error occurs when building NodeJS. In this case its NodeJS 18, in #261817 it's the exact same error message, but for NodeJS 14.
I don't have a Mac, so I can't test things, but the following patch should fix things:
From d55ffa42ffecdf2250254885e1886e39702e267a Mon Sep 17 00:00:00 2001
From: Florian Warzecha <liketechnik@disroot.org>
Date: Wed, 18 Oct 2023 23:14:06 +0200
Subject: [PATCH] nodejs: fix build on darwin architectures
---
pkgs/development/web/nodejs/nodejs.nix | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/pkgs/development/web/nodejs/nodejs.nix b/pkgs/development/web/nodejs/nodejs.nix
index 8b615a55dd3a..bbb4ca93b601 100644
--- a/pkgs/development/web/nodejs/nodejs.nix
+++ b/pkgs/development/web/nodejs/nodejs.nix
@@ -120,6 +120,11 @@ let
inherit patches;
+ postPatch = ''
+ chmod +x ./gyp-mac-tool
+ patchShebangs ./gyp-mac-tool
+ '';
+
doCheck = lib.versionAtLeast version "16"; # some tests fail on v14
# Some dependencies required for tools/doc/node_modules (and therefore
--
2.42.0
For python3.pkgs.gyp
the afaict same issue has been fixed in https://github.com/NixOS/nixpkgs/pull/248708/files .
Thanks for the really quick reply! I applied an overlay containing the following:
nodejs = prev.nodejs.overrideAttrs (old: {
postPatch = (old.postPatch or "") + ''
chmod +x ./gyp-mac-tool
patchShebangs ./gyp-mac-tool
'';
});
Unfortunately, I receive the following error...
@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking sources
unpacking source archive /nix/store/yjm67hjq80rw9hpxxi7hxjz34shibj64-node-v18.18.0.tar.xz
source root is node-v18.18.0
setting SOURCE_DATE_EPOCH to timestamp 1695014296 of file node-v18.18.0/vcbuild.bat
@nix { "action": "setPhase", "phase": "patchPhase" }
patching sources
applying patch /nix/store/cqn0y2wklslppakf9zwrwm6c139mh9dj-disable-darwin-v8-system-instrumentation.patch
patching file tools/v8_gypfiles/features.gypi
applying patch /nix/store/z3xpyq84lw3ij13g0g3khcy136da1g7g-bypass-darwin-xcrun-node16.patch
patching file deps/npm/node_modules/node-gyp/gyp/pylib/gyp/xcode_emulation.py
applying patch /nix/store/cg9pf70w99aa7nbwm08kwchplb6rqshc-revert-arm64-pointer-auth.patch
patching file configure.py
Hunk #1 succeeded at 1291 (offset 55 lines).
applying patch /nix/store/889zczjmypfbn0lpkd0abcfkzwv8anz7-node-npm-build-npm-package-logic.patch
patching file deps/npm/node_modules/@npmcli/run-script/lib/set-path.js
patching file deps/npm/node_modules/pacote/lib/git.js
patching file deps/npm/node_modules/@npmcli/arborist/lib/arborist/build-ideal-tree.js
Hunk #1 succeeded at 733 (offset -7 lines).
patching file deps/npm/node_modules/@npmcli/arborist/lib/arborist/load-actual.js
Hunk #1 succeeded at 135 with fuzz 1 (offset -12 lines).
applying patch /nix/store/1sdhl2qrslipa15h1ngj55j6nbkw2qfz-trap-handler-backport.patch
patching file deps/v8/src/trap-handler/handler-inside-posix.cc
patching file deps/v8/src/trap-handler/handler-inside-win.cc
patching file deps/v8/src/trap-handler/handler-outside-simulator.cc
chmod: cannot access './gyp-mac-tool': No such file or directory
/nix/store/zv9jfmqy3lwknq5y3mxhp99r9n50ykjl-stdenv-darwin/setup: line 153: pop_var_context: head of shell_variables not a function context
...the key part of which seems to be "chmod: cannot access './gyp-mac-tool': No such file or directory".
I inserted this at the beginning of the addition to postPatch
:
ls -al
find . -name 'gyp-mac-tool'
echo "done finding"
The output was:
total 664
drwxr-xr-x 40 _nixbld1 nixbld 1280 Oct 19 10:54 .
drwx------ 5 _nixbld1 nixbld 160 Oct 19 10:54 ..
-rw-r--r-- 1 _nixbld1 nixbld 3187 Sep 18 05:18 .clang-format
-rw-r--r-- 1 _nixbld1 nixbld 142 Sep 18 05:18 .cpplint
drwxr-xr-x 4 _nixbld1 nixbld 128 Sep 18 05:18 .devcontainer
-rw-r--r-- 1 _nixbld1 nixbld 332 Sep 18 05:18 .nycrc
-rw-r--r-- 1 _nixbld1 nixbld 707 Sep 18 05:18 .yamllint.yaml
-rw-r--r-- 1 _nixbld1 nixbld 268 Sep 18 05:18 BSDmakefile
-rw-r--r-- 1 _nixbld1 nixbld 33591 Sep 18 05:18 BUILDING.md
-rw-r--r-- 1 _nixbld1 nixbld 51676 Sep 18 05:18 CHANGELOG.md
-rw-r--r-- 1 _nixbld1 nixbld 203 Sep 18 05:18 CODE_OF_CONDUCT.md
-rw-r--r-- 1 _nixbld1 nixbld 2565 Sep 18 05:18 CONTRIBUTING.md
-rw-r--r-- 1 _nixbld1 nixbld 7614 Sep 18 05:18 GOVERNANCE.md
-rw-r--r-- 1 _nixbld1 nixbld 115561 Sep 18 05:18 LICENSE
-rw-r--r-- 1 _nixbld1 nixbld 53771 Sep 18 05:18 Makefile
-rw-r--r-- 1 _nixbld1 nixbld 37320 Sep 18 05:18 README.md
-rw-r--r-- 1 _nixbld1 nixbld 9691 Sep 18 05:18 SECURITY.md
-rwxr-xr-x 1 _nixbld1 nixbld 1441 Sep 18 05:18 android-configure
drwxr-xr-x 3 _nixbld1 nixbld 96 Sep 18 05:18 android-patches
-rw-r--r-- 1 _nixbld1 nixbld 2794 Sep 18 05:18 android_configure.py
drwxr-xr-x 56 _nixbld1 nixbld 1792 Sep 18 05:18 benchmark
-rw-r--r-- 1 _nixbld1 nixbld 475 Sep 18 05:18 codecov.yml
-rw-r--r-- 1 _nixbld1 nixbld 22389 Sep 18 05:18 common.gypi
-rwxr-xr-x 1 _nixbld1 nixbld 1425 Sep 18 05:18 configure
-rwxr-xr-x 1 _nixbld1 nixbld 77424 Oct 19 10:54 configure.py
-rwxr-xr-x 1 _nixbld1 nixbld 77471 Sep 18 05:18 configure.py.orig
drwxr-xr-x 25 _nixbld1 nixbld 800 Sep 18 05:18 deps
drwxr-xr-x 14 _nixbld1 nixbld 448 Sep 18 05:18 doc
-rw-r--r-- 1 _nixbld1 nixbld 1360 Sep 18 05:18 glossary.md
drwxr-xr-x 69 _nixbld1 nixbld 2208 Sep 18 05:18 lib
-rw-r--r-- 1 _nixbld1 nixbld 48445 Sep 18 05:18 node.gyp
-rw-r--r-- 1 _nixbld1 nixbld 11844 Sep 18 05:18 node.gypi
-rw-r--r-- 1 _nixbld1 nixbld 13597 Sep 18 05:18 onboarding.md
-rw-r--r-- 1 _nixbld1 nixbld 1067 Sep 18 05:18 pyproject.toml
drwxr-xr-x 242 _nixbld1 nixbld 7744 Sep 18 05:18 src
drwxr-xr-x 32 _nixbld1 nixbld 1024 Sep 18 05:18 test
drwxr-xr-x 63 _nixbld1 nixbld 2016 Sep 18 05:18 tools
-rw-r--r-- 1 _nixbld1 nixbld 3832 Sep 18 05:18 tsconfig.json
drwxr-xr-x 5 _nixbld1 nixbld 160 Sep 18 05:18 typings
-rw-r--r-- 1 _nixbld1 nixbld 34407 Sep 18 05:18 vcbuild.bat
done finding
Thanks for trying that out and searching for the file :)
In the meantime, I found out that the node build process automatically generates that executable during the build process. The following patch changes the base file used for generation instead:
From 13cc1da2e15995f3012831275fff76b34199e26a Mon Sep 17 00:00:00 2001
From: Florian Warzecha <liketechnik@disroot.org>
Date: Wed, 18 Oct 2023 23:14:06 +0200
Subject: [PATCH] nodejs: fix build on darwin architectures
---
pkgs/development/web/nodejs/nodejs.nix | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/pkgs/development/web/nodejs/nodejs.nix b/pkgs/development/web/nodejs/nodejs.nix
index 8b615a55dd3a..5b8ef9b84bd3 100644
--- a/pkgs/development/web/nodejs/nodejs.nix
+++ b/pkgs/development/web/nodejs/nodejs.nix
@@ -120,6 +120,13 @@ let
inherit patches;
+ postPatch = ''
+ chmod +x deps/npm/node_modules/node-gyp/gyp/pylib/gyp/mac_tool.py
+ chmod +x tools/gyp/pylib/gyp/mac_tool.py
+ patchShebangs --build deps/npm/node_modules/node-gyp/gyp/pylib/gyp/mac_tool.py
+ patchShebangs --build tools/gyp/pylib/gyp/mac_tool.py
+ '';
+
doCheck = lib.versionAtLeast version "16"; # some tests fail on v14
# Some dependencies required for tools/doc/node_modules (and therefore
--
2.42.0
Hm, I’ll try building node on aarch64-darwin
with sandbox enabled. I think, ideally, Node.js build system should invoke Python directly on gyp-mac-tool
since I assume it is also copied to the output, and we don’t want to reference native build inputs in the output. Although patchShebang --build
followed by patchShebang --update --host
for mac_tool.py
should be fine too.
I've tried your latest patch, @liketechnik, but it causes a number of test failures (although I'm not confident that these aren't the result of macOS interfering with the tests due to network permissions or some such 🙄).
https://gist.github.com/nerosnm/33a525e9abbc27096090f78f25789a62
(Sorry the logs are so long!)
I'm pretty much clueless about this (I haven't worked on/with NodeJS nor MacOS), so this most likely needs somebody else to take a look at.
But, on a quick glance, https://github.com/nodejs/node/issues/25353 looks like it's the same issue (same Testsuite, same error condition, i.e. expectation = 0
but reality = -16
.
Wrt failing test with the patch above, this is Nix on Darwin with sandbox = yes
, so I assume that there may be some local networking restrictions (i.e. __darwinAllowLocalNetworking
)?
Edit:
The following patch should fix the issue, and it doesn’t look like patched shebang leaks to the derivation output.
From 11ba0ba3672f42fcb5a439fb7bcd954579558bb2 Mon Sep 17 00:00:00 2001
From: Ivan Trubach <mr.trubach@icloud.com>
Date: Thu, 19 Oct 2023 21:20:15 +0300
Subject: [PATCH] nodejs: fix sandboxed build on darwin
---
pkgs/development/web/nodejs/nodejs.nix | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/pkgs/development/web/nodejs/nodejs.nix b/pkgs/development/web/nodejs/nodejs.nix
index 8b615a55dd3a..69f0cb3afc50 100644
--- a/pkgs/development/web/nodejs/nodejs.nix
+++ b/pkgs/development/web/nodejs/nodejs.nix
@@ -120,6 +120,15 @@ let
inherit patches;
+ # Used at build time, but not copied to the output, so we should not get
+ # Python reference in the output.
+ # See also https://github.com/NixOS/nixpkgs/issues/261820
+ postPatch = lib.optionalString stdenv.hostPlatform.isDarwin ''
+ patchShebangs --build tools/gyp/pylib/gyp/mac_tool.py
+ '';
+
+ __darwinAllowLocalNetworking = true; # for tests
+
doCheck = lib.versionAtLeast version "16"; # some tests fail on v14
# Some dependencies required for tools/doc/node_modules (and therefore
I’ve opened PR that should fix this issue, but I won’t have enough time to properly test this until Saturday.
The latest patch does fix the nodejs
test failures (thanks to __darwinAllowLocalNetworking = true;
, I assume) but the iosevka
custom build still fails with the same error:
npm ERR! sh: ./gyp-mac-tool: /usr/bin/env: bad interpreter: Operation not permitted
npm ERR! make: *** [../node-addon-api/nothing.target.mk:160: Release/nothing.a] Error 126
If I read the log correctly, the error occurs when building NodeJS.
I'm really confused about this now. My own original gist doesn't contain the same log I was expecting it to, looking back. You're right that the original log reflects a build failure in the nodejs
package, but what I was seeing (and what I'm back to now) is a failure in iosevka-custom
, not nodejs
. Here's a fresh log.
Edit: also, sorry, I can't find any interface to nix log
that will let/help me remove the ANSI colour codes from that log.
gyp-mac-tool
is generated at https://github.com/nodejs/gyp-next/blob/d1355e3266c9c9a7e63fa56011b8d6d4174744af/pylib/gyp/common.py#L456-L491
It inserts some code after the shebang line, but otherwise copies mac_tool.py
as-is. Note that shebang uses /usr/bin/env
, which does not exist in sandboxed Nix builds. We can either
./gyp-”name”-tool
tool invocations with python3 gyp-“name”-tool
in generated Make and ninja configurations.python
to buildInputs
and (note --host
instead of --build
)
patchShebangs --host deps/npm/node_modules/node-gyp/gyp/pylib/gyp/mac_tool.py
This is a bit unfortunate, since there is no reason Node.js should depend on Python at runtime.
I’m more in favor of 1.ii option, that is, explicitly invoking python3 interpreter in the generated build configuration. The behavior should be identical to #!/usr/bin/env
, except that PATH lookup is done before executing the tool.
@nerosnm, feel free to try https://github.com/NixOS/nixpkgs/pull/262124, it applies https://github.com/tie/gyp-next/commit/2c73b7900a8e9887cadf5cf9127101e896e06257 to gyp, I’ll open upstream PR tomorrow if this works for you. With this PR, I can build Node.js, but iosevka gets stuck at 66% progress with 100% CPU usage building TTF iosevka-lightitalic and I’m not sure if it’s actually doing anything and how long it should take. edit: iosevka builds too.
@tie thanks, I'll try it out. Might take some time, as it appears to require me to build 502 packages from source...
@nerosnm, sorry if this is a bit late, it looks like staging branch bumped LLVM/clang version used on macOS, so Node.js v18 build fails due to -Wenum-constexpr-conversion
errors (https://github.com/NixOS/nixpkgs/pull/253009 implies that v18 patch was dropped because it was cherry-picked upstream, but it doesn’t look like it was released).
If you are not cherry-picking changes from the PR on top of stable Nixpkgs branch fork, you can apply the PR as an overlay, for example:
{
inputs = {
nixpkgs.url = "nixpkgs/nixos-23.05";
systems.url = "systems";
# See https://github.com/NixOS/nixpkgs/pull/262124
nixpkgs-pr-262124.url = "github:NixOS/nixpkgs/refs/pull/262124/head";
};
outputs = inputs: {
legacyPackages = inputs.nixpkgs.lib.genAttrs (import inputs.systems) (system:
import inputs.nixpkgs {
localSystem = { inherit system; };
overlays = [ inputs.self.overlays.backports ];
});
overlays.backports = final: prev:
let
nodejsPath = inputs.nixpkgs-pr-262124 + "/pkgs/development/web/nodejs";
nodeVersion = x: final.callPackage (nodejsPath + "/v${x}.nix");
v14 = nodeVersion "14";
v16 = nodeVersion "16";
v18 = nodeVersion "18";
v20 = nodeVersion "20";
in
{
nodejs_14 = v14 { openssl = final.openssl_1_1; };
nodejs-slim_14 = v14 { enableNpm = false; openssl = final.openssl_1_1; };
nodejs_16 = v16 { };
nodejs-slim_16 = v16 { enableNpm = false; };
nodejs_18 = v18 { };
nodejs-slim_18 = v18 { enableNpm = false; };
nodejs_20 = v20 { };
nodejs-slim_20 = v20 { enableNpm = false; };
};
};
}
Thanks - I was applying it as an overlay, but not as comprehensively/cleanly as your suggestion. I did in fact run into the v18 build failure you mentioned; I'll try again with your suggested overlay (unless you're saying the errors are unresolved?).
Yes, that patch works for me! Thank you so much.
Do you know if there's been any progress on this anywhere?
@nerosnm, unfortunately, upstream PR https://github.com/nodejs/gyp-next/pull/216 is not getting enough attention. I’ve pinged the reviewer, so hopefully we’ll get a decision on this change. Meanwhile, I’ve rebased https://github.com/NixOS/nixpkgs/pull/262124.
Steps To Reproduce
Steps to reproduce the behavior:
iosevka
package at nixpkgs commit 80e825ede84cd9b5b04db94b7173041d22156838:Build log
the error at the end of the full log is:
Additional context
this problem was introduced at commit 80e825ede84cd9b5b04db94b7173041d22156838 (determined by bisecting nixpkgs). i'm on an aarch64-darwin machine.
Notify maintainers
maintainers:
iosevka
: @jfrankenau, @ttuegel, @babariviere, @rileyinman, @AluisioASG, @lunik1nodejs
: @cillianderoiste, @gilligan, @cko, @marsamMetadata
Please run
nix-shell -p nix-info --run "nix-info -m"
and paste the result.