search

aspect-build / rules_js

High-performance Bazel rules for running Node.js tools and building JavaScript projects
https://docs.aspect.build/rules/aspect_rules_js
Apache License 2.0
301 stars 105 forks source link

node-gyp build error when building locally on osx and windows #217

Open kormide opened 2 years ago

kormide commented 2 years ago 

ERROR: /Users/derek/Code/rules_js/BUILD.bazel:10:22: RunBinary node_modules/.aspect_rules_js/bufferutil@4.0.1/node_modules/bufferutil failed: (Exit 1): lifecycle-hooks.sh failed: error executing command bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/npm/private/lifecycle/lifecycle-hooks.sh bufferutil ../../../bazel-out/darwin_arm64-fastbuild/bin/external/npm__bufferutil__4.0.1/pkg ... (remaining 1 argument skipped)

Use --sandbox_debug to see verbose messages from the sandbox
/private/var/tmp/_bazel_derek/6745edc468b1987dbcff5df31bdec8bd/sandbox/darwin-sandbox/303/execroot/aspect_rules_js/bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/npm/private/lifecycle/lifecycle-hooks.sh.runfiles/aspect_rules_js/npm/private/lifecycle/min/node-gyp-bin/node-gyp: line 5: 1010: command not found
Error: bufferutil@4.0.1 install: `node-gyp-build`
spawn ENOENT
    at ChildProcess.<anonymous> (/private/var/tmp/_bazel_derek/6745edc468b1987dbcff5df31bdec8bd/sandbox/darwin-sandbox/303/execroot/aspect_rules_js/bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/npm/private/lifecycle/lifecycle-hooks.sh.runfiles/aspect_rules_js/npm/private/lifecycle/min/index.min.js:1:79676)
    at ChildProcess.emit (node:events:394:28)
    at maybeClose (node:internal/child_process:1064:16)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:301:5) {
  code: 'ELIFECYCLE',
  errno: 'ENOENT',
  syscall: 'spawn',
  file: 'sh',
  pkgid: 'bufferutil@4.0.1',
  stage: 'install',
  script: 'node-gyp-build',
  pkgname: 'bufferutil'
}

> bufferutil@4.0.1 install /private/var/tmp/_bazel_derek/6745edc468b1987dbcff5df31bdec8bd/sandbox/darwin-sandbox/303/execroot/aspect_rules_js/bazel-out/darwin_arm64-fastbuild/bin/node_modules/.aspect_rules_js/bufferutil@4.0.1/node_modules/bufferutil
> node-gyp-build```
lummax commented 2 years ago

I have the same error trying to install https://www.npmjs.com/package/sharp.

As I understand it, this comes from external/aspect_rules_js/npm/private/lifecycle/min/node-gyp-bin/node-gyp which runs "$npm_config_node_gyp" "$@" in the else-branch and we have (for some reason) "$npm_config_node_gyp" == "1010".

lummax commented 2 years ago

Just for the documentation:

In my case the error was:

prebuild-install warn install EROFS: read-only file system, access '/home/XXX/.npm'
/home/XXX/.cache/bazel/XXX/3feea6cc28786273c03984ee070c8149/sandbox/linux-sandbox/455/execroot/XXX/bazel-out/k8-opt-exec-DF1BA9A6/bin/external/aspect_rules_js/npm/private/lifecycle/lifecycle-hooks.sh.runfiles/aspect_rules_js/npm/private/lifecycle/min/node-gyp-bin/node-gyp: 6: 1010: not found
Error: sharp@0.30.7 install: `(node install/libvips && node install/dll-copy && prebuild-install) || (node install/can-compile && node-gyp rebuild && node install/dll-copy)`
spawn ENOENT
    at ChildProcess.<anonymous> (/home/XXX/.cache/bazel/XXX/3feea6cc28786273c03984ee070c8149/sandbox/linux-sandbox/455/execroot/XXX/bazel-out/k8-opt-exec-DF1BA9A6/bin/external/aspect_rules_js/npm/private/lifecycle/lifecycle-hooks.sh.runfiles/aspect_rules_js/npm/private/lifecycle/min/index.min.js:1:79922)
    at ChildProcess.emit (node:events:527:28)
    at maybeClose (node:internal/child_process:1092:16)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5) {
  code: 'ELIFECYCLE',
  errno: 'ENOENT',
  syscall: 'spawn',
  file: 'sh',
  pkgid: 'sharp@0.30.7',
  stage: 'install',
  script: '(node install/libvips && node install/dll-copy && prebuild-install) || (node install/can-compile && node-gyp rebuild && node install/dll-copy)',
  pkgname: 'sharp'
}

Where the actual error is not about node-gyp rebuild, but the prebuild-install warn. I think https://github.com/prebuild/prebuild-install/blob/8ebc076e6067e26c5420807befd6e60eb116ee72/util.js#L81 is used for the cache directory, but interestingly a --action_env npm_config_cache=FOO does not propagate through.

Fortunately my error went away with a --action_env APPDATA=something

JiaLiPassion commented 2 years ago

@lummax , thank you for the repo, yeah, I also use the APPDATA to make sharp install pass, I will continue to implement this PR.

cgrindel commented 2 years ago

@kormide Has this issue been addressed by #269?

kormide commented 2 years ago

I can't reproduce the error, so going to close.

Silic0nS0ldier commented 2 years ago

Running into this issue on macOS in a brand new repo across 2 packages.

bazel build //web:node_modules --keep_going
INFO: Analyzed target //web:node_modules (24 packages loaded, 272 targets configured).
INFO: Found 1 target...
ERROR: /Users/mele/source/private/rules_js-testing/BUILD.bazel:5:22: Running lifecycle hooks on npm package segfault-handler@1.3.0 failed: (Exit 1): lifecycle-hooks.sh failed: error executing command bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/aspect_rules_js/npm/private/lifecycle/lifecycle-hooks.sh segfault-handler ../../../external/npm__segfault-handler__1.3.0/package ... (remaining 1 argument skipped)
/private/var/tmp/_bazel_mele/a4b4e400f716832758409d5e63552ba4/execroot/__main__/bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/aspect_rules_js/npm/private/lifecycle/lifecycle-hooks.sh.runfiles/aspect_rules_js/npm/private/lifecycle/min/node-gyp-bin/node-gyp: line 5: 1010: command not found
Error: segfault-handler@1.3.0 install: `node-gyp rebuild`
spawn ENOENT
    at ChildProcess.<anonymous> (/private/var/tmp/_bazel_mele/a4b4e400f716832758409d5e63552ba4/execroot/__main__/bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/aspect_rules_js/npm/private/lifecycle/lifecycle-hooks.sh.runfiles/aspect_rules_js/npm/private/lifecycle/min/index.min.js:1:79716)
    at ChildProcess.emit (node:events:527:28)
    at maybeClose (node:internal/child_process:1092:16)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5) {
  code: 'ELIFECYCLE',
  errno: 'ENOENT',
  syscall: 'spawn',
  file: 'sh',
  pkgid: 'segfault-handler@1.3.0',
  stage: 'install',
  script: 'node-gyp rebuild',
  pkgname: 'segfault-handler'
}

> segfault-handler@1.3.0 install /private/var/tmp/_bazel_mele/a4b4e400f716832758409d5e63552ba4/execroot/__main__/bazel-out/darwin_arm64-fastbuild/bin/node_modules/.aspect_rules_js/segfault-handler@1.3.0/node_modules/segfault-handler
> node-gyp rebuild

ERROR: /Users/mele/source/private/rules_js-testing/BUILD.bazel:5:22: Running lifecycle hooks on npm package iltorb@2.4.5 failed: (Exit 1): lifecycle-hooks.sh failed: error executing command bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/aspect_rules_js/npm/private/lifecycle/lifecycle-hooks.sh iltorb ../../../external/npm__iltorb__2.4.5/package ... (remaining 1 argument skipped)
info install installing standalone, skipping download.
/private/var/tmp/_bazel_mele/a4b4e400f716832758409d5e63552ba4/execroot/__main__/bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/aspect_rules_js/npm/private/lifecycle/lifecycle-hooks.sh.runfiles/aspect_rules_js/npm/private/lifecycle/min/node-gyp-bin/node-gyp: line 5: 1010: command not found
Error: iltorb@2.4.5 install: `node ./scripts/install.js || node-gyp rebuild`
spawn ENOENT
    at ChildProcess.<anonymous> (/private/var/tmp/_bazel_mele/a4b4e400f716832758409d5e63552ba4/execroot/__main__/bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/aspect_rules_js/npm/private/lifecycle/lifecycle-hooks.sh.runfiles/aspect_rules_js/npm/private/lifecycle/min/index.min.js:1:79716)
    at ChildProcess.emit (node:events:527:28)
    at maybeClose (node:internal/child_process:1092:16)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5) {
  code: 'ELIFECYCLE',
  errno: 'ENOENT',
  syscall: 'spawn',
  file: 'sh',
  pkgid: 'iltorb@2.4.5',
  stage: 'install',
  script: 'node ./scripts/install.js || node-gyp rebuild',
  pkgname: 'iltorb'
}

> iltorb@2.4.5 install /private/var/tmp/_bazel_mele/a4b4e400f716832758409d5e63552ba4/execroot/__main__/bazel-out/darwin_arm64-fastbuild/bin/node_modules/.aspect_rules_js/iltorb@2.4.5/node_modules/iltorb
> node ./scripts/install.js || node-gyp rebuild

Target //web:node_modules failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 4.352s, Critical Path: 1.17s
INFO: 3 processes: 3 internal.
FAILED: Build did NOT complete successfully

Pulling out a command with --subcommands...

(cd /private/var/tmp/_bazel_mele/a4b4e400f716832758409d5e63552ba4/execroot/__main__ && \
  exec env - \
    BAZEL_BINDIR=bazel-out/darwin_arm64-fastbuild/bin \
    BAZEL_BUILD_FILE_PATH=BUILD.bazel \
    BAZEL_COMPILATION_MODE=fastbuild \
    BAZEL_TARGET=//:.aspect_rules_js/node_modules/segfault-handler@1.3.0/lc \
    BAZEL_TARGET_CPU=darwin_arm64 \
    BAZEL_WORKSPACE=__main__ \
    JS_BINARY__PATCH_NODE_FS=1 \
    JS_BINARY__SILENT_ON_SUCCESS=1 \
  bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/aspect_rules_js/npm/private/lifecycle/lifecycle-hooks.sh segfault-handler ../../../external/npm__segfault-handler__1.3.0/package ../../../bazel-out/darwin_arm64-fastbuild/bin/node_modules/.aspect_rules_js/segfault-handler@1.3.0/node_modules/segfault-handler)

I turned my attention to the relative paths. Neither seem to be pointing to valid locations under my current setup, which may help explain the command not found errors.

Silic0nS0ldier commented 2 years ago

Mislead by my own lack of familiarity. The ../../.. is to overcome the default current working directory.

What I've found;

Given node-gyp is technically an optional component and usually must be installed manually, satisfying this requirement seems out of scope of Rules JS (provided API surfaces exist to allow adding a new input).

Fun fact, debugging this demonstrated for the first time to me the timing differences between console.log and console.error in NodeJS (.log is deferred, .error is immediate). Presumably the task queue was full enough for this to matter.

Silic0nS0ldier commented 2 years ago

🤔 I do have a request out of this however. The current DX is a bit of a mess, all the needed information is technically available but presented in a manner which lends towards misattribution of the error source.

I'll get another ticket opened for this.

flolu commented 1 year ago

@lummax I'm facing this same error when installing testcontainers, because of cpu-features. How can I fix this error?

ERROR: /home/flolu/Desktop/drakery/BUILD.bazel:74:22: Running lifecycle hooks on npm package cpu-features@0.0.4 failed: (Exit 1): lifecycle-hooks.sh failed: error executing command (from target //:.aspect_rules_js/node_modules/cpu-features@0.0.4/lc) bazel-out/k8-opt-exec-2B5CBBC6/bin/external/aspect_rules_js/npm/private/lifecycle/lifecycle-hooks.sh cpu-features ../../../external/npm__cpu-features__0.0.4/package ... (remaining 1 argument skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
/home/flolu/.cache/bazel/_bazel_flolu/7213acaffdabec4878e1d1fe276483d0/sandbox/linux-sandbox/12948/execroot/__main__/bazel-out/k8-fastbuild/bin/external/aspect_rules_js/npm/private/lifecycle/min/node-gyp-bin/node-gyp: line 5: 1010: command not found
Error: cpu-features@0.0.4 install: `node buildcheck.js > buildcheck.gypi && node-gyp rebuild`
spawn ENOENT
    at ChildProcess.<anonymous> (/home/flolu/.cache/bazel/_bazel_flolu/7213acaffdabec4878e1d1fe276483d0/sandbox/linux-sandbox/12948/execroot/__main__/bazel-out/k8-fastbuild/bin/external/aspect_rules_js/npm/private/lifecycle/min/index.min.js:1:79716)
    at ChildProcess.emit (node:events:394:28)
    at maybeClose (node:internal/child_process:1064:16)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:301:5) {
  code: 'ELIFECYCLE',
  errno: 'ENOENT',
  syscall: 'spawn',
  file: 'sh',
  pkgid: 'cpu-features@0.0.4',
  stage: 'install',
  script: 'node buildcheck.js > buildcheck.gypi && node-gyp rebuild',
  pkgname: 'cpu-features'
}

> cpu-features@0.0.4 install /home/flolu/.cache/bazel/_bazel_flolu/7213acaffdabec4878e1d1fe276483d0/sandbox/linux-sandbox/12948/execroot/__main__/bazel-out/k8-fastbuild/bin/node_modules/.aspect_rules_js/cpu-features@0.0.4/node_modules/cpu-features
> node buildcheck.js > buildcheck.gypi && node-gyp rebuild

INFO: Elapsed time: 4.408s, Critical Path: 2.55s
INFO: 18 processes: 18 internal.
FAILED: Build did NOT complete successfully
alexjpwalker commented 1 year ago

Experiencing this same error on MacOS Monterey 12.5.1, M1 chip, Xcode version 14.1, in a project that depends on rules_js.

Running lifecycle hooks on npm package nice-napi@1.0.2 failed: (Exit 1): lifecycle-hooks.sh failed: error executing command bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/aspect_rules_js/npm/private/lifecycle/lifecycle-hooks.sh nice-napi ../../../external/npm__nice-napi__1.0.2/package ... (remaining 1 argument skipped)
/private/var/tmp/_bazel_aw/94e220d1b5fbaf18d2d4dce56099223c/execroot/__main__/bazel-out/darwin_arm64-opt-exec-2B5CBBC6/bin/external/aspect_rules_js/npm/private/lifecycle/lifecycle-hooks.sh.runfiles/aspect_rules_js/npm/private/lifecycle/min/node-gyp-bin/node-gyp: line 5: 1010: command not found
joac commented 1 year ago

I have the same error trying to build node-canvas on an M1 machine:

node-pre-gyp WARN Pre-built binaries not installable for canvas@2.10.2 and node@16.13.0 (node-v93 ABI, unknown) (falling back to source compile with node-gyp)
node-pre-gyp WARN Hit error response status 404 Not Found on https://github.com/Automattic/node-canvas/releases/download/v2.10.2/canvas-v2.10.2-node-v93-darwin-unkn
own-arm64.tar.gz
/private/var/tmp/_bazel_joac/ecc5d860480861a3892a48c75a59acbf/execroot/__main__/bazel-out/darwin_arm64-fastbuild/bin/external/aspect_rules_js/npm/private/lif
ecycle/min/node-gyp-bin/node-gyp: line 5: 1010: command not found
joac commented 1 year ago

@kormide You can find a reproducer at https://github.com/joac/node-gyph-error-reproducer. You will need an arm64 machine to be able ro reproduce the issue. The bazel command that needs to be runned is on the README.

joac commented 1 year ago

To debug this error I took my reproducer-repo and made the setup defined at CONTRIBUTING.md to use this repo as an override.

After the first test, the 1010 become 1211. That value is defined inside npm/private/lifecycle/min/index.min.js. The minified code is hard to grasp, and after checking that the code is minified with ncc decided to try an unminified version, by running npm i in npm/private/lifecycle/ and adding the node_modules folder to the js_binary:


--- a/npm/private/lifecycle/BUILD.bazel
+++ b/npm/private/lifecycle/BUILD.bazel
@@ -6,7 +6,7 @@ js_binary(
     name = "lifecycle-hooks",
     # NB: intentionally include package.json in the runfiles of this binary to prevent
     # https://github.com/aspect-build/rules_js/issues/447
-    data = glob(["min/**"]) + ["package.json"],
+    data = glob(["min/**"]) + glob(["node_modules/**"]) + ["package.json"],
     entry_point = "min/index.min.js",
     include_npm = True,
     visibility = ["//visibility:public"],

--- a/npm/private/lifecycle/min/node-gyp-bin/node-gyp
+++ b/npm/private/lifecycle/min/node-gyp-bin/node-gyp
@@ -2,5 +2,5 @@
 if [ "x$npm_config_node_gyp" = "x" ]; then
   node "`dirname "$0"`/../../node_modules/node-gyp/bin/node-gyp.js" "$@"
 else
-  "$npm_config_node_gyp" "$@"
+  node "`dirname "$0"`/../../node_modules/node-gyp/bin/node-gyp.js" "$@"
 fi

With this change, I was able to get node-gyp working correctly. The other changes required were related to paths and environment variables: https://github.com/joac/node-gyph-error-reproducer/commit/fe2e607a7e61e0be9727d35e1e2d670e9d6fa3e0

Seems that ncc is the source of this problem. I had problems in the past with it, trying to ship cli tools.

joac commented 1 year ago

A bit more of data: DEFAULT_NODE_GYP_PATH comes from https://github.com/pnpm/npm-lifecycle/blob/main/index.js#L20 The relevant code is the following:

let DEFAULT_NODE_GYP_PATH
try {
  DEFAULT_NODE_GYP_PATH = resolveFrom(__dirname, 'node-gyp/bin/node-gyp')
} catch (err) {}

After ncc compilation (with the rm for the original code removed on bundle.sh The code becomes:

let DEFAULT_NODE_GYP_PATH
try {
  DEFAULT_NODE_GYP_PATH = /*require.resolve*/( 1010)
} catch (err) {}

Later the terser minification removes the comments away

gregmagolan commented 1 year ago

Thanks for all the debugging @joac

I did add land an example of a package that calls node-gyp recently. https://github.com/aspect-build/rules_js/pull/586#issuecomment-1363298639

I'll catch up this thread next week as its a quiet week and will have spare time for OSS work.

hunshcn commented 1 year ago

any progress?

hunshcn commented 1 year ago

The wrong value of npm_config_node_gyp seems to be caused by index.min.js (due to compilation).

flolu commented 1 year ago

The issue came back for me starting from Bazel version 6.3.0. Downgrading to 6.2.1 fixed the problem for now.

I'm using those fixes:

https://github.com/aspect-build/rules_js/blob/0cd08de06ce8610e19cc58c1229c6b6e9cae92d3/e2e/npm_link_package/WORKSPACE#L28

https://github.com/aspect-build/rules_js/blob/0cd08de06ce8610e19cc58c1229c6b6e9cae92d3/e2e/npm_link_package/WORKSPACE#L31