[BUG] failed optional dependency builds should not prevent global install of packages

rigor789 commented 4 months ago

Is there an existing issue for this?

[X] I have searched the existing issues

This issue exists in the latest npm version

[X] I am using the latest npm

Current Behavior

With the inclusion of https://github.com/npm/cli/pull/7126 in v10.4 global installs of packages that have transient dependencies on packages like fsevents started failing.

Example Logs

``` npm ERR! code 1 npm ERR! path /Users/rigor789/.nvm/versions/node/v21.7.2/lib/node_modules/nativescript/node_modules/fsevents npm ERR! command failed npm ERR! command sh -c node-gyp rebuild npm ERR! gyp info it worked if it ends with ok npm ERR! gyp info using node-gyp@10.1.0 npm ERR! gyp info using node@21.7.2 | darwin | arm64 npm ERR! gyp info find Python using Python version 3.9.6 found at "/Applications/Xcode-15.2.0.app/Contents/Developer/usr/bin/python3" npm ERR! gyp info spawn /Applications/Xcode-15.2.0.app/Contents/Developer/usr/bin/python3 npm ERR! gyp info spawn args [ npm ERR! gyp info spawn args '/Users/rigor789/.nvm/versions/node/v21.7.2/lib/node_modules/nativescript/node_modules/node-gyp/gyp/gyp_main.py', npm ERR! gyp info spawn args 'binding.gyp', npm ERR! gyp info spawn args '-f', npm ERR! gyp info spawn args 'make', npm ERR! gyp info spawn args '-I', npm ERR! gyp info spawn args '/Users/rigor789/.nvm/versions/node/v21.7.2/lib/node_modules/nativescript/node_modules/fsevents/build/config.gypi', npm ERR! gyp info spawn args '-I', npm ERR! gyp info spawn args '/Users/rigor789/.nvm/versions/node/v21.7.2/lib/node_modules/nativescript/node_modules/node-gyp/addon.gypi', npm ERR! gyp info spawn args '-I', npm ERR! gyp info spawn args '/Users/rigor789/Library/Caches/node-gyp/21.7.2/include/node/common.gypi', npm ERR! gyp info spawn args '-Dlibrary=shared_library', npm ERR! gyp info spawn args '-Dvisibility=default', npm ERR! gyp info spawn args '-Dnode_root_dir=/Users/rigor789/Library/Caches/node-gyp/21.7.2', npm ERR! gyp info spawn args '-Dnode_gyp_dir=/Users/rigor789/.nvm/versions/node/v21.7.2/lib/node_modules/nativescript/node_modules/node-gyp', npm ERR! gyp info spawn args '-Dnode_lib_file=/Users/rigor789/Library/Caches/node-gyp/21.7.2/<(target_arch)/node.lib', npm ERR! gyp info spawn args '-Dmodule_root_dir=/Users/rigor789/.nvm/versions/node/v21.7.2/lib/node_modules/nativescript/node_modules/fsevents', npm ERR! gyp info spawn args '-Dnode_engine=v8', npm ERR! gyp info spawn args '--depth=.', npm ERR! gyp info spawn args '--no-parallel', npm ERR! gyp info spawn args '--generator-output', npm ERR! gyp info spawn args 'build', npm ERR! gyp info spawn args '-Goutput_dir=.' npm ERR! gyp info spawn args ] npm ERR! gyp: binding.gyp not found (cwd: /Users/rigor789/.nvm/versions/node/v21.7.2/lib/node_modules/nativescript/node_modules/fsevents) while trying to load binding.gyp npm ERR! gyp ERR! configure error npm ERR! gyp ERR! stack Error: `gyp` failed with exit code: 1 npm ERR! gyp ERR! stack at ChildProcess. (/Users/rigor789/.nvm/versions/node/v21.7.2/lib/node_modules/nativescript/node_modules/node-gyp/lib/configure.js:297:18) npm ERR! gyp ERR! stack at ChildProcess.emit (node:events:519:28) npm ERR! gyp ERR! stack at ChildProcess._handle.onexit (node:internal/child_process:294:12) npm ERR! gyp ERR! System Darwin 23.4.0 npm ERR! gyp ERR! command "/Users/rigor789/.nvm/versions/node/v21.7.2/bin/node" "/Users/rigor789/.nvm/versions/node/v21.7.2/lib/node_modules/nativescript/node_modules/.bin/node-gyp" "rebuild" npm ERR! gyp ERR! cwd /Users/rigor789/.nvm/versions/node/v21.7.2/lib/node_modules/nativescript/node_modules/fsevents npm ERR! gyp ERR! node -v v21.7.2 npm ERR! gyp ERR! node-gyp -v v10.1.0 npm ERR! gyp ERR! not ok npm ERR! A complete log of this run can be found in: /Users/rigor789/.npm/_logs/2024-04-09T09_05_41_833Z-debug-0.log ```

Expected Behavior

Global installs should not fail. fsevents is an optional peerDependency, and installing it should not trigger a rebuild, nor fail on unsupported systems (like Linux/Windows).

Steps To Reproduce

install npm 10.4 or newer
run npm i -g nativescript
the install will fail

To confirm the linked change breaks it:

open node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/build-ideal-tree.js of your global packages
edit line 1235 and comment out // fullMetadata: true,
re-run: npm i -g nativescript
the install works as expected.

Environment

npm: 10.4.0 (or newer)
Node.js: v21.7.2
OS Name: macOS Sonoma 14.4.1
System Model Name: MacBook Pro

npm config:


; "user" config from /Users/rigor789/.npmrc

//registry.npmjs.org/:_authToken = (protected)

; node bin location = /Users/rigor789/.nvm/versions/node/v21.7.2/bin/node ; node version = v21.7.2 ; npm local prefix = /Users/rigor789/Code/ ; npm version = 10.4.0 ; cwd = /Users/rigor789/Code/ ; HOME = /Users/rigor789 ; Run npm config ls -l to show all defaults.

rigor789 commented 4 months ago

If it's helpful, here are the manifests before/after including fullMetadata: true

Without `fullMetadata: true`

```js { name: 'fsevents', version: '2.3.3', devDependencies: { 'node-gyp': '^9.4.0' }, dist: { integrity: 'sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==', shasum: 'cac6407785d03675a2a5e1a5305c697b347d90d6', tarball: 'https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz', fileCount: 6, unpackedSize: 173224, signatures: [ [Object] ] }, engines: { node: '^8.16.0 || ^10.6.0 || >=11.0.0' }, os: [ 'darwin' ], hasInstallScript: true, _id: 'fsevents@2.3.3', _resolved: 'https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz', _from: 'fsevents@~2.3.2', _integrity: 'sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==', _signatures: [ { keyid: 'SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA', sig: 'MEUCIBqKmzRJwFJcSLXc/SdNTdFsUQvsS/tX+pFHr2JkniDbAiEAiTjoxV3W2IK2g8Udbes/ZP46+TtL40PX3IX/bxosfWs=' } ] } ```

With `fullMetadata: true`

```js { name: 'fsevents', version: '2.3.3', description: 'Native Access to MacOS FSEvents', main: 'fsevents.js', types: 'fsevents.d.ts', os: [ 'darwin' ], engines: { node: '^8.16.0 || ^10.6.0 || >=11.0.0' }, scripts: { clean: 'node-gyp clean && rm -f fsevents.node', build: 'node-gyp clean && rm -f fsevents.node && node-gyp rebuild && node-gyp clean', test: '/bin/bash ./test.sh 2>/dev/null', prepublishOnly: 'npm run build', install: 'node-gyp rebuild' }, repository: { type: 'git', url: 'git+https://github.com/fsevents/fsevents.git' }, keywords: [ 'fsevents', 'mac' ], contributors: [ { name: 'Philipp Dunkel', email: 'pip@pipobscure.com' }, { name: 'Ben Noordhuis', email: 'info@bnoordhuis.nl' }, { name: 'Elan Shankar', email: 'elan.shanker@gmail.com' }, { name: 'Miroslav Bajtoš', email: 'mbajtoss@gmail.com' }, { name: 'Paul Miller', url: 'https://paulmillr.com' } ], bugs: { url: 'https://github.com/fsevents/fsevents/issues' }, homepage: 'https://github.com/fsevents/fsevents', devDependencies: { 'node-gyp': '^9.4.0' }, gypfile: true, gitHead: '2db891e51aa0f2975c5eaaf6aa30f13d720a830a', _id: 'fsevents@2.3.3', _nodeVersion: '18.17.1', _npmVersion: '9.6.7', dist: { integrity: 'sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==', shasum: 'cac6407785d03675a2a5e1a5305c697b347d90d6', tarball: 'https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz', fileCount: 6, unpackedSize: 173224, signatures: [ [Object] ] }, _npmUser: { name: 'pipobscure', email: 'pip@pipobscure.com' }, directories: {}, maintainers: [ { name: 'pipobscure', email: 'pip@pipobscure.com' }, { name: 'paulmillr', email: 'paul@paulmillr.com' } ], _npmOperationalInternal: { host: 's3://npm-registry-packages', tmp: 'tmp/fsevents_2.3.3_1692635062642_0.01197687980904627' }, _hasShrinkwrap: false, _time: '2023-08-21T16:24:22.854Z', _resolved: 'https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz', _from: 'fsevents@~2.3.2', _integrity: 'sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==', _signatures: [ { keyid: 'SHA256:jl3bwswu80PjjokCgh0o2w5c2U4LhQAE57gj9cz1kzA', sig: 'MEUCIBqKmzRJwFJcSLXc/SdNTdFsUQvsS/tX+pFHr2JkniDbAiEAiTjoxV3W2IK2g8Udbes/ZP46+TtL40PX3IX/bxosfWs=' } ] } ```

wraithgar commented 4 months ago

This appears to have been working by accident before. The minified packument does not have scripts which likely means that any package that required a build step would not work globally if we reverted that line.

This is still a bug, but the bug is that this should not be failing in global mode. Local mode gracefully catches this:

npm info run fsevents@2.3.3 install node_modules/fsevents node-gyp rebuild
npm info run fsevents@2.3.3 install { code: 1, signal: null }
npm verb reify failed optional dependency /Users/wraithgar/Development/npm/scratch/optional/node_modules/fsevents
npm info run nativescript@8.7.0 postinstall node_modules/nativescript node postinstall.js
npm info run nativescript@8.7.0 postinstall { code: 0, signal: null }

rigor789 commented 4 months ago

Thanks for confirming, indeed that sounds more reasonable as an explanation - I wouldn't expect optional dependencies to fail a global install.

I do wonder when that install: 'node-gyp rebuild' is being added however, because it's not in the published package.json of fsevents:

"scripts": {
  "clean": "node-gyp clean && rm -f fsevents.node",
  "build": "node-gyp clean && rm -f fsevents.node && node-gyp rebuild && node-gyp clean",
  "test": "/bin/bash ./test.sh 2>/dev/null",
  "prepublishOnly": "npm run build"
},

(from https://www.npmjs.com/package/fsevents?activeTab=code)

wraithgar commented 4 months ago

It's added automatically by npm during publish because it detects no existing install script but it finds a binding.gyp file. You can see an old issue where folks had similar problems here: https://github.com/npm/cli/issues/5234

koiker commented 2 weeks ago

Got the same issue. The fix was to downgrade npm: npm i -g npm@~10.3

npm / cli