[BUG] Starting with v10.4.0, --include=optional not working (for transitive dependency (sharp))

bhallstein commented 4 months ago

Is there an existing issue for this?

[X] I have searched the existing issues

This issue exists in the latest npm version

[X] I am using the latest npm

Current Behavior

We are deploying an app that uses the sharp image manipulation library. A deployment bundle is built on macs, with npm i --cpu=x64 --os=linux --include=optional sharp@0.33.3 called afterwards to install cross-platform deps that can't be bundled, following which a tarball is created and used for deployments.

On npm@10.3.0 and below, after running the above npm i command node_modules contains dirs for sharp and also the platform-specific transitive dependencies, @img/sharp-linux-x64, @img/sharp-libvips-linux-x64, etc.

Starting with npm@10.4.0, sharp is installed but @img/... is not, breaking our deployments.

Expected Behavior

I believe the npm@10.3.0 behaviour is correct, the platform-specific transitive deps should be installed.

Steps To Reproduce

Latest node, npm@10.4.0 or 10.5.0
With this config: none needed
Run:

mkdir temp; cd temp
npm i --cpu=x64 --os=linux --include=optional sharp@0.33.3

Note that node_modules/@img is an empty directory.

Environment

npm: 10.4.0, 10.5.0, 10.5.2
Node.js: 21.7.2
OS Name: macOS Sonoma 14.4.1
System Model Name: Mac Mini M1

npm config:


; "builtin" config from /opt/homebrew/lib/node_modules/npm/npmrc

prefix = "/opt/homebrew"

; "user" config from /Users/xyz/.npmrc

//registry.npmjs.org/:_authToken = (protected)

; node bin location = /opt/homebrew/Cellar/node/21.7.3/bin/node ; node version = v21.7.3 ; npm version = 10.3.0 ; HOME = /Users/xyz ; Run npm config ls -l to show all defaults.

bhallstein commented 4 months ago

Further, tested on ubuntu as well, where cross-compilation continues to work on 10.5.0:

Mac:

Desktop:$ mkdir temp; cd temp
temp:$ node -p "[process.arch,process.platform]"
[ 'arm64', 'darwin' ]
temp:$ npm -v
10.5.0
temp:$ npm i --cpu=x64 --os=linux --include=optional sharp@0.33.3
added 11 packages in 1s
temp:$ ls -l node_modules/\@img/
total 0

Ubuntu:

muffins@localhost:~$ mkdir temp; cd temp
muffins@localhost:~/temp$ npm -v
10.5.0
muffins@localhost:~/temp$ npm i --cpu=arm64 --os=darwin --include=optional sharp@0.33.3
added 13 packages in 9s
muffins@localhost:~/temp$ ls -l node_modules/@img/
total 8
drwxrwxr-x 3 muffins muffins 4096 Apr 23 17:31 sharp-darwin-arm64
drwxrwxr-x 3 muffins muffins 4096 Apr 23 17:31 sharp-libvips-darwin-arm64

bhallstein commented 4 months ago

After some more digging over on https://github.com/lovell/sharp/issues/4080 we found that on 10.4.0+ --libc=glibc is needed for the platform-specific deps to be installed when cross-compiling sharp from mac to linux. 10.3.0 and older it works without this.

At this point I'm uncertain if this is a regression in npm, or intended behaviour. I looked at the release notes but couldn't immediately see a justification for the change. The behaviour may reside in https://github.com/npm/npm-install-checks

wraithgar commented 4 months ago

This is one of those tricky situations where a bug was fixed and some folks may have been relying on the broken behavior. The fix was in arborist. The minified packument didn't include the libc field from the manifest so npm didn't know not to install it.

npm is now currently behaving as intended.

bhallstein commented 4 months ago

Thanks for clarifying @wraithgar 🙌

npm / cli