search

npm / npm

This repository is moving to: https://github.com/npm/cli
http://npm.community
17.54k stars 3.02k forks source link

npm@3.9.0 broke nested dependency inside of optional dependency if shrinkwrapped #12698

Open johnhaley81 opened 8 years ago

johnhaley81 commented 8 years ago

Hi all! First off thanks for the awesomeness that is npm :)

The bug

So it seems that updating to npm@3.9.0 will break a package that meets the following conditions:

This happens even if the sub-dependency is also listed as an optionalDependency.

I think that this commit is most likely the culprit. Nope, that commit is in 3.9.1. Note that 3.9.1 pre-release does not fix the issue.

Repro case

I have a repo that sets up a nice test to show off the repro case of this. Just follow the README.md and you should see the bug.

Workaround

Downgrading to npm@3.8.9 fixes the issue.

johnhaley81 commented 8 years ago

Bisected out the commit responsible.

https://github.com/npm/npm/commit/e928a30947477a09245f54e9381f46b97bee32d5

iarna commented 8 years ago

I reproduced this on MacOS by replacing my compiler binaries with false:

3.3.6 (at least) to 3.8.9 output:

> fs-xattr@0.1.14 install /Users/rebecca/code/npmtest/12698/npm-optional-dependency-test/node_modules/fs-xattr
> node-gyp rebuild

gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR
gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR
gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR
  CXX(target) Release/obj.target/xattr/src/async.o
make: *** [Release/obj.target/xattr/src/async.o] Error 1
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/Users/rebecca/code/npm/node_modules/node-gyp/lib/build.js:276:23)
gyp ERR! stack     at emitTwo (events.js:106:13)
gyp ERR! stack     at ChildProcess.emit (events.js:191:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:204:12)
gyp ERR! System Darwin 15.4.0
gyp ERR! command "/Users/rebecca/.nvm/versions/node/v6.0.0/bin/node" "/Users/rebecca/code/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users/rebecca/code/npmtest/12698/npm-optional-dependency-test/node_modules/fs-xattr
gyp ERR! node -v v6.0.0
gyp ERR! node-gyp -v v3.3.1
gyp ERR! not ok
npm WARN install:fs-xattr@0.1.14 fs-xattr@0.1.14 install: `node-gyp rebuild`
npm WARN install:fs-xattr@0.1.14 Exit status 1

> macos-alias@0.2.9 install /Users/rebecca/code/npmtest/12698/npm-optional-dependency-test/node_modules/ds-store/node_modules/macos-alias
> node-gyp rebuild

gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR
gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR
gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR
  CXX(target) Release/obj.target/volume/src/volume.o
make: *** [Release/obj.target/volume/src/volume.o] Error 1
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/Users/rebecca/code/npm/node_modules/node-gyp/lib/build.js:276:23)
gyp ERR! stack     at emitTwo (events.js:106:13)
gyp ERR! stack     at ChildProcess.emit (events.js:191:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:204:12)
gyp ERR! System Darwin 15.4.0
gyp ERR! command "/Users/rebecca/.nvm/versions/node/v6.0.0/bin/node" "/Users/rebecca/code/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users/rebecca/code/npmtest/12698/npm-optional-dependency-test/node_modules/ds-store/node_modules/macos-alias
gyp ERR! node -v v6.0.0
gyp ERR! node-gyp -v v3.3.1
gyp ERR! not ok
npm WARN install:macos-alias@0.2.9 macos-alias@0.2.9 install: `node-gyp rebuild`
npm WARN install:macos-alias@0.2.9 Exit status 1

> macos-alias@0.2.9 install /Users/rebecca/code/npmtest/12698/npm-optional-dependency-test/node_modules/macos-alias
> node-gyp rebuild

gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR
gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR
gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR
  CXX(target) Release/obj.target/volume/src/volume.o
make: *** [Release/obj.target/volume/src/volume.o] Error 1
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/Users/rebecca/code/npm/node_modules/node-gyp/lib/build.js:276:23)
gyp ERR! stack     at emitTwo (events.js:106:13)
gyp ERR! stack     at ChildProcess.emit (events.js:191:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:204:12)
gyp ERR! System Darwin 15.4.0
gyp ERR! command "/Users/rebecca/.nvm/versions/node/v6.0.0/bin/node" "/Users/rebecca/code/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users/rebecca/code/npmtest/12698/npm-optional-dependency-test/node_modules/macos-alias
gyp ERR! node -v v6.0.0
gyp ERR! node-gyp -v v3.3.1
gyp ERR! not ok
npm WARN install:macos-alias@0.2.9 macos-alias@0.2.9 install: `node-gyp rebuild`
npm WARN install:macos-alias@0.2.9 Exit status 1
npm-optional-dependency-test@1.0.0 /Users/rebecca/code/npmtest/12698/npm-optional-dependency-test
├── async@1.5.2  extraneous
├── bplist-creator@0.0.6  extraneous
├── fs-temp@1.0.0  extraneous
├── image-size@0.4.0  extraneous
├── minimist@1.2.0  extraneous
├── repeat-string@1.5.4  extraneous
└── unorm@1.3.3  extraneous

npm latest:

> fs-xattr@0.1.14 install /Users/rebecca/code/npmtest/12698/npm-optional-dependency-test/node_modules/fs-xattr
> node-gyp rebuild

gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR
gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR
gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR
  CXX(target) Release/obj.target/xattr/src/async.o
make: *** [Release/obj.target/xattr/src/async.o] Error 1
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/Users/rebecca/code/npm/node_modules/node-gyp/lib/build.js:276:23)
gyp ERR! stack     at emitTwo (events.js:87:13)
gyp ERR! stack     at ChildProcess.emit (events.js:172:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:200:12)
gyp ERR! System Darwin 15.4.0
gyp ERR! command "/Users/rebecca/.nvm/versions/node/v4.4.0/bin/node" "/Users/rebecca/code/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users/rebecca/code/npmtest/12698/npm-optional-dependency-test/node_modules/fs-xattr
gyp ERR! node -v v4.4.0
gyp ERR! node-gyp -v v3.3.1
gyp ERR! not ok
npm-optional-dependency-test@1.0.0 /Users/rebecca/code/npmtest/12698/npm-optional-dependency-test
├── async@1.5.2
├─┬ ds-store@0.1.4
│ ├─┬ bplist-creator@0.0.6
│ │ └── stream-buffers@2.2.0
│ ├── macos-alias@0.2.9
│ └── unorm@1.3.3
├── fs-temp@1.0.0
├── image-size@0.4.0
├─┬ macos-alias@0.2.9
│ └── nan@2.3.3
├── minimist@1.2.0
└── repeat-string@1.5.4

npm ERR! Darwin 15.4.0
npm ERR! argv "/Users/rebecca/.nvm/versions/node/v4.4.0/bin/node" "/Users/rebecca/code/npm" "install"
npm ERR! node v4.4.0
npm ERR! npm  v3.9.1
npm ERR! code ELIFECYCLE

npm ERR! fs-xattr@0.1.14 install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the fs-xattr@0.1.14 install script 'node-gyp rebuild'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the fs-xattr package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node-gyp rebuild
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs fs-xattr
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls fs-xattr
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /Users/rebecca/code/npmtest/12698/npm-optional-dependency-test/npm-debug.log

npm 2.x output:

> fs-xattr@0.1.14 install /Users/rebecca/code/npmtest/12698/npm-optional-dependency-test/node_modules/fs-xattr
> node-gyp rebuild

gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR
gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR
gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR
  CXX(target) Release/obj.target/xattr/src/async.o
make: *** [Release/obj.target/xattr/src/async.o] Error 1
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/Users/rebecca/code/npm/node_modules/node-gyp/lib/build.js:276:23)
gyp ERR! stack     at emitTwo (events.js:87:13)
gyp ERR! stack     at ChildProcess.emit (events.js:172:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:200:12)
gyp ERR! System Darwin 15.4.0
gyp ERR! command "/Users/rebecca/.nvm/versions/node/v4.4.0/bin/node" "/Users/rebecca/code/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users/rebecca/code/npmtest/12698/npm-optional-dependency-test/node_modules/fs-xattr
gyp ERR! node -v v4.4.0
gyp ERR! node-gyp -v v3.3.1
gyp ERR! not ok

> macos-alias@0.2.9 install /Users/rebecca/code/npmtest/12698/npm-optional-dependency-test/node_modules/ds-store/node_modules/macos-alias
> node-gyp rebuild

gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR
gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR
gyp WARN download NVM_NODEJS_ORG_MIRROR is deprecated and will be removed in node-gyp v4, please use NODEJS_ORG_MIRROR
  CXX(target) Release/obj.target/volume/src/volume.o
make: *** [Release/obj.target/volume/src/volume.o] Error 1
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/Users/rebecca/code/npm/node_modules/node-gyp/lib/build.js:276:23)
gyp ERR! stack     at emitTwo (events.js:87:13)
gyp ERR! stack     at ChildProcess.emit (events.js:172:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:200:12)
gyp ERR! System Darwin 15.4.0
gyp ERR! command "/Users/rebecca/.nvm/versions/node/v4.4.0/bin/node" "/Users/rebecca/code/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users/rebecca/code/npmtest/12698/npm-optional-dependency-test/node_modules/ds-store/node_modules/macos-alias
gyp ERR! node -v v4.4.0
gyp ERR! node-gyp -v v3.3.1
gyp ERR! not ok
npm ERR! Darwin 15.4.0
npm ERR! argv "/Users/rebecca/.nvm/versions/node/v4.4.0/bin/node" "/Users/rebecca/code/npm" "install"
npm ERR! node v4.4.0
npm ERR! npm  v2.15.5
npm ERR! code ELIFECYCLE

npm ERR! fs-xattr@0.1.14 install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the fs-xattr@0.1.14 install script 'node-gyp rebuild'.
npm ERR! This is most likely a problem with the fs-xattr package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node-gyp rebuild
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs fs-xattr
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!
npm ERR!     npm owner ls fs-xattr
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /Users/rebecca/code/npmtest/12698/npm-optional-dependency-test/npm-debug.log

$ npm ls
npm-optional-dependency-test@1.0.0 /Users/rebecca/code/npmtest/12698/npm-optional-dependency-test
├─┬ appdmg@0.3.7
│ ├── async@1.5.2
│ ├── UNMET DEPENDENCY ds-store@~0.1.4
│ ├── fs-temp@1.0.0
│ ├── UNMET DEPENDENCY fs-xattr@~0.1.11
│ ├── image-size@0.4.0
│ ├── minimist@1.2.0
│ └── repeat-string@1.5.4
├── bplist-creator@0.0.6 extraneous
├── UNMET OPTIONAL DEPENDENCY macos-alias@^0.2.9
├── nan@2.3.3 extraneous
└── unorm@1.3.3 extraneous

npm ERR! extraneous: bplist-creator@0.0.6 /Users/rebecca/code/npmtest/12698/npm-optional-dependency-test/node_modules/bplist-creator
npm ERR! extraneous: nan@2.3.3 /Users/rebecca/code/npmtest/12698/npm-optional-dependency-test/node_modules/nan
npm ERR! extraneous: unorm@1.3.3 /Users/rebecca/code/npmtest/12698/npm-optional-dependency-test/node_modules/unorm
npm ERR! missing: ds-store@~0.1.4, required by appdmg@0.3.7
npm ERR! missing: fs-xattr@~0.1.11, required by appdmg@0.3.7
iarna commented 8 years ago

Now, historically, for things we didn't change on purpose we've used npm@2 as the baseline for how npm@3 should behave. It looks to me like npm@3.9.0 is back into alignment with npm@2.

That being said, the cli team seems to be in consensus that it would be desirable for optional deps to actually still be optional when using a shrinkwrap. And when they DO fail to install, they shouldn't leave a bunch of cruft around your node_modules in the form of their subdeps.

joaomoreno commented 8 years ago

We (Microsoft/vscode) will be hitting this badly once we get to 3.9.x.

joelday commented 8 years ago

Especially annoying because I have one project (VSCode) that can't use 3.9.x or newer and another project that can't use anything older than 3.9.x...

joelday commented 8 years ago

I suggest that NPM have regression tests that run on packages that opt into it.

iarna commented 8 years ago

Confirmed that this is still happening with 3.10.8.

johnhaley81 commented 7 years ago

@iarna was this also fixed with #14042 ?

iarna commented 7 years ago

npm@4.0.0 stopped producing erroring installs IF you regenerated your npm-shrinkwrap.json with npm@ >= 3.9.0 (that is, if your shrinkwrap had the optional annotations). That said, I'm not going to close this just yet as while there's a work around now, I'd like to find the root cause.

15086 further will fix the "ending in error" part of this across all installs, even with older shrinkwrap files.