yarn 1.0.1 is generating extraneous dependencies

[camwest]

Do you want to request a feature or report a bug?

Bug

What is the current behavior?

Running yarn introduces an extraneous dependency. This causes npm ls to subsequently return non-zero exit code which is breaking our CI system.

If the current behavior is a bug, please provide the steps to reproduce.

package.json

{
  "name": "foo",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "dependencies": {
    "nyc": "^11.2.1"
  }
}

1. Run yarn
2. Run npm ls
3. Run echo $?

It outputs 1 because of extraneous dependencies:

npm ERR! extraneous: archy@1.0.0 /Users/camwest/Desktop/foo/node_modules/archy
npm ERR! extraneous: caching-transform@1.0.1 /Users/camwest/Desktop/foo/node_modules/caching-transform
npm ERR! extraneous: convert-source-map@1.5.0 /Users/camwest/Desktop/foo/node_modules/convert-source-map
npm ERR! extraneous: debug-log@1.0.1 /Users/camwest/Desktop/foo/node_modules/debug-log
npm ERR! extraneous: find-cache-dir@0.1.1 /Users/camwest/Desktop/foo/node_modules/find-cache-dir
npm ERR! extraneous: istanbul-lib-hook@1.0.7 /Users/camwest/Desktop/foo/node_modules/istanbul-lib-hook
npm ERR! extraneous: istanbul-lib-instrument@1.8.0 /Users/camwest/Desktop/foo/node_modules/istanbul-lib-instrument
npm ERR! extraneous: istanbul-lib-report@1.1.1 /Users/camwest/Desktop/foo/node_modules/istanbul-lib-report
npm ERR! extraneous: istanbul-lib-source-maps@1.2.1 /Users/camwest/Desktop/foo/node_modules/istanbul-lib-source-maps
npm ERR! extraneous: istanbul-reports@1.1.2 /Users/camwest/Desktop/foo/node_modules/istanbul-reports
npm ERR! extraneous: merge-source-map@1.0.4 /Users/camwest/Desktop/foo/node_modules/merge-source-map
npm ERR! extraneous: resolve-from@2.0.0 /Users/camwest/Desktop/foo/node_modules/resolve-from
npm ERR! extraneous: spawn-wrap@1.3.8 /Users/camwest/Desktop/foo/node_modules/spawn-wrap
npm ERR! extraneous: test-exclude@4.1.1 /Users/camwest/Desktop/foo/node_modules/test-exclude
npm ERR! extraneous: yargs@8.0.2 /Users/camwest/Desktop/foo/node_modules/yargs
npm ERR! extraneous: yargs-parser@5.0.0 /Users/camwest/Desktop/foo/node_modules/yargs-parser

What is the expected behavior?

Expected no extraneous dependencies or npm ls to return with a 0 exit code.

Please mention your node.js, yarn and operating system version.

node v6.11.1 yarn v1.0.1 macOS Sierra 10.12.6

[BYK]

Yarn and npm have different resolution algorithms so it is normal for npm to think there are extra packages.

Would you mind if I ask you why you are using npm and yarn together?

[camwest]

We're using this script: https://github.com/whitesource/npm-plugin which internally calls npm (https://github.com/whitesource/npm-plugin/blob/68a400e00ea5365b9b0e2afd2aa616afbdaa9d10/bin/whitesource.js#L234)

Our company requires us to submit dependencies using this tool...

[camwest]

@BYK also I don't think Yarn's behaviour is correct. The nyc package.json has bundledDependencies specified which seems to be causing the problem. There are two sets of dependencies in my node_modules directory. One a set at the top level and a second set inside node_modules/nyc/node_modules.

[BYK]

@camwest yarn doesn't guarantee compatibility with npm ls so this workflow is not safe to follow. If you are required to use that script, I strongly suggest using npm. Alternatively, you can modify the script to use yarn check or something similar.

For the actual, potential bug, are you saying that Yarn incorrectly installs bundledDependencies even if they are used? Can you give more details about how to reproduce this and investigate locally. (just trying to get into your mind so I can follow and diagnose).

[camwest]

OK fair enough about the workflow point.

Regarding the bug. You'll see two sets of dependencies installed.

➜  foo ls node_modules
align-text                   detect-indent                is-extglob                   ms                           require-main-filename
amdefine                     error-ex                     is-finite                    normalize-package-data       resolve-from
ansi-regex                   escape-string-regexp         is-fullwidth-code-point      normalize-path               right-align
ansi-styles                  esutils                      is-glob                      npm-run-path                 rimraf
append-transform             execa                        is-number                    number-is-nan                semver
archy                        expand-brackets              is-posix-bracket             nyc                          set-blocking
arr-diff                     expand-range                 is-primitive                 object-assign                shebang-command
arr-flatten                  extglob                      is-stream                    object.omit                  shebang-regex
array-unique                 filename-regex               is-utf8                      once                         signal-exit
arrify                       fill-range                   isarray                      optimist                     slide
async                        find-cache-dir               isexe                        os-homedir                   source-map
babel-code-frame             find-up                      isobject                     os-locale                    spawn-wrap
babel-generator              for-in                       istanbul-lib-coverage        p-finally                    spdx-correct
babel-messages               for-own                      istanbul-lib-hook            p-limit                      spdx-expression-parse
babel-runtime                foreground-child             istanbul-lib-instrument      p-locate                     spdx-license-ids
babel-template               fs.realpath                  istanbul-lib-report          parse-glob                   string-width
babel-traverse               get-caller-file              istanbul-lib-source-maps     parse-json                   strip-ansi
babel-types                  get-stream                   istanbul-reports             path-exists                  strip-bom
babylon                      glob                         js-tokens                    path-is-absolute             strip-eof
balanced-match               glob-base                    jsesc                        path-key                     supports-color
brace-expansion              glob-parent                  kind-of                      path-parse                   test-exclude
braces                       globals                      lazy-cache                   path-type                    to-fast-properties
builtin-modules              graceful-fs                  lcid                         pify                         trim-right
caching-transform            handlebars                   load-json-file               pinkie                       uglify-js
camelcase                    has-ansi                     locate-path                  pinkie-promise               uglify-to-browserify
center-align                 has-flag                     lodash                       pkg-dir                      validate-npm-package-license
chalk                        hosted-git-info              longest                      preserve                     which
cliui                        imurmurhash                  loose-envify                 pseudomap                    which-module
code-point-at                inflight                     lru-cache                    randomatic                   window-size
commondir                    inherits                     md5-hex                      read-pkg                     wordwrap
concat-map                   invariant                    md5-o-matic                  read-pkg-up                  wrap-ansi
convert-source-map           invert-kv                    mem                          regenerator-runtime          wrappy
core-js                      is-arrayish                  merge-source-map             regex-cache                  write-file-atomic
cross-spawn                  is-buffer                    micromatch                   remove-trailing-separator    y18n
debug                        is-builtin-module            mimic-fn                     repeat-element               yallist
debug-log                    is-dotfile                   minimatch                    repeat-string                yargs
decamelize                   is-equal-shallow             minimist                     repeating                    yargs-parser
default-require-extensions   is-extendable                mkdirp                       require-directory

➜  foo ls node_modules/nyc/node_modules
align-text                   detect-indent                is-extglob                   ms                           resolve-from
amdefine                     error-ex                     is-finite                    normalize-package-data       right-align
ansi-regex                   escape-string-regexp         is-fullwidth-code-point      normalize-path               rimraf
ansi-styles                  esutils                      is-glob                      npm-run-path                 semver
append-transform             execa                        is-number                    number-is-nan                set-blocking
archy                        expand-brackets              is-posix-bracket             object-assign                shebang-command
arr-diff                     expand-range                 is-primitive                 object.omit                  shebang-regex
arr-flatten                  extglob                      is-stream                    once                         signal-exit
array-unique                 filename-regex               is-utf8                      optimist                     slide
arrify                       fill-range                   isarray                      os-homedir                   source-map
async                        find-cache-dir               isexe                        os-locale                    spawn-wrap
babel-code-frame             find-up                      isobject                     p-finally                    spdx-correct
babel-generator              for-in                       istanbul-lib-coverage        p-limit                      spdx-expression-parse
babel-messages               for-own                      istanbul-lib-hook            p-locate                     spdx-license-ids
babel-runtime                foreground-child             istanbul-lib-instrument      parse-glob                   string-width
babel-template               fs.realpath                  istanbul-lib-report          parse-json                   strip-ansi
babel-traverse               get-caller-file              istanbul-lib-source-maps     path-exists                  strip-bom
babel-types                  get-stream                   istanbul-reports             path-is-absolute             strip-eof
babylon                      glob                         js-tokens                    path-key                     supports-color
balanced-match               glob-base                    jsesc                        path-parse                   test-exclude
brace-expansion              glob-parent                  kind-of                      path-type                    to-fast-properties
braces                       globals                      lazy-cache                   pify                         trim-right
builtin-modules              graceful-fs                  lcid                         pinkie                       uglify-js
caching-transform            handlebars                   load-json-file               pinkie-promise               uglify-to-browserify
camelcase                    has-ansi                     locate-path                  pkg-dir                      validate-npm-package-license
center-align                 has-flag                     lodash                       preserve                     which
chalk                        hosted-git-info              longest                      pseudomap                    which-module
cliui                        imurmurhash                  loose-envify                 randomatic                   window-size
code-point-at                inflight                     lru-cache                    read-pkg                     wordwrap
commondir                    inherits                     md5-hex                      read-pkg-up                  wrap-ansi
concat-map                   invariant                    md5-o-matic                  regenerator-runtime          wrappy
convert-source-map           invert-kv                    mem                          regex-cache                  write-file-atomic
core-js                      is-arrayish                  merge-source-map             remove-trailing-separator    y18n
cross-spawn                  is-buffer                    micromatch                   repeat-element               yallist
debug                        is-builtin-module            mimic-fn                     repeat-string                yargs
debug-log                    is-dotfile                   minimatch                    repeating                    yargs-parser
decamelize                   is-equal-shallow             minimist                     require-directory
default-require-extensions   is-extendable                mkdirp                       require-main-filename

Notice how nyc/node_modules directory contains a second set of node_modules. My guess is due to them using bundledDependencies. I'd expect Yarn to de-dupe these.

[BYK]

Yeah, looks like a bug to me too. Thanks for the report. Will investigate more.

[camwest]

I might be able to help fix. Is there a place you can point me to that handles bundledDependencies?

[BYK]

@camwest there's this code: https://github.com/yarnpkg/yarn/blob/master/src/package-linker.js#L110-L128 and https://github.com/yarnpkg/yarn/blob/master/src/package-linker.js#L415-L417 but I don't think we do anything extra for bundledDependencies which is probably what we should do.

[BYK]

Related: #4532.