search

pivotal / LicenseFinder

Find licenses for your project's dependencies.
MIT License
1.73k stars 340 forks source link

LicenseFinder fails with npm projects #519

Open gravis opened 6 years ago

gravis commented 6 years ago

When using LicenseFinder on npm projects, we often have an error exit.

It's using npm install and npm list under the hood to get a list of dependencies, but npm list fails with:

[...]
LicenseFinder::NPM: is active
/usr/local/rvm/gems/ruby-2.5.1/gems/license_finder-5.1.0/lib/license_finder/package_managers/npm.rb:35:in `npm_json': Command 'npm list --json --long' failed to execute: npm ERR! peer dep missing: @nuxtjs/axios@^4.5.2, required by nuxtent@1.4.1 (RuntimeError)
npm ERR! missing: hawk@3.1.3, required by node-pre-gyp@0.6.39
npm ERR! missing: mkdirp@0.5.1, required by node-pre-gyp@0.6.39
npm ERR! missing: rimraf@2.6.1, required by node-pre-gyp@0.6.39
npm ERR! missing: tar@2.2.1, required by node-pre-gyp@0.6.39
npm ERR! missing: boom@2.10.1, required by hawk@3.1.3
npm ERR! missing: cryptiles@2.0.5, required by hawk@3.1.3
npm ERR! missing: hoek@2.16.3, required by hawk@3.1.3
npm ERR! missing: sntp@1.0.9, required by hawk@3.1.3
npm ERR! missing: hoek@2.16.3, required by boom@2.10.1
npm ERR! missing: boom@2.10.1, required by cryptiles@2.0.5
npm ERR! missing: hoek@2.16.3, required by sntp@1.0.9
npm ERR! missing: minimist@0.0.8, required by mkdirp@0.5.1
npm ERR! missing: console-control-strings@1.1.0, required by npmlog@4.1.0
npm ERR! missing: readable-stream@2.2.9, required by are-we-there-yet@1.1.4
npm ERR! missing: console-control-strings@1.1.0, required by gauge@2.7.4
npm ERR! missing: string-width@1.0.2, required by gauge@2.7.4
npm ERR! missing: strip-ansi@3.0.1, required by gauge@2.7.4
npm ERR! missing: code-point-at@1.1.0, required by string-width@1.0.2
npm ERR! missing: is-fullwidth-code-point@1.0.0, required by string-width@1.0.2
npm ERR! missing: strip-ansi@3.0.1, required by string-width@1.0.2
npm ERR! missing: number-is-nan@1.0.1, required by is-fullwidth-code-point@1.0.0
npm ERR! missing: ansi-regex@2.1.1, required by strip-ansi@3.0.1
npm ERR! missing: string-width@1.0.2, required by wide-align@1.1.2
npm ERR! missing: combined-stream@1.0.5, required by request@2.81.0
npm ERR! missing: hawk@3.1.3, required by request@2.81.0
npm ERR! missing: mime-types@2.1.15, required by request@2.81.0
npm ERR! missing: safe-buffer@5.0.1, required by request@2.81.0
npm ERR! missing: delayed-stream@1.0.0, required by combined-stream@1.0.5
npm ERR! missing: combined-stream@1.0.5, required by form-data@2.1.4
npm ERR! missing: mime-types@2.1.15, required by form-data@2.1.4
npm ERR! missing: extsprintf@1.0.2, required by jsprim@1.4.0
npm ERR! missing: extsprintf@1.0.2, required by verror@1.3.6
npm ERR! missing: mime-db@1.27.0, required by mime-types@2.1.15
npm ERR! missing: safe-buffer@5.0.1, required by tunnel-agent@0.6.0
npm ERR! missing: glob@7.1.2, required by rimraf@2.6.1
npm ERR! missing: fs.realpath@1.0.0, required by glob@7.1.2
npm ERR! missing: inflight@1.0.6, required by glob@7.1.2
npm ERR! missing: inherits@2.0.3, required by glob@7.1.2
npm ERR! missing: minimatch@3.0.4, required by glob@7.1.2
npm ERR! missing: once@1.4.0, required by glob@7.1.2
npm ERR! missing: path-is-absolute@1.0.1, required by glob@7.1.2
npm ERR! missing: once@1.4.0, required by inflight@1.0.6
npm ERR! missing: wrappy@1.0.2, required by inflight@1.0.6
npm ERR! missing: brace-expansion@1.1.7, required by minimatch@3.0.4
npm ERR! missing: balanced-match@0.4.2, required by brace-expansion@1.1.7
npm ERR! missing: concat-map@0.0.1, required by brace-expansion@1.1.7
npm ERR! missing: block-stream@0.0.9, required by tar@2.2.1
npm ERR! missing: fstream@1.0.11, required by tar@2.2.1
npm ERR! missing: inherits@2.0.3, required by tar@2.2.1
npm ERR! missing: inherits@2.0.3, required by block-stream@0.0.9
npm ERR! missing: graceful-fs@4.1.11, required by fstream@1.0.11
npm ERR! missing: inherits@2.0.3, required by fstream@1.0.11
npm ERR! missing: mkdirp@0.5.1, required by fstream@1.0.11
npm ERR! missing: rimraf@2.6.1, required by fstream@1.0.11
npm ERR! missing: fstream@1.0.11, required by tar-pack@3.4.0
npm ERR! missing: once@1.4.0, required by tar-pack@3.4.0
npm ERR! missing: readable-stream@2.2.9, required by tar-pack@3.4.0
npm ERR! missing: rimraf@2.6.1, required by tar-pack@3.4.0
npm ERR! missing: tar@2.2.1, required by tar-pack@3.4.0
npm ERR! missing: fstream@1.0.11, required by fstream-ignore@1.0.5
npm ERR! missing: inherits@2.0.3, required by fstream-ignore@1.0.5
npm ERR! missing: minimatch@3.0.4, required by fstream-ignore@1.0.5
npm ERR! missing: wrappy@1.0.2, required by once@1.4.0
npm ERR! missing: buffer-shims@1.0.0, required by readable-stream@2.2.9
npm ERR! missing: core-util-is@1.0.2, required by readable-stream@2.2.9
npm ERR! missing: inherits@2.0.3, required by readable-stream@2.2.9
npm ERR! missing: isarray@1.0.0, required by readable-stream@2.2.9
npm ERR! missing: process-nextick-args@1.0.7, required by readable-stream@2.2.9
npm ERR! missing: string_decoder@1.0.1, required by readable-stream@2.2.9
npm ERR! missing: util-deprecate@1.0.2, required by readable-stream@2.2.9
npm ERR! missing: safe-buffer@5.0.1, required by string_decoder@1.0.1
npm ERR! peer dep missing: ajv@^6.0.0, required by ajv-keywords@3.1.0
npm ERR! peer dep missing: ajv@^6.0.0, required by ajv-keywords@3.1.0
npm ERR! peer dep missing: ajv@^6.0.0, required by ajv-keywords@3.1.0
    from /usr/local/rvm/gems/ruby-2.5.1/gems/license_finder-5.1.0/lib/license_finder/package_managers/npm.rb:7:in `current_packages'
    from /usr/local/rvm/gems/ruby-2.5.1/gems/license_finder-5.1.0/lib/license_finder/package_manager.rb:90:in `current_packages_with_relations'
    from /usr/local/rvm/gems/ruby-2.5.1/gems/license_finder-5.1.0/lib/license_finder/scanner.rb:15:in `each'
    from /usr/local/rvm/gems/ruby-2.5.1/gems/license_finder-5.1.0/lib/license_finder/scanner.rb:15:in `flat_map'
    from /usr/local/rvm/gems/ruby-2.5.1/gems/license_finder-5.1.0/lib/license_finder/scanner.rb:15:in `active_packages'
    from /usr/local/rvm/gems/ruby-2.5.1/gems/license_finder-5.1.0/lib/license_finder/core.rb:81:in `current_packages'
    from /usr/local/rvm/gems/ruby-2.5.1/gems/license_finder-5.1.0/lib/license_finder/core.rb:76:in `decision_applier'
    from /usr/local/rvm/rubies/ruby-2.5.1/lib/ruby/2.5.0/forwardable.rb:223:in `acknowledged'
    from /usr/local/rvm/gems/ruby-2.5.1/gems/license_finder-5.1.0/lib/license_finder/license_aggregator.rb:47:in `block in aggregate_packages'
    from /usr/local/rvm/gems/ruby-2.5.1/gems/license_finder-5.1.0/lib/license_finder/license_aggregator.rb:45:in `each'
    from /usr/local/rvm/gems/ruby-2.5.1/gems/license_finder-5.1.0/lib/license_finder/license_aggregator.rb:45:in `flat_map'
    from /usr/local/rvm/gems/ruby-2.5.1/gems/license_finder-5.1.0/lib/license_finder/license_aggregator.rb:45:in `aggregate_packages'
    from /usr/local/rvm/gems/ruby-2.5.1/gems/license_finder-5.1.0/lib/license_finder/license_aggregator.rb:9:in `dependencies'
    from /usr/local/rvm/gems/ruby-2.5.1/gems/license_finder-5.1.0/lib/license_finder/cli/main.rb:127:in `report'
    from /usr/local/rvm/gems/ruby-2.5.1/gems/thor-0.20.0/lib/thor/command.rb:27:in `run'
    from /usr/local/rvm/gems/ruby-2.5.1/gems/thor-0.20.0/lib/thor/invocation.rb:126:in `invoke_command'
    from /usr/local/rvm/gems/ruby-2.5.1/gems/thor-0.20.0/lib/thor.rb:387:in `dispatch'
    from /usr/local/rvm/gems/ruby-2.5.1/gems/thor-0.20.0/lib/thor/base.rb:466:in `start'
    from /usr/local/rvm/gems/ruby-2.5.1/gems/license_finder-5.1.0/bin/license_finder:5:in `<top (required)>'
    from /usr/local/rvm/gems/ruby-2.5.1/bin/license_finder:23:in `load'
    from /usr/local/rvm/gems/ruby-2.5.1/bin/license_finder:23:in `<main>'
ERROR: Job failed: exit code 1

(with the project https://gitlab.com/gitlab-org/security-products/tests/js-npm for example)

cybercussion commented 6 years ago

+1 on a Angular project.

/usr/local/rvm/gems/ruby-2.5.1/gems/license_finder-5.1.0/lib/license_finder/package_managers/npm.rb:35:in `npm_json': Command 'npm list --json --long' failed to execute: npm ERR! peer dep missing: @angular/common@5.2.10, required by @angular/forms@5.2.10 (RuntimeError)
npm ERR! peer dep missing: @angular/common@5.2.10, required by @angular/platform-browser@5.2.10
npm ERR! peer dep missing: @angular/common@5.2.10, required by @angular/platform-browser-dynamic@5.2.10
npm ERR! peer dep missing: @angular/common@5.2.10, required by @angular/router@5.2.10
npm ERR! peer dep missing: @angular/core@5.2.11, required by @angular/animations@5.2.11
npm ERR! peer dep missing: @angular/core@5.2.11, required by @angular/common@5.2.11
npm ERR! peer dep missing: @angular/core@^4.0.0, required by angular2-powerbi@0.1.1
thekatiemcneil commented 6 years ago

I'm having the same issue as well. Anyone figure out a solution?

cybercussion commented 5 years ago

I'm going to follow up on a couple things I did to reduce some of the errors. So after locally looking at the output of npm list --json --long I did some research.

rm -r node_modules package-lock.json
npm install --no-optional
npm dedupe

This now left me with 1 ERR!

npm ERR! peer dep missing: @angular/core@^4.0.0, required by angular2-powerbi@0.1.1
josemigallas commented 5 years ago

No matter how many I have, LicenseFinder always fails when there are missing peer dependencies 😢

Command 'npm list --json --long --production' failed to execute: npm ERR! peer dep missing: X, required by Y(RuntimeError)

pmverma commented 5 years ago

I am also having same problem. Is there any workaround for this?

ajsosa commented 5 years ago

The issue is here.

https://github.com/pivotal/LicenseFinder/blob/ad1ebf4b201a88c483dd53712568f773a02740c5/lib/license_finder/package_managers/npm.rb#L35-L40

Looks like when a package has unmet peer dependencies, npm will return a list of the missing peer dependencies in stderr. An exception is raised in this case even though stdout still contains the valid data needed for license extraction.

This is probably NOT the correct way to permanently fix this as it may blow up if there are more than peer dependency errors present. But if the only error is regarding unmet peer dependencies, the below work around should work in a pinch.

    def npm_json
      command = "#{NPM.package_management_command} list --json --long#{production_flag}"
      stdout, stderr, status = Dir.chdir(project_path) { Cmd.run(command) }

      if not status.success? and not stderr.include? "npm ERR! peer dep missing:"
        raise "Command '#{command}' failed to execute: #{stderr}"
      end

      JSON.parse(stdout)
    end