search

pivotal / LicenseFinder

Find licenses for your project's dependencies.
MIT License
1.73k stars 340 forks source link

Support for npm v7 and later #916

Open ganta opened 2 years ago

ganta commented 2 years ago

Currently, License Finder only supports npm v6. It barely works, as it can load lockfile v2.

However, projects using Workspaces do not work properly.

cf-gitbot commented 2 years ago

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

ganta commented 2 years ago

The workaround for this isssue is to use npm v8 in a local environment instead of a Docker image and passing --npm-options option --all to License Finder.

I would like npm v8 to work by default, even when using Docker images.

xtreme-shane-lattanzio commented 2 years ago

Hey @ganta ! Thanks for raising this.

We can add code to the npm.rb to change how license info is fetched by checking the installed version before running anything else. We have done something like this in yarn. We do not know when this would be prioritized so if you would like to take a stab at a PR, I would be happy to take a look!

Mistic92 commented 1 year ago

Hi, any update for this?

ganta commented 1 year ago

I have not been able to get started on this issue. If there is no action from the Pivotal team, it looks like there will be no update.

rhuitl commented 1 year ago

@ganta and @Mistic92 try this image from my pull request: docker.io/licensefinder/license_finder@sha256: 1f268d81217cc5dd230e627b2675d67d33698be741e93b39b208bafd74ecf778

I found that after updating a project from package-lock.json v1 to v2, the license and other details of some of its dependencies were no longer recognized. With the newer NPM version from NodeJS 17 (and --all), it works.

Mistic92 commented 1 year ago

Hi, unfortunatelly it's not working for me :/

rhuitl commented 1 year ago

Hi, unfortunatelly it's not working for me :/

What is not working? What output do you get? Can you share a reproduction case, e.g.

mkdir test && cd test && npm -y init && npm install some-package-that-causes-issues
Mistic92 commented 1 year ago

It's like it doesn't see any dependencies. I'm running it on Gitlab pipeline with command below. For yarn projects it is working. 

license_finder --recursive --npm-options="--all" --use_spdx_id true --decisions_file config/decisions.yml

We have npm and pulumi in Golang code, it is finding only Golang dependencies and main project from package.json

LicenseFinder::NPM: is active
LicenseFinder::GoModules: is active
Dependencies that need approval:
github.com/aead/chacha20, v0.0.0-20180709150244-8b13a72661da, MIT
github.com/blang/semver, v3.5.1, MIT
github.com/cheggaaa/pb, v1.0.18, BSD-3-Clause
github.com/djherbis/times, v1.2.0, MIT
github.com/emirpasic/gods, v1.12.0, BSD-2-Clause
github.com/gofrs/uuid, v3.3.0, MIT
github.com/gogo/protobuf, v1.3.2, BSD-3-Clause
github.com/golang/glog, v0.0.0-20160126235308-23def4e6c14b, Apache-2.0
github.com/golang/protobuf, v1.4.2, BSD-3-Clause
github.com/grpc-ecosystem/grpc-opentracing, v0.0.0-20180507213350-8e809c8a8645, BSD-3-Clause
github.com/hashicorp/errwrap, v1.0.0, MPL-2.0
github.com/hashicorp/go-multierror, v1.0.0, MPL-2.0
github.com/jbenet/go-context, v0.0.0-20150711004518-d14ea06fba99, MIT
github.com/kevinburke/ssh_config, v0.0.0-20190725054713-01f96b0aa0cd, MIT
github.com/mattn/go-runewidth, v0.0.8, MIT
github.com/mitchellh/go-homedir, v1.1.0, MIT
github.com/opentracing/basictracer-go, v1.0.0, MIT
github.com/opentracing/opentracing-go, v1.1.0, Apache-2.0
github.com/pkg/errors, v0.9.1, BSD-2-Clause
github.com/pkg/term, v1.1.0, BSD-2-Clause
github.com/pulumi/pulumi-gcp/sdk/v6, v6.29.0, Apache-2.0
github.com/pulumi/pulumi/sdk/v3, v3.35.3, Apache-2.0
github.com/rivo/uniseg, v0.2.0, MIT
github.com/rogpeppe/go-internal, v1.8.1, BSD-3-Clause
github.com/sabhiram/go-gitignore, v0.0.0-20180611051255-d3107576ba94, MIT
github.com/sergi/go-diff, v1.1.0, "Apache-2.0, MIT"
github.com/spf13/cast, v1.3.1, MIT
github.com/spf13/cobra, v1.4.0, Apache-2.0
github.com/spf13/pflag, v1.0.5, BSD-3-Clause
github.com/src-d/gcfg, v1.4.0, BSD-3-Clause
github.com/texttheater/golang-levenshtein, v0.0.0-20191208221605-eb6844b05fc6, MIT
github.com/tweekmonster/luser, v0.0.0-201610031726[36](https://gitlab.com/yosh-ai/yosh-clients/lpp/reserved-multilang-jovo/-/jobs/3749492980#L36)-3fa[38](https://gitlab.com/yosh-ai/yosh-clients/lpp/reserved-multilang-jovo/-/jobs/3749492980#L38)070dbd7, MIT
github.com/uber/jaeger-client-go, v2.22.1, Apache-2.0
github.com/uber/jaeger-lib, v2.2.0, Apache-2.0
github.com/xanzy/ssh-agent, v0.2.1, Apache-2.0
go.uber.org/atomic, v1.6.0, MIT
golang.org/x/crypto, v0.0.0-20200622213623-75b288015ac9, BSD-3-Clause
golang.org/x/net, v0.0.0-20201021035429-f5854403a974, BSD-3-Clause
golang.org/x/sys, v0.0.0-20210817190340-bfb29a6856f2, BSD-3-Clause
golang.org/x/text, v0.3.3, BSD-3-Clause
google.golang.org/genproto, v0.0.0-20200608115520-7c474a2e3482, Apache-2.0
google.golang.org/grpc, v1.29.1, Apache-2.0
google.golang.org/protobuf, v1.24.0, BSD-3-Clause
gopkg.in/src-d/go-billy.v4, v4.3.2, Apache-2.0
gopkg.in/src-d/go-git.v4, v4.13.1, Apache-2.0
gopkg.in/warnings.v0, v0.1.2, BSD-2-Clause
gopkg.in/yaml.v2, v2.4.0, "Apache-2.0, MIT"
XXXXXXXXXXX<PROJECT>, 1.6.0, ""
lukechampine.com/frand, v1.4.2, MIT
rhuitl commented 1 year ago

The --npm-options you provide are not necessary, but also don't explain the issue you're facing (the --all flag is automatic https://github.com/pivotal/LicenseFinder/pull/963/files#diff-2868af8118be9ff03093bf90dd5f95244c2fb7b509783822189585033787e7e1R58).

Are you sure you have NPM > 6 installed?

Try this command on your local machine:

docker run -v `pwd`:/scan -w /scan -t license_finder@sha256:1f268d81217cc5dd230e627b2675d67d33698be741e93b39b208bafd74ecf778 bash -lc 'license_finder --recursive --use_spdx_id true --decisions_file config/decisions.yml'

It will use the NPM from Node 17 that is included in the Docker image.

Mistic92 commented 1 year ago

Running it locally I got some pip error but got results for licenses in this project. But dependencies were not listed but grouped into project name. Also more Go modules were activated. I'll try to find why there is this difference running locally vs pipeline with this docker image. 

LicenseFinder::NPM: is active
LicenseFinder::GoModules: is active
LicenseFinder::Go15VendorExperiment: is active
LicenseFinder::NPM: is active
LicenseFinder::Pip: is active
pip2 install: did not succeed.
pip2 install: LicenseFinder command 'python /usr/share/rvm/gems/ruby-3.1.1/gems/license_finder-7.1.0/bin/license_finder_pip.py /scan/pulumi/vendor/github.com/pulumi/pulumi/sdk/v3/python/requirements.txt' failed:
        Traceback (most recent call last):
  File "/usr/share/rvm/gems/ruby-3.1.1/gems/license_finder-7.1.0/bin/license_finder_pip.py", line 42, in <module>
    packages = [transform(dist) for dist in pkg_resources.working_set.resolve(reqs)]
  File "/usr/local/lib/python2.7/dist-packages/pip/_vendor/pkg_resources/__init__.py", line 784, in resolve
    raise DistributionNotFound(req, requirers)
pip._vendor.pkg_resources.DistributionNotFound: The 'black>=1.0.0' distribution was not found and is required by the application

LicenseFinder::Glide: is active
LicenseFinder::Dep: is active

Dependencies that need approval:
github.com/BurntSushi/toml, 3012a1dbe2e4bd1391d42b32f0577cb7bbc7f005, ""
github.com/aead/chacha20, v0.0.0-20180709150244-8b13a72661da, MIT
github.com/beorn7/perks, 37c8de3658fcb183f997c4e13e8337516ab753e6, ""
github.com/blang/semver, v3.5.1, MIT
github.com/cheggaaa/pb, v1.0.18, BSD-3-Clause
github.com/codahale/hdrhistogram, 3a0bb77429bd3a61596f5e8a3172445844342120, ""
github.com/crossdock/crossdock-go, 049aabb0122b03bc9bd30cab8f3f91fb60166361, ""
github.com/davecgh/go-spew, 8991bc29aa16c548c550c7ff78260e27b9ab7c73, ""
github.com/davecgh/go-spew, d8f796af33cc11cb798c1aaeb27a4ebc5099927d, ""
github.com/djherbis/times, v1.2.0, MIT
github.com/emirpasic/gods, v1.12.0, BSD-2-Clause
github.com/gofrs/uuid, v3.3.0, MIT
github.com/gogo/protobuf, v1.3.2, BSD-3-Clause
github.com/golang/glog, v0.0.0-20160126235308-23def4e6c14b, Apache-2.0
github.com/golang/protobuf, 1680a479a2cfb3fa22b972af7e36d0a0fde47bf8, ""
github.com/golang/protobuf, 6c65a5562fc06764971b7c5d05c76c75e84bdbf7, ""
github.com/golang/protobuf, v1.4.2, BSD-3-Clause
github.com/grpc-ecosystem/grpc-opentracing, v0.0.0-20180507213350-8e809c8a8645, BSD-3-Clause
github.com/hashicorp/errwrap, v1.0.0, MPL-2.0
github.com/hashicorp/go-multierror, v1.0.0, MPL-2.0
github.com/jbenet/go-context, v0.0.0-20150711004518-d14ea06fba99, MIT
github.com/kevinburke/ssh_config, v0.0.0-20190725054713-01f96b0aa0cd, MIT
github.com/mattn/go-runewidth, v0.0.8, MIT
github.com/matttproud/golang_protobuf_extensions, c12348ce28de40eed0136aa2b644d0ee0650e56c, ""
github.com/matttproud/golang_protobuf_extensions, c182affec369e30f25d3eb8cd8a478dee585ae7d, ""
github.com/mitchellh/go-homedir, v1.1.0, MIT
github.com/opentracing/basictracer-go, v1.0.0, MIT
github.com/opentracing/opentracing-go, 659c90643e714681897ec2521c60567dd21da733, ""
github.com/opentracing/opentracing-go, v1.1.0, Apache-2.0
github.com/pkg/errors, ba968bfe8b2f7e042a574c888954fccecfa385b4, ""
github.com/pkg/errors, v0.9.1, BSD-2-Clause
github.com/pkg/term, v1.1.0, BSD-2-Clause
github.com/pmezard/go-difflib, 5d4384ee4fb2527b0a1256a821ebfc92f91efefc, ""
github.com/pmezard/go-difflib, 792786c7400a136282c1664665ae0a8db921c6c2, ""
github.com/prometheus/client_golang, 170205fb58decfd011f1550d4cfb737230d7ae4f, ""
github.com/prometheus/client_model, 14fe0d1b01d4d5fc031dd4bec1823bd3ebbe8016, ""
github.com/prometheus/common, 287d3e634a1e550c9e463dd7e5a75a422c614505, ""
github.com/prometheus/procfs, 499c85531f756d1129edd26485a5f73871eeb308, ""
github.com/prometheus/procfs, de25ac347ef9305868b04dc42425c973b863b18c, ""
github.com/pulumi/pulumi-gcp/sdk/v6, v6.29.0, Apache-2.0
github.com/pulumi/pulumi/sdk/v3, v3.35.3, Apache-2.0
github.com/rivo/uniseg, v0.2.0, MIT
github.com/rogpeppe/go-internal, v1.8.1, BSD-3-Clause
github.com/sabhiram/go-gitignore, v0.0.0-20180611051255-d3107576ba94, MIT
github.com/sergi/go-diff, v1.1.0, "Apache-2.0, MIT"
github.com/spf13/cast, v1.3.1, MIT
github.com/spf13/cobra, v1.4.0, Apache-2.0
github.com/spf13/pflag, v1.0.5, BSD-3-Clause
github.com/src-d/gcfg, v1.4.0, BSD-3-Clause
github.com/stretchr/testify, 221dbe5ed46703ee255b1da0dec05086f5035f62, ""
github.com/stretchr/testify, 85f2b59c4459e5bf57488796be8c3667cb8246d6, ""
github.com/texttheater/golang-levenshtein, v0.0.0-20191208221605-eb6844b05fc6, MIT
github.com/tweekmonster/luser, v0.0.0-20161003172636-3fa38070dbd7, MIT
github.com/uber-go/atomic, df976f2515e274675050de7b3f42545de80594fd, ""
github.com/uber/jaeger-client-go, v2.22.1, Apache-2.0
github.com/uber/jaeger-lib, a87ae9d84fb038a8d79266298970720be7c80fcd, ""
github.com/uber/jaeger-lib, v2.2.0, Apache-2.0
github.com/xanzy/ssh-agent, v0.2.1, Apache-2.0
go.uber.org/atomic, 9dc4df04d0d1c39369750a9f6c32c39560672089, ""
go.uber.org/atomic, df976f2515e274675050de7b3f42545de80594fd, ""
go.uber.org/atomic, v1.6.0, MIT
go.uber.org/multierr, 3c4937480c32f4c13a875a1829af76c98ca3d40a, ""
go.uber.org/multierr, c3fc3d02ec864719d8e25be2d7dde1e35a36aa27, ""
go.uber.org/tools, 2cfd321de3ee5d5f8a5fda2521d1703478334d98, ""
go.uber.org/zap, 27376062155ad36be76b0f12cf1572a221d3a48c, ""
go.uber.org/zap, a6015e13fab9b744d96085308ce4e8f11bad1996, ""
golang.org/x/crypto, v0.0.0-20200622213623-75b288015ac9, BSD-3-Clause
golang.org/x/lint, 16217165b5de779cb6a5e4fc81fa9c1166fda457, ""
golang.org/x/net, 0deb6923b6d97481cb43bc1043fe5b72a0143032, ""
golang.org/x/net, aa69164e4478b84860dc6769c710c699c67058a3, ""
golang.org/x/net, v0.0.0-20201021035429-f5854403a974, BSD-3-Clause
golang.org/x/sys, 0a153f010e6963173baba2306531d173aa843137, ""
golang.org/x/sys, f43be2a4598cf3a47be9f94f0c28197ed9eae611, ""
golang.org/x/sys, v0.0.0-20210817190340-bfb29a6856f2, BSD-3-Clause
golang.org/x/text, v0.3.3, BSD-3-Clause
golang.org/x/tools, 8dbcdeb83d3faec5315146800b375c4962a42fc6, ""
google.golang.org/genproto, v0.0.0-20200608115520-7c474a2e3482, Apache-2.0
google.golang.org/grpc, v1.29.1, Apache-2.0
google.golang.org/protobuf, v1.24.0, BSD-3-Clause
gopkg.in/src-d/go-billy.v4, v4.3.2, Apache-2.0
gopkg.in/src-d/go-git.v4, v4.13.1, Apache-2.0
gopkg.in/warnings.v0, v0.1.2, BSD-2-Clause
gopkg.in/yaml.v2, 51d6538a90f86fe93ac480b35f37b2be17fef232, ""
gopkg.in/yaml.v2, f221b8435cfb71e54062f6c6e99e9ade30b124d5, ""
gopkg.in/yaml.v2, v2.4.0, "Apache-2.0, MIT"
honnef.co/go/tools, afd67930eec2a9ed3e9b19f684d17a062285f16a, ""
XXXXXXXXX<PROJECT>, 1.6.0, "Apache-2.0, MIT, MPL-2.0, BSD-3-Clause, BSD-2-Clause"
lukechampine.com/frand, v1.4.2, MIT
semver, 3.5.1, MIT
sourcegraph.com/sourcegraph/appdash, v0.0.0-20190731080439-ebfcffb1b5c0, MIT
Mistic92 commented 1 year ago

Hi @rhuitl so it was reporting licenses locally for xxxxxxx<project> because go vendor dir was created locally. When there is no vendor dir it is reporting empty string, same as on pipeline. Can I debug it somehow?

When I remove whole dir with Pulumi and Golang it is reporting only that

LicenseFinder::NPM: is active

Dependencies that need approval:
XXXXXXXXX<PROJECT>, 1.6.0, ""
rhuitl commented 1 year ago

Are you sure you have the dependencies installed? See also https://github.com/pivotal/LicenseFinder#usage ("Make sure your dependencies are installed", npm install). If you don't, LF won't see them. You can also let LF install them if it's inconvenient to do that manually: "use the --prepare or -p option", but I personally have never tried that.

Mistic92 commented 1 year ago

That's the catch. For yarn it was working without installing first. Issue with --prepare is that it take long time from what I see. Also when I need to generate reports in html, md and just output to console it looks like it is installing everything 3 times. I had to slightly change our scripting for pipeline to check if its npm repo and install dependencies before running licensefinder. It is running very slow and after few minutes throw below error

/usr/share/rvm/gems/ruby-3.1.1/gems/license_finder-7.1.0/lib/license_finder/package_managers/npm.rb:64:in `npm_version': Command 'npm -v' failed to execute:  (RuntimeError)
    from /usr/share/rvm/gems/ruby-3.1.1/gems/license_finder-7.1.0/lib/license_finder/package_managers/npm.rb:58:in `all_flag'
    from /usr/share/rvm/gems/ruby-3.1.1/gems/license_finder-7.1.0/lib/license_finder/package_managers/npm.rb:42:in `npm_json'
    from /usr/share/rvm/gems/ruby-3.1.1/gems/license_finder-7.1.0/lib/license_finder/package_managers/npm.rb:14:in `current_packages'
    from /usr/share/rvm/gems/ruby-3.1.1/gems/license_finder-7.1.0/lib/license_finder/package_manager.rb:105:in `current_packages_with_relations'
    from /usr/share/rvm/gems/ruby-3.1.1/gems/license_finder-7.1.0/lib/license_finder/scanner.rb:42:in `each'
    from /usr/share/rvm/gems/ruby-3.1.1/gems/license_finder-7.1.0/lib/license_finder/scanner.rb:42:in `flat_map'
    from /usr/share/rvm/gems/ruby-3.1.1/gems/license_finder-7.1.0/lib/license_finder/scanner.rb:42:in `active_packages'
    from /usr/share/rvm/gems/ruby-3.1.1/gems/license_finder-7.1.0/lib/license_finder/core.rb:84:in `current_packages'
    from /usr/share/rvm/gems/ruby-3.1.1/gems/license_finder-7.1.0/lib/license_finder/core.rb:79:in `decision_applier'
    from /usr/share/rvm/rubies/ruby-3.1.1/lib/ruby/3.1.0/forwardable.rb:232:in `any_packages?'
    from /usr/share/rvm/gems/ruby-3.1.1/gems/license_finder-7.1.0/lib/license_finder/license_aggregator.rb:17:in `block in any_packages?'
    from /usr/share/rvm/gems/ruby-3.1.1/gems/license_finder-7.1.0/lib/license_finder/license_aggregator.rb:15:in `map'
    from /usr/share/rvm/gems/ruby-3.1.1/gems/license_finder-7.1.0/lib/license_finder/license_aggregator.rb:15:in `any_packages?'
    from /usr/share/rvm/gems/ruby-3.1.1/gems/license_finder-7.1.0/lib/license_finder/cli/main.rb:121:in `action_items'
    from /usr/share/rvm/gems/ruby-3.1.1/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
    from /usr/share/rvm/gems/ruby-3.1.1/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
    from /usr/share/rvm/gems/ruby-3.1.1/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'
    from /usr/share/rvm/gems/ruby-3.1.1/gems/thor-1.2.1/lib/thor/base.rb:485:in `start'
    from /usr/share/rvm/gems/ruby-3.1.1/gems/license_finder-7.1.0/bin/license_finder:6:in `<top (required)>'
    from /usr/share/rvm/gems/ruby-3.1.1/bin/license_finder:25:in `load'
    from /usr/share/rvm/gems/ruby-3.1.1/bin/license_finder:25:in `<main>'
    from /usr/share/rvm/gems/ruby-3.1.1/bin/ruby_executable_hooks:22:in `eval'
    from /usr/share/rvm/gems/ruby-3.1.1/bin/ruby_executable_hooks:22:in `<main>'

But thank you for work with handling new npm! Also do you think it will be possible to use node LTS 18.x?

rhuitl commented 1 year ago

Maybe yarn installs dependencies automatically. The time for --prepare is probably normal and means it's downloading the dependencies.

Regarding the error you got, I don't know. It's weird. LF calls npm -v and gets back the string (RuntimeError) on the stderr channel. If you can, modify npm.rb:64 and print stdout instead/in addition to stderr. Maybe that has more details. My guess is that there's something weird with the Node installation.

Node 18 should work just fine. For the Docker image, the switch to Node 18 should be very simple, but the whole image needs to be migrated to a more recent Ubuntu version. Node 18 is not supported on Bionic/18.04 (and it runs out of support in six weeks). @xtreme-shane-lattanzio are there plans to update to a later version?

I would also like to have a way to write out multiple reports in one call. I run LF three times to generate one HTML report and two JSON, with different details. You could open a feature request for that and maybe one of the core developers can say how hard it would be to implement, or maybe it's possible and I just don't know how.

xtreme-shane-lattanzio commented 1 year ago

@rhuitl I will need to update the docker image when I have time. It is on my radar. Sorry for the delays on that!

PJ-ISC commented 1 year ago

@xtreme-shane-lattanzio any updates for adding support for the newer npm versions?

xtreme-shane-lattanzio commented 9 months ago

Hey @PJ-ISC The dev version of the LF image is now on jammy but I haven't released it. Ill try to do that soon and hopefully it will resolve this

surc54 commented 7 months ago

@xtreme-shane-lattanzio Thanks so much! I tried the latest commit on master with an npm v10 project and it was working well :smile:

I'd love to have that released on rubygems, so are there any release blockers you need a hand with?