Non-vendored Go modules are not properly detected

[sgnn7]

When using this codebase against a go.mod GO111MODULES-backed codebase that doesn't have code vendored, no license data for dependencies is returned as the path prefix for the packages is marked as <CWD>/vendored/.... May be related to this upstream issue when the listing is done here

To replicate save and run this file:

#!/bin/bash -e

docker run --rm \
           licensefinder/license_finder \
           /bin/bash -elc "mkdir /scan &&
                           cd /scan &&
                           wget -q https://raw.githubusercontent.com/cyberark/conjur-api-go/master/go.mod
                           wget -q https://raw.githubusercontent.com/cyberark/conjur-api-go/master/go.sum
                           license_finder report"

Output (notice the unknowns for everything):

$ ./runit.sh 
mesg: ttyname failed: Inappropriate ioctl for device
LicenseFinder::GoModules: is active
github.com/bgentry/go-netrc, v0.0.0-20140422174119-9fd32a8b3d3d, unknown
github.com/gopherjs/gopherjs, v0.0.0-20181103185306-d547d1d9531e, unknown
github.com/jtolds/gls, v4.2.2-0.20181110203027-b4936e06046b+incompatible, unknown
github.com/sirupsen/logrus, v1.0.5, unknown
github.com/smartystreets/assertions, v0.0.0-20190215210624-980c5ac6f3ac, unknown
github.com/smartystreets/goconvey, v0.0.0-20190222223459-a17d461953aa, unknown
golang.org/x/crypto, v0.0.0-20180621125126-a49355c7e3f8, unknown
golang.org/x/sys, v0.0.0-20180709060233-1b2967e3c290, unknown
gopkg.in/yaml.v2, v2.2.1, unknown

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

[xtreme-shane-lattanzio]

@sgnn7 Those paths listed are the ones that are explicitly inside of the go.mod file. The license is unknown because there does not look like there is a way to find the license unless we actually hit the web. Is this more about the fact they are unknown or you are expecting to see more than just the vendored stuff because of the -mod=vendor flag? This was done a while ago so I forget why we explicitly put that flag in there :S

[sgnn7]

@xtreme-shane-lattanzio

Those paths listed are the ones that are explicitly inside of the go.mod file

The paths in error are the ones from go list that instead of using the cache path ($GOPATH/pkg/mod) or downloading them from the web are forced to look into ./vendored/<pkgname>

The license is unknown because there does not look like there is a way to find the license unless we actually hit the web

go mod download should be able to fetch these into the local GOPATH cache for checking of licenses

Is this more about the fact they are unknown or you are expecting to see more than just the vendored stuff because of the -mod=vendor flag?

It's more about the unknown part of this - without the license identification, there's no point in running license_checker at all since it's pretty much just giving out formatted go list output and appending unknown to it. If you change the -mod=vendor in the code to -mod=readonly you can see a clear difference in internal resolution of paths.

[xtreme-shane-lattanzio]

We rely on having the code and having everything vendored in to get the source information. The prepare command that LF is running is GO111MODULE=on go mod tidy && GO111MODULE=on go mod vendor If there is no code, tidy will just clear everything out. We want to just affect the project itself and not the system.

Looking quickly regarding the -mod=vendor flag, I see the difference and in this example the path is just blank but there is still not license information that is reported. There is also one additional package that is shown.

[sgnn7]

@xtreme-shane-lattanzio Yeah maybe this particular example I've given isn't the best (I was initially running this example on this repo) but doing a go mod download and removing the -mod=vendor returned about 1/2 of the import_paths correctly from read_package which is much better than the current output.