Change the way component versions are detected for Golang projects

[foamkeen]

Currently for Golang projects component detection works by scanning the go.sum file (or using go mod graph command as an alternative, but the result should be the same as using go.sum file because go mod graph prints out all the versions in the graph). However according to the Golang documentation (https://go.dev/ref/mod#go-sum-files https://go.dev/ref/mod#minimal-version-selection) go.sum files shouldn’t be treated as "lock" files and they are not a reliable way to find out the versions of modules actually used by the build.

go list -m all command would be a proper way to find the actual versions of the components used during a build as it uses the minimal version selection algorithm. However it lists all the components, even no longer used ones, if they were previously used as dependencies. For example https://github.com/prometheus/pushgateway, for which go list -m all command does show github.com/gogo/protobuf v1.1.1 component, however it is no longer included in the final binary (see https://github.com/prometheus/client_golang/issues/990#issuecomment-1050207921).

The proper way to find a list of actual modules and their versions used in the build would be by either using go version -m command on the final binary or by the following command:

go list -deps -f '{{define "M"}}{{.Path}}@{{.Version}}{{end}}{{with .Module}}{{if not .Main}}{{if .Replace}}{{template "M" .Replace}}{{else}}{{template "M" .}}{{end}}{{end}}{{end}}' | sort -u

as answered here.

Could you please change the scan behaviour enabled by EnableGoCliScan environment variable to use the aforementioned go list -deps command instead of currently used go mod graph.

Thanks.

[grvillic]

@foamkeen - After experimenting with that go list -deps -f command a bit I noticed a few issues that would introduce other issues which make this a change that would introduce other problems:

• This needs to be run for every folder where a .go file is located. This would only slow the scanner performance + we would need to keep track of the directories we already scanned to prevent dup scans
• We lose the dependency graph generation unless we end up running go mod graph as well
• Package load errors in code bubble up to the detector and we shouldn't get in the business of having to manually checking each exception thrown by CLI and decide if scan was good or not. See sample scan:

  go list -deps -f "{{define \"M\"}}{{.Path}}@{{.Version}}{{end}}{{with .Module}}{{if not .Main}}{{if .Replace}}{{template \"M\" .Replace}}{{else}}{{template \"M\" .}}{{end}}{{end}}{{end}}" -test
  can't load test package: example_basic_test.go:6:2: no required module provides package github.com/sirupsen/logrus; to add it:
      go get github.com/sirupsen/logrus
  github.com/konsorten/go-windows-terminal-sequences@v1.0.1
  github.com/davecgh/go-spew@v1.1.1
  github.com/pmezard/go-difflib@v1.0.0
  github.com/stretchr/testify@v1.2.2
  github.com/stretchr/objx@v0.1.1

Not all is bad news. The following actions can be done:

1. Module Graph Pruning added on go 1.17 or higher allows go to remove any dependency that is not needed to build any package or test in a module. Using go list -m all will make more sense.

   Since go 1.17 go.mod file includes a require directive for every dependency needed to build any package or test in that module, the pruned module graph includes all of the dependencies needed to go build or go test the packages in any dependency explicitly required by the main module. A module that is not needed to build any package or test in a given module cannot affect the run-time behavior of its packages, so the dependencies that are pruned out of the module graph would only cause interference between otherwise-unrelated modules.

2. As of today go list -m all is the best choice, as it resolves to only the modules that are present in the build list. The problem here is that we still lose the dependency graph. This has an easy fix, by running both go mod graph and go list -m all, we are able to remove all nodes that were not resolved in the build list. These commands only need to be run once each, where the go.sum and go.mod files are.
3. Users can do go mod tidy so the go.mod file matches source code
4. Users can exclude unused/vulnerable dependencies from their go.sum file by adding exclude statements to their go.mod files, see exclude

[yuyue9284]

1. Module Graph Pruning added on go 1.17 or higher allows go to remove any dependency that is not needed to build any package or test in a module. Using go list -m all will make more sense.

But from the page it states that:

Modules whose requirements have been pruned out still appear in the module graph and are still reported by go list -m all

Does this mean go list -m all still have the false positive issue?