GoMod analysis reports too many dependencies

fviernau commented 2 years ago

Reproduce:

get a copy of this directory https://github.com/oss-review-toolkit/ort/tree/7725f9285eb6d6671937384d0f2a7896cfeb3332/analyzer/src/funTest/assets/projects/synthetic/gomod
cd into dir and run fossa analyze
Find that the result contains 'gopkg.in/check.v1' which is not a (transitive) dependency of the main module, nor of its tests.

pascal-hofmann commented 1 year ago

@csasarak can you help with that? We experience the same issue and reverted to fossa v1 again because of this.

ort seems to correctly report dependencies. Theres also this interesting question on stackoverflow regarding that topic:

https://stackoverflow.com/questions/64371466/with-go-list-how-to-list-only-go-modules-used-in-the-binary

pascal-hofmann commented 1 year ago

I guess this is related to https://github.com/fossas/fossa-cli/blob/master/docs/references/strategies/languages/golang/gomodules.md#why-go-list--m--json-all-is-used-instead-of-go-list--json--deps-to-infer-dependencies - is there any progress on this?

spatten commented 1 year ago

Hi @pascal-hofmann ,

Here's what I'm seeing when I take a look at the test project in ort.

I did the following:

git clone https://github.com/oss-review-toolkit/ort.git
cd ort/analyzer/src/funTest/assets/projects/synthetic/gomod
fossa analyze --output

When I look at the output, I see these relevant entries:

          {
            "imports": [],
            "locator": "go+gopkg.in/check.v1$20d25e280405"
          },
          {
            "imports": [
              "go+gopkg.in/check.v1$20d25e280405"
            ],
            "locator": "go+gopkg.in/yaml.v3$v3.0.1"
          }

          {
            "imports": [
              "go+github.com/atomtree/go-spew$v1.1.0",
              "go+github.com/pmezard/go-difflib$v1.0.0",
              "go+github.com/stretchr/objx$v0.1.0",
              "go+gopkg.in/yaml.v3$v3.0.1"
            ],
            "locator": "go+github.com/stretchr/testify$v1.7.2"
          },
        "Imports": [
          "go+github.com/fatih/color$v1.13.0",
          "go+github.com/pborman/uuid$v1.2.1",
          "go+github.com/stretchr/testify$v1.7.2"
        ],

This means that we should see the following dependency tree:

the project imports testify (go+github.com/stretchr/testify$v1.7.2)
testify imports yaml.v3 (go+gopkg.in/yaml.v3$v3.0.1)
yaml.v3 imports check.v1 (go+gopkg.in/check.v1$20d25e280405)

So we have a three level deep dependency. That's not too bad to manually inspect, so let's do that.

I've put my notes below, but it looks to me like this is correct. The project imports testify, which imports yaml.v3, which imports check.v1.

Does this look correct to you?

Here are my notes:

Project imports testify

https://github.com/oss-review-toolkit/ort/blob/main/analyzer/src/funTest/assets/projects/synthetic/gomod/go.mod#L23

module github.com/oss-review-toolkit/ort/gomod-synthetic-test-project

go 1.18

require (
    github.com/fatih/color v1.13.0
    github.com/pborman/uuid v1.2.1
    github.com/stretchr/testify v1.7.1
)

...

replace (
    github.com/davecgh/go-spew => github.com/atomtree/go-spew v1.1.0
    github.com/stretchr/testify => github.com/stretchr/testify v1.7.2
)

Note that there's a replace directive that upgrades testify from 1.7.1 to 1.7.2.

Testify imports yaml.v3

https://github.com/stretchr/testify/blob/v1.7.2/go.mod

module github.com/stretchr/testify

go 1.13

require (
    github.com/davecgh/go-spew v1.1.1
    github.com/pmezard/go-difflib v1.0.0
    github.com/stretchr/objx v0.4.0
    gopkg.in/yaml.v3 v3.0.1
)

yaml.v3 imports check.v1

https://github.com/go-yaml/yaml/blob/v3.0.1/go.mod

module "gopkg.in/yaml.v3"

require (
    "gopkg.in/check.v1" v0.0.0-20161208181325-20d25e280405
)

pascal-hofmann commented 1 year ago

Does this look correct to you?

It does not look correct to me, as (theoretical) test dependencies are irrelevant and won't be included in the resulting binary. So there is no need to include them in the license report.

Even with included tests: testify depends on yaml which depends on check, but no code of these two packages is used in the tests.

IMHO these are the only things that should be detected as a dependency:

$ git clone https://github.com/oss-review-toolkit/ort.git
$ cd ort/analyzer/src/funTest/assets/projects/synthetic/gomod
$ go list -deps -f '{{define "M"}}{{.Path}}@{{.Version}}{{end}}{{with .Module}}{{if not .Main}}{{if .Replace}}{{template "M" .Replace}}{{else}}{{template "M" .}}{{end}}{{end}}{{end}}' | sort -u
github.com/fatih/color@v1.13.0
github.com/google/uuid@v1.0.0
github.com/mattn/go-colorable@v0.1.12
github.com/mattn/go-isatty@v0.0.14
github.com/pborman/uuid@v1.2.1
golang.org/x/sys@v0.0.0-20220610221304-9f5ed59c137d

csasarak commented 1 year ago

Hi,

I agree with you that we should filter out test imports. Ways to improve our Go support is something we're discussing on an ongoing basis. I will make sure that we keep this this requirement in mind with whatever solution we come up with so that it matches our behavior for other languages. I have also backlogged some work to try to do this filtering with our current strategy.

Regarding your second point I want to make sure I follow what you're saying. Are you saying that if I have a dependency A that I import directly which transitively imports package C, none of the code I use from A relies on code in C then C should not be included as a dependency?

pascal-hofmann commented 1 year ago

Hi,

Regarding your second point I want to make sure I follow what you're saying. Are you saying that if I have a dependency A that I import directly which transitively imports package C, none of the code I use from A relies on code in C then C should not be included as a dependency?

Yes. But even if I have something as direct dependency it should only be included if it's actually used. This is also how the old fossa-cli used to work. Only packages from which code is linked into the binary, or which the binary links against dynamically need to be included in the license report. The rest is irrelevant.

With the current behavior you get very big attribution reports including a lot of things that are not used at all.

Funny example: We have https://github.com/go-gl/gl (Go bindings for OpenGL) included in an attribution report for a backend server because we use protobuf which has a lot of transitive dependencies.

For this project the difference in reporting between fossa cli v3 and the old version is:

291 instead of 22 dependencies
33 instead of 12 licenses
4 instead of 0 issues

pascal-hofmann commented 1 year ago

Is there any update on this? It would be great to get this fixed.

fossa is pretty useless for us with this buggy reporting behavior.

csasarak commented 1 year ago

Hi Pascal,

We are currently working on a new version of our Go strategy that we're hoping to release in the next few weeks which should address these issues. I have you on my list of people to reach out to directly when it is available.

Regards, Chris

csasarak commented 1 year ago

Hi @pascal-hofmann,

Today we released a new Go resolver in version 3.7.5, but in order to collect feedback we've released it under a flag. To use this functionality you will need to run fossa analyze with the new flag: fossa analyze --experimental-use-v3-go-resolver. To verify that the new strategy is running, you can run using the --debug flag and should look for messages like analysis using go list (V3 Resolver) that aren't followed by messages indicating the use of static analysis or analysis using go list (modules).

Users can expect to see a different set of dependencies. The common case should be that they will see fewer dependencies. More specific things that users of this flag should expect:

It should mark transitive/direct deps correctly.
It should not include test dependencies.
~~Path dependencies from Go replace directives aren't included in the dependency list, but their transitive dependencies should be. You can find more docs here.~~ I apologize, I wrote the above in error. For now we are excluding path deps and their children.

You're welcome to respond here if you run into problems, but to do any debugging on non-public code we will likely need the debug bundle that fossa outputs to fossa.debug.gz. It may be best to send that by filing an issue at https://support.fossa.com

pascal-hofmann commented 1 year ago

@csasarak Awesome thank you for your work on this!

I just tried to use the new resolver, but somehow it seems like the old resolver is still being used too. What am I doing wrong here?

2023-04-13 08:57:11+02 + fossa analyze '--with-telemetry-scope=off' --experimental-use-v3-go-resolver --debug --branch pr-233
2023-04-13 08:57:11+02 [DEBUG] Loading configuration file from "…"
2023-04-13 08:57:12+02 [DEBUG] [TASK 2] cabal
2023-04-13 08:57:12+02 [DEBUG] [TASK 2] cabal > Finding Projects
2023-04-13 08:57:12+02 [DEBUG] [TASK 4] swift
2023-04-13 08:57:12+02 [DEBUG] [TASK 3] godep
2023-04-13 08:57:12+02 [DEBUG] [TASK 2] cabal > Finding Projects > Walking the filetree
2023-04-13 08:57:12+02 [DEBUG] [TASK 4] swift > Finding Projects
2023-04-13 08:57:12+02 [DEBUG] [TASK 4] swift > Finding Projects > Finding swift package projects
2023-04-13 08:57:12+02 [DEBUG] [TASK 3] godep > Finding Projects
2023-04-13 08:57:12+02 [DEBUG] [TASK 4] swift > Finding Projects > Finding swift package projects > Walking the filetree
2023-04-13 08:57:12+02 [DEBUG] [TASK 3] godep > Finding Projects > Walking the filetree
2023-04-13 08:57:12+02 [DEBUG] [TASK 1] bundler
2023-04-13 08:57:12+02 [ INFO] [ 31 Waiting / 4 Running / 1 Completed ]
2023-04-13 08:57:12+02 [DEBUG] [TASK 1] bundler > Finding Projects
2023-04-13 08:57:12+02 [DEBUG] [TASK 1] bundler > Finding Projects > Walking the filetree
2023-04-13 08:57:12+02 [DEBUG] [TASK 5] stack
2023-04-13 08:57:12+02 [DEBUG] [TASK 5] stack > Finding Projects
2023-04-13 08:57:12+02 [DEBUG] [TASK 5] stack > Finding Projects > Walking the filetree
2023-04-13 08:57:12+02 [DEBUG] [TASK 6] setuptools
2023-04-13 08:57:12+02 [DEBUG] [TASK 6] setuptools > Finding Projects
2023-04-13 08:57:12+02 [DEBUG] [TASK 6] setuptools > Finding Projects > Walking the filetree
2023-04-13 08:57:12+02 [DEBUG] [TASK 7] scala
2023-04-13 08:57:12+02 [ INFO] [ 28 Waiting / 4 Running / 4 Completed ]
2023-04-13 08:57:12+02 [DEBUG] [TASK 7] scala > Finding Projects
2023-04-13 08:57:12+02 [DEBUG] [TASK 7] scala > Finding Projects > Walking the filetree
2023-04-13 08:57:12+02 [DEBUG] [TASK 4] swift > Finding Projects > Finding xcode projects using swift package manager
2023-04-13 08:57:12+02 [DEBUG] [TASK 4] swift > Finding Projects > Finding xcode projects using swift package manager > Walking the filetree
2023-04-13 08:57:12+02 [DEBUG] [TASK 8] repomanifest
2023-04-13 08:57:12+02 [DEBUG] [TASK 8] repomanifest > Finding Projects
2023-04-13 08:57:12+02 [DEBUG] [TASK 8] repomanifest > Finding Projects > Walking the filetree
2023-04-13 08:57:12+02 [DEBUG] [TASK 9] rebar3
2023-04-13 08:57:12+02 [DEBUG] [TASK 9] rebar3 > Finding Projects
2023-04-13 08:57:12+02 [DEBUG] [TASK 9] rebar3 > Finding Projects > Walking the filetree
2023-04-13 08:57:12+02 [ INFO] [ 27 Waiting / 4 Running / 5 Completed ]
2023-04-13 08:57:12+02 [DEBUG] [TASK 10] rpm
2023-04-13 08:57:12+02 [ INFO] [ 25 Waiting / 4 Running / 7 Completed ]
2023-04-13 08:57:12+02 [DEBUG] [TASK 10] rpm > Finding Projects
2023-04-13 08:57:12+02 [DEBUG] [TASK 10] rpm > Finding Projects > Walking the filetree
2023-04-13 08:57:12+02 [DEBUG] [TASK 11] renv
2023-04-13 08:57:12+02 [DEBUG] [TASK 11] renv > Finding Projects
2023-04-13 08:57:12+02 [DEBUG] [TASK 11] renv > Finding Projects > Walking the filetree
2023-04-13 08:57:12+02 [ INFO] [ 24 Waiting / 4 Running / 8 Completed ]
2023-04-13 08:57:12+02 [DEBUG] [TASK 12] pub
2023-04-13 08:57:12+02 [DEBUG] [TASK 12] pub > Finding Projects
2023-04-13 08:57:12+02 [DEBUG] [TASK 12] pub > Finding Projects > Walking the filetree
2023-04-13 08:57:12+02 [DEBUG] [TASK 13] projectjson
2023-04-13 08:57:12+02 [ INFO] [ 23 Waiting / 4 Running / 9 Completed ]
2023-04-13 08:57:12+02 [DEBUG] [TASK 13] projectjson > Finding Projects
2023-04-13 08:57:12+02 [DEBUG] [TASK 13] projectjson > Finding Projects > Walking the filetree
2023-04-13 08:57:12+02 [ INFO] [ 22 Waiting / 4 Running / 10 Completed ]
2023-04-13 08:57:12+02 [DEBUG] [TASK 14] projectassetsjson
2023-04-13 08:57:12+02 [DEBUG] [TASK 14] projectassetsjson > Finding Projects
2023-04-13 08:57:12+02 [ INFO] [ 21 Waiting / 4 Running / 11 Completed ]
2023-04-13 08:57:12+02 [DEBUG] [TASK 14] projectassetsjson > Finding Projects > Walking the filetree
2023-04-13 08:57:12+02 [DEBUG] [TASK 15] poetry
2023-04-13 08:57:12+02 [ INFO] [ 20 Waiting / 4 Running / 12 Completed ]
2023-04-13 08:57:12+02 [DEBUG] [TASK 15] poetry > Finding Projects
2023-04-13 08:57:12+02 [DEBUG] [TASK 15] poetry > Finding Projects > Walking the filetree
2023-04-13 08:57:12+02 [DEBUG] [TASK 16] pipenv
2023-04-13 08:57:12+02 [DEBUG] [TASK 16] pipenv > Finding Projects
2023-04-13 08:57:12+02 [DEBUG] [TASK 16] pipenv > Finding Projects > Walking the filetree
2023-04-13 08:57:12+02 [ INFO] [ 19 Waiting / 4 Running / 13 Completed ]
2023-04-13 08:57:12+02 [DEBUG] [TASK 17] perl
2023-04-13 08:57:12+02 [DEBUG] [TASK 17] perl > Finding Projects
2023-04-13 08:57:12+02 [DEBUG] [TASK 17] perl > Finding Projects > Walking the filetree
2023-04-13 08:57:12+02 [ INFO] [ 18 Waiting / 4 Running / 14 Completed ]
2023-04-13 08:57:12+02 [DEBUG] [TASK 18] paket
2023-04-13 08:57:12+02 [DEBUG] [TASK 18] paket > Finding Projects
2023-04-13 08:57:12+02 [DEBUG] [TASK 18] paket > Finding Projects > Walking the filetree
2023-04-13 08:57:12+02 [ INFO] [ 17 Waiting / 4 Running / 15 Completed ]
2023-04-13 08:57:12+02 [DEBUG] [TASK 19] packagesconfig
2023-04-13 08:57:12+02 [ INFO] [ 16 Waiting / 4 Running / 16 Completed ]
2023-04-13 08:57:12+02 [DEBUG] [TASK 19] packagesconfig > Finding Projects
2023-04-13 08:57:12+02 [DEBUG] [TASK 19] packagesconfig > Finding Projects > Walking the filetree
2023-04-13 08:57:12+02 [DEBUG] [TASK 20] packagereference
2023-04-13 08:57:12+02 [DEBUG] [TASK 20] packagereference > Finding Projects
2023-04-13 08:57:12+02 [DEBUG] [TASK 20] packagereference > Finding Projects > Walking the filetree
2023-04-13 08:57:12+02 [ INFO] [ 15 Waiting / 4 Running / 17 Completed ]
2023-04-13 08:57:12+02 [DEBUG] [TASK 21] nuspec
2023-04-13 08:57:12+02 [DEBUG] [TASK 21] nuspec > Finding Projects
2023-04-13 08:57:12+02 [DEBUG] [TASK 21] nuspec > Finding Projects > Walking the filetree
2023-04-13 08:57:12+02 [ INFO] [ 14 Waiting / 4 Running / 18 Completed ]
2023-04-13 08:57:12+02 [DEBUG] [TASK 23] nimble
2023-04-13 08:57:12+02 [DEBUG] [TASK 23] nimble > Finding Projects
2023-04-13 08:57:12+02 [DEBUG] [TASK 23] nimble > Finding Projects > Walking the filetree
2023-04-13 08:57:12+02 [DEBUG] [TASK 22] NodeJS
2023-04-13 08:57:12+02 [ INFO] [ 13 Waiting / 4 Running / 19 Completed ]
2023-04-13 08:57:12+02 [DEBUG] [TASK 22] NodeJS > Finding nodejs projects
2023-04-13 08:57:12+02 [ INFO] [ 12 Waiting / 4 Running / 20 Completed ]
2023-04-13 08:57:12+02 [DEBUG] [TASK 22] NodeJS > Finding nodejs projects > Walking the filetree
2023-04-13 08:57:12+02 [DEBUG] [TASK 24] mix
2023-04-13 08:57:12+02 [DEBUG] [TASK 24] mix > Finding Projects
2023-04-13 08:57:12+02 [DEBUG] [TASK 24] mix > Finding Projects > Walking the filetree
2023-04-13 08:57:12+02 [ INFO] [ 11 Waiting / 4 Running / 21 Completed ]
2023-04-13 08:57:12+02 [DEBUG] [TASK 25] maven
2023-04-13 08:57:12+02 [DEBUG] [TASK 25] maven > Finding Projects
2023-04-13 08:57:12+02 [DEBUG] [TASK 25] maven > Finding Projects > Finding pom files
2023-04-13 08:57:12+02 [DEBUG] [TASK 25] maven > Finding Projects > Finding pom files > Walking the filetree
2023-04-13 08:57:12+02 [ INFO] [ 10 Waiting / 4 Running / 22 Completed ]
2023-04-13 08:57:12+02 [DEBUG] [TASK 26] leiningen
2023-04-13 08:57:12+02 [DEBUG] [TASK 26] leiningen > Finding Projects
2023-04-13 08:57:12+02 [DEBUG] [TASK 26] leiningen > Finding Projects > Walking the filetree
2023-04-13 08:57:12+02 [ INFO] [ 9 Waiting / 4 Running / 23 Completed ]
2023-04-13 08:57:12+02 [DEBUG] [TASK 22] NodeJS > Reading package.json files
2023-04-13 08:57:12+02 [DEBUG] [TASK 22] NodeJS > Reading package.json files > Parsing JSON file '…'
2023-04-13 08:57:12+02 [DEBUG] [TASK 22] NodeJS > Building global workspace graph
2023-04-13 08:57:12+02 [DEBUG] [TASK 22] NodeJS > Splitting global graph into chunks
2023-04-13 08:57:12+02 [DEBUG] [TASK 22] NodeJS > Converting graphs to analysis targets
2023-04-13 08:57:12+02 [ INFO] Analyzing yarn project at …
2023-04-13 08:57:12+02 [ INFO] [ 9 Waiting / 4 Running / 24 Completed ]
2023-04-13 08:57:12+02 [DEBUG] [TASK 27] Project Analysis: YarnProjectType
2023-04-13 08:57:12+02 [DEBUG] [TASK 27] Project Analysis: YarnProjectType > Reading yarn.lock file
2023-04-13 08:57:12+02 [DEBUG] [TASK 28] gradle
2023-04-13 08:57:12+02 [DEBUG] [TASK 28] gradle > Finding Projects
2023-04-13 08:57:12+02 [DEBUG] [TASK 28] gradle > Finding Projects > Walking the filetree
2023-04-13 08:57:12+02 [ INFO] [ 8 Waiting / 4 Running / 25 Completed ]
2023-04-13 08:57:13+02 [DEBUG] [TASK 25] maven > Finding Projects > Building global closure
2023-04-13 08:57:13+02 [DEBUG] [TASK 25] maven > Finding Projects > Building project closures
2023-04-13 08:57:13+02 [DEBUG] [TASK 29] gomod
2023-04-13 08:57:13+02 [DEBUG] [TASK 29] gomod > Finding Projects
2023-04-13 08:57:13+02 [DEBUG] [TASK 29] gomod > Finding Projects > Walking the filetree
2023-04-13 08:57:13+02 [ INFO] [ 7 Waiting / 4 Running / 26 Completed ]
2023-04-13 08:57:13+02 [DEBUG] [TASK 30] glide
2023-04-13 08:57:13+02 [DEBUG] [TASK 30] glide > Finding Projects
2023-04-13 08:57:13+02 [DEBUG] [TASK 30] glide > Finding Projects > Walking the filetree
2023-04-13 08:57:13+02 [ INFO] [ 6 Waiting / 4 Running / 27 Completed ]
2023-04-13 08:57:13+02 [DEBUG] [TASK 31] fpm
2023-04-13 08:57:13+02 [DEBUG] [TASK 31] fpm > Finding Projects
2023-04-13 08:57:13+02 [DEBUG] [TASK 31] fpm > Finding Projects > Walking the filetree
2023-04-13 08:57:13+02 [ INFO] [ 5 Waiting / 4 Running / 28 Completed ]
2023-04-13 08:57:13+02 [ INFO] Analyzing gomod project at …
2023-04-13 08:57:13+02 [DEBUG] [TASK 32] Project Analysis: GomodProjectType
2023-04-13 08:57:13+02 [DEBUG] [TASK 32] Project Analysis: GomodProjectType > Gomodules
2023-04-13 08:57:13+02 [DEBUG] [TASK 32] Project Analysis: GomodProjectType > Gomodules > Dynamic analysis
2023-04-13 08:57:13+02 [DEBUG] [TASK 32] Project Analysis: GomodProjectType > Gomodules > Dynamic analysis > analysis using go list (V3 Resolver)
2023-04-13 08:57:13+02 [DEBUG] [TASK 32] Project Analysis: GomodProjectType > Gomodules > Dynamic analysis > analysis using go list (V3 Resolver) > Getting dependencies using 'go list -e -json -deps all'
2023-04-13 08:57:13+02 [DEBUG] [TASK 32] Project Analysis: GomodProjectType > Gomodules > Dynamic analysis > analysis using go list (V3 Resolver) > Getting dependencies using 'go list -e -json -deps all' > Running command 'go'
2023-04-13 08:57:13+02 [ INFO] [ 5 Waiting / 4 Running / 29 Completed ]
2023-04-13 08:57:13+02 [DEBUG] [TASK 33] conda
2023-04-13 08:57:13+02 [DEBUG] [TASK 33] conda > Finding Projects
2023-04-13 08:57:13+02 [DEBUG] [TASK 33] conda > Finding Projects > Walking the filetree
2023-04-13 08:57:13+02 [ INFO] [ 4 Waiting / 4 Running / 30 Completed ]
2023-04-13 08:57:13+02 [DEBUG] [TASK 34] composer
2023-04-13 08:57:13+02 [DEBUG] [TASK 34] composer > Finding Projects
2023-04-13 08:57:13+02 [DEBUG] [TASK 34] composer > Finding Projects > Walking the filetree
2023-04-13 08:57:13+02 [ INFO] [ 3 Waiting / 4 Running / 31 Completed ]
2023-04-13 08:57:13+02 [DEBUG] [TASK 27] Project Analysis: YarnProjectType > Parsing yarn.lock file
2023-04-13 08:57:13+02 [DEBUG] [TASK 27] Project Analysis: YarnProjectType > Building yarn.lock package graph
2023-04-13 08:57:13+02 [ INFO] [ 3 Waiting / 3 Running / 32 Completed ]
2023-04-13 08:57:13+02 [ INFO] [ 2 Waiting / 4 Running / 32 Completed ]
2023-04-13 08:57:13+02 [DEBUG] [TASK 35] cocoapods
2023-04-13 08:57:13+02 [DEBUG] [TASK 35] cocoapods > Finding Projects
2023-04-13 08:57:13+02 [DEBUG] [TASK 35] cocoapods > Finding Projects > Walking the filetree
2023-04-13 08:57:13+02 [DEBUG] [TASK 36] carthage
2023-04-13 08:57:13+02 [DEBUG] [TASK 36] carthage > Finding Projects
2023-04-13 08:57:13+02 [DEBUG] [TASK 36] carthage > Finding Projects > Walking the filetree
2023-04-13 08:57:13+02 [ INFO] [ 1 Waiting / 4 Running / 33 Completed ]
2023-04-13 08:57:13+02 [DEBUG] [TASK 37] cargo
2023-04-13 08:57:13+02 [DEBUG] [TASK 37] cargo > Finding Projects
2023-04-13 08:57:13+02 [DEBUG] [TASK 37] cargo > Finding Projects > Walking the filetree
2023-04-13 08:57:13+02 [ INFO] [ 0 Waiting / 4 Running / 34 Completed ]
2023-04-13 08:57:14+02 [ INFO] [ 0 Waiting / 3 Running / 35 Completed ]
2023-04-13 08:57:14+02 [ INFO] [ 0 Waiting / 2 Running / 36 Completed ]
2023-04-13 08:57:14+02 [ INFO] [ 0 Waiting / 1 Running / 37 Completed ]
2023-04-13 08:57:22+02 [DEBUG] [TASK 32] Project Analysis: GomodProjectType > Gomodules > Dynamic analysis > analysis using go list (V3 Resolver) > Analyzing dependencies
2023-04-13 08:57:22+02 [DEBUG] [TASK 32] Project Analysis: GomodProjectType > Gomodules > Dynamic analysis > analysis using go mod graph
2023-04-13 08:57:22+02 [DEBUG] [TASK 32] Project Analysis: GomodProjectType > Gomodules > Dynamic analysis > analysis using go mod graph > Getting selected dependencies versions using, Command {cmdName = "go", cmdArgs = ["list","-m","-json","all"], cmdAllowErr = Never}
2023-04-13 08:57:22+02 [DEBUG] [TASK 32] Project Analysis: GomodProjectType > Gomodules > Dynamic analysis > analysis using go mod graph > Getting selected dependencies versions using, Command {cmdName = "go", cmdArgs = ["list","-m","-json","all"], cmdAllowErr = Never} > Running command 'go'
2023-04-13 08:57:22+02 [DEBUG] [TASK 32] Project Analysis: GomodProjectType > Gomodules > Dynamic analysis > analysis using go mod graph > Getting selected dependencies versions using, Command {cmdName = "go", cmdArgs = ["mod","graph"], cmdAllowErr = Never}
2023-04-13 08:57:22+02 [DEBUG] [TASK 32] Project Analysis: GomodProjectType > Gomodules > Dynamic analysis > analysis using go mod graph > Getting selected dependencies versions using, Command {cmdName = "go", cmdArgs = ["mod","graph"], cmdAllowErr = Never} > Running command 'go'
2023-04-13 08:57:23+02 [DEBUG] [TASK 32] Project Analysis: GomodProjectType > Collect go standard library information
2023-04-13 08:57:23+02 [DEBUG] [TASK 32] Project Analysis: GomodProjectType > Collect go standard library information > Running command 'go'
2023-04-13 08:57:27+02 [ INFO] 
2023-04-13 08:57:27+02 [ INFO] Scan Summary
2023-04-13 08:57:27+02 [ INFO] ------------
2023-04-13 08:57:27+02 [ INFO] fossa-cli version 3.7.5 (revision 2ed6f9b02b13 compiled with ghc-9.0)
2023-04-13 08:57:27+02 [ INFO] fossa endpoint server version: 4.10.22
2023-04-13 08:57:27+02 [ INFO] 
2023-04-13 08:57:27+02 [ INFO] 2 projects scanned;  0 skipped,  2 succeeded,  0 failed,  0 analysis warnings

This is the config file used:

# Generated by FOSSA CLI (https://github.com/fossas/fossa-cli)
# Visit https://fossa.com to learn more

version: 3

server: https://app.fossa.com

project:
  id: …

paths:
  exclude:
    - deployments/terraform/.terraform

csasarak commented 1 year ago

HI Pascal,

Based on that output you're invoking the command correctly. If this is a public project, can you let me know so I can try to repro it? If not, I'll need to see our debug bundle. This is a file called fossa.debug.gz that is output when fossa is run with --debug. Can you file a ticket at https://support.fossa.com including it? Mention that I asked for it so that our support staff can send it on to me.

Thanks! -Chris

pascal-hofmann commented 1 year ago

Hi Chris, I was able to fix the issue based on the error message in the fossa.debug.json.gz. I think fossa should just bail out with the error message instead of falling back to the old resolver.

In my case generated code was missing during fossa analyze (we only generate it inside a temporary build container so it is not available during fossa analyze). The error message that was hidden in the debug logs made me aware of it.

Unused dependencies have gone down a lot, but I think test dependencies are still included. I'll have a closer look at this tomorrow.

Cheers Pascal

csasarak commented 1 year ago

Thanks for that feedback! The new go resolver looks at the build list rather than the module graph to do its work so I can see how the change in data may change the requirements. The point about bailing out rather than falling back is a good one I'll take to our team.

Do let me know if the test deps are included, keeping them out of the results is a priority for us so I want to know about any problems. For reference, we use go list -json -deps all to get data about dependencies. It may be useful to you in finding out why a particular dep was included or not.

Best wishes, Chris

pascal-hofmann commented 1 year ago

Hi Chris, I think I found one last issue with the new go resolver: Excludes are not taken into account.

We have a project where code in the test directory depends on github.com/stretchr/testify. According to the FAQ for fossa analyze this directory should be excluded automatically due to "production path filtering". I also tried filtering it explicitly via "exclusion filters" (paths.exclude: [test/]) which had no effect.

If I just delete test/ the dependencies are determined correctly.

Cheers Pascal

csasarak commented 1 year ago

Hi Pascal,

Can you clarify if there is a separate go.mod in your test directory, or is it a directory that is included in a project you're scanning? If it's the latter, this is expected behavior for path filtering.

Path filtering applies specifically to finding targets which is roughly equivalent to the idea of a project. So while path exclusions may cause fossa to skip scanning a project in a directory tree because it appears in an excluded path path, it does not cause us to exclude dependencies from code in one of those directories when referenced from a target found outside of it. More about that here.

In the case of Go, a target would be a place where there is a go.mod file. You can see a list of the detected targets by running fossa list-targets in your project directory. test/ should not appear there. paths.exclude works similarly in that it only excludes targets found at a particular path.

Regardless, the desired behavior here for the new Go strategy is for test dependencies not to be included. Could you file a ticket a https://support.fossa.com with a debug bundle? I want to see the output of go list -json -deps all especially, which should be in that file and can give me an idea of why github.com/stretchr/testify may have been included.

Thanks again for all your help here! -Chris

csasarak commented 1 year ago

Hi @pascal-hofmann,

I just wanted to check in. If you've done any additional work with the new Go strategy how has it been working for you? We're hoping to move towards using the new strategy more often but if you have feedback before then I'd like to try to address any more of your concerns.

Best wishes, -Chris

pascal-hofmann commented 1 year ago

Hi @csasarak, now that the fallback to the old strategy is disabled this works for us like a charm. I'm currently caught up in other work, but will follow up on the issue with test dependencies soon.

Cheers, Pascal

pascal-hofmann commented 1 year ago

Sorry for the late reply.

This is no issue for us anymore. I guess due to https://github.com/fossas/fossa-cli/pull/1194.

Cheers, Pascal

csasarak commented 1 year ago

That's good news, we will hopefully make this the default soon . The experimental flag will be supported for some time after though with a warning.

fossas / fossa-cli