Go dependency crashes Analyzer

[qequ]

Running the analyzer in a go.mod with this specific dependency

github.com/blend/go-sdk v2.0.0+incompatible

makes ort crashes. Further investigation showed that is because the dependency format is incompatible with Go's current standard. A workaround applied was to update go.sum and go.mod by running go mod tidy

CC: @arieltorti @dgutson

[tsteenbe]

@fviernau Could we change the analyzer not to crash but simple continue the analyzer and throw an issue?

[fviernau]

Hi @qequ,

Can you please provide a minimal example for reproducing this? What exactly I'd like to receive I've outlined already in a previous bug you've been involved in too, see https://github.com/oss-review-toolkit/ort/issues/6615#issuecomment-1455735479.

[sschuberth]

Ping @qequ.

[sschuberth]

Closing this as there was no feedback from the OP in months. Feel free to reopen if new information becomes available.

[fviernau]

@sschuberth I believe that ORT should require the definition files to be in sync with what tidy would output. If we would agree this make sense, we could implement a check for that and provide a more meaningful analyzer issue, which asks the user to run tidy. What do you think? (I'm no a Go expert, so it'd need to be confirmed whether such requirement would be feasible)

[sschuberth]

What do you think?

Wouldn't it make more sense then to simply run tidy beforehand ourselves? Assuming it's a tool that's always available when go itself is.

[dgutson]

Ping @qequ.

Pinging him myself. He will come back to this, sorry about the delay.

[qequ]

@sschuberth @fviernau the go.mod that produces the crash

module github.com/path/repo

go 1.18

require (
    github.com/BurntSushi/toml v1.2.0
    github.com/PuerkitoBio/goquery v1.8.0
    github.com/SebastiaanKlippert/go-wkhtmltopdf v1.7.2
    github.com/agiledragon/gomonkey v2.0.2+incompatible
    github.com/cnkei/gospline v0.0.0-20191204052713-d67fac29a294
    github.com/gin-gonic/gin v1.8.1
    github.com/go-kit/kit v0.12.0
    github.com/go-kit/log v0.2.1
    github.com/gogo/protobuf v1.3.2
    github.com/golang/protobuf v1.5.2
    github.com/grpc-ecosystem/grpc-gateway v1.16.0
    github.com/kkyr/fig v0.3.0
    github.com/koding/multiconfig v0.0.0-20171124222453-69c27309b2d7
    github.com/krzysztofreczek/go-structurizr v0.1.32
    github.com/nicksnyder/go-i18n/v2 v2.2.1
    github.com/remeh/sizedwaitgroup v1.0.0
    github.com/rs/zerolog v1.28.0
    github.com/stretchr/testify v1.8.1
    github.com/wcharczuk/go-chart v2.0.2-0.20190219185203-9852fce5a172+incompatible
    golang.org/x/exp v0.0.0-20230105000112-eab7a2c85304
    golang.org/x/image v0.0.0-20220902085622-e7cb96979f69
    golang.org/x/net v0.0.0-20221004154528-8021a29435af
    golang.org/x/oauth2 v0.0.0-20221006150949-b44042a4b9c1
    golang.org/x/text v0.4.0
    google.golang.org/genproto v0.0.0-20221010155953-15ba04fc1c0e
    google.golang.org/grpc v1.50.0
    gopkg.in/h2non/gock.v1 v1.1.2
)

require (
    github.com/andybalholm/cascadia v1.3.1 // indirect
    github.com/blend/go-sdk v2.0.0+incompatible // indirect
    github.com/cnf/structhash v0.0.0-20201127153200-e1b16c1ebc08 // indirect
    github.com/davecgh/go-spew v1.1.1 // indirect
    github.com/fatih/camelcase v1.0.0 // indirect
    github.com/fatih/structs v1.1.0 // indirect
    github.com/gin-contrib/sse v0.1.0 // indirect
    github.com/go-logfmt/logfmt v0.5.1 // indirect
    github.com/go-playground/locales v0.14.0 // indirect
    github.com/go-playground/universal-translator v0.18.0 // indirect
    github.com/go-playground/validator/v10 v10.10.0 // indirect
    github.com/goccy/go-json v0.9.7 // indirect
    github.com/golang/freetype v0.0.0-20170609003504-e2365dfdc4a0 // indirect
    github.com/h2non/parth v0.0.0-20190131123155-b4df798d6542 // indirect
    github.com/json-iterator/go v1.1.12 // indirect
    github.com/leodido/go-urn v1.2.1 // indirect
    github.com/mattn/go-colorable v0.1.12 // indirect
    github.com/mattn/go-isatty v0.0.14 // indirect
    github.com/mitchellh/mapstructure v1.4.2 // indirect
    github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
    github.com/modern-go/reflect2 v1.0.2 // indirect
    github.com/pelletier/go-toml v1.9.3 // indirect
    github.com/pelletier/go-toml/v2 v2.0.1 // indirect
    github.com/pkg/errors v0.9.1 // indirect
    github.com/pmezard/go-difflib v1.0.0 // indirect
    github.com/stretchr/objx v0.5.0 // indirect
    github.com/ugorji/go/codec v1.2.7 // indirect
    golang.org/x/crypto v0.0.0-20220214200702-86341886e292 // indirect
    golang.org/x/sys v0.1.0 // indirect
    google.golang.org/appengine v1.6.7 // indirect
    google.golang.org/protobuf v1.28.1 // indirect
    gopkg.in/yaml.v2 v2.4.0 // indirect
    gopkg.in/yaml.v3 v3.0.1 // indirect
)

[fviernau]

Wouldn't it make more sense then to simply run tidy beforehand ourselves? Assuming it's a tool that's always available when go itself is.

In order to check whether the files are in line with tidy we would need to run tidy anyway. However, the reason why I proposed to flag an issue was, that I believe it's better to treat this as a problem the user needs to fix. So, that when GoMod is run, it produces the outputs corresponding the the actual build script, not to modified uncommitted version of it.

[fviernau]

@sschuberth @fviernau the go.mod that produces the crash

@dgutson could you attach the output of ORT as well?

[dgutson]

@sschuberth @fviernau the go.mod that produces the crash

@dgutson could you attach the output of ORT as well?

@qequ ? :)

[qequ]

@sschuberth

when running analyzer on repo;

12:20:08.197 [DefaultDispatcher-worker-1] ERROR org.ossreviewtoolkit.analyzer.PackageManager - GoMod failed to resolve dependencies for path 'go.mod': IOException: Running 'go list -m -json -buildvcs=false all' in '/home/alvaro/repo' failed with exit code 1:
go: github.com/blend/go-sdk@v2.0.0+incompatible: invalid version: unknown revision v2.0.0
go: unrecognized import path "gopkg.in/h2non/gock.v1": reading https://gopkg.in/h2non/gock.v1?go-get=1: 502 Bad Gateway
    server response: Cannot obtain refs from GitHub: cannot talk to GitHub: Get https://github.com/h2non/gock.git/info/refs?service=git-upload-pack: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
go: unrecognized import path "gopkg.in/yaml.v2": reading https://gopkg.in/yaml.v2?go-get=1: 502 Bad Gateway
    server response: Cannot obtain refs from GitHub: cannot talk to GitHub: Get https://github.com/go-yaml/yaml.git/info/refs?service=git-upload-pack: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
go: github.com/blend/go-sdk@v2.0.0+incompatible: reading github.com/blend/go-sdk/go.mod at revision v2.0.0: unknown revision v2.0.0

running with source, latest from main,

./cli/build/install/ort/bin/ort analyze -i ~/repo/ -o ~/repo/

[fviernau]

@qequ - The log says that one cannot run go list -m -json -buildvcs=false all, which is required by ORT. So, the bug is not within ORT but within the build files of the project. It can be fixed automatically by just using tidy, right?

[qequ]

@sschuberth I ran go mod tidy and keeps crashing with the same issue;

11:26:54.127 [DefaultDispatcher-worker-1] ERROR org.ossreviewtoolkit.analyzer.PackageManager - GoMod failed to resolve dependencies for path 'go.mod': IOException: Running 'go list -m -json -buildvcs=false all' in '/home/alvaro/project' failed with exit code 1:
go: github.com/blend/go-sdk@v2.0.0+incompatible: invalid version: unknown revision v2.0.0
go: github.com/blend/go-sdk@v2.0.0+incompatible: reading github.com/blend/go-sdk/go.mod at revision v2.0.0: unknown revision v2.0.0

[fviernau]

I can reproduce the issue above with GOPROXY=direct which is the setting GoMod in ORT uses. Without GOPROXY set, the issue does not reproduce.

ORT uses GOPROXY=direct, because this ensures that VCS info is available for all resolved dependencies. If proxies are used, this is not guaranteed. So, ORT relies on that.

The reason why this issue happens is because go list tries to find the git tag v2.0.0 matching the version specified in go.mod. That git tag does not exist anymore , so it must have been deleted. So, using the proxy only works because that proxy has the module cached. If the proxy cleared it cache, it would also not find the release. Note that the tag has been renamed to legacy-v2.0.0.

As a work around you can just alter the entry in go.mod (mind update the sum file):

github.com/blend/go-sdk legacy-v2.0.0

Go tooling will alter the entry to the following, which probably would be the right one to use in the first place:

require github.com/blend/go-sdk v0.0.0-20190401002810-24dca5bdfc9a

Closing this as won't fix, since the problem was caused by release tag deletion. So, ORT is actually right in not being able to find the release.

@qequ Mind trying the above work around and leaving some feedback below?

[qequ]

@fviernau tested it and updated the dependency with

require github.com/blend/go-sdk v0.0.0-20190401002810-24dca5bdfc9a // indirect

and the analyzer produced the output correctly :+1:

[qequ]

@dgutson @arieltorti FYI