cmd/go: `go list -m all` stuck forever

wangmir commented 1 year ago

What version of Go are you using (`go version`)?

$ go version
go version go1.20.4 darwin/arm64

Does this issue reproduce with the latest release?

yes.

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE=""
GOARCH="arm64"
GOBIN="/Users/wangmir/go/bin"
GOCACHE="/Users/wangmir/Library/Caches/go-build"
GOENV="/Users/wangmir/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/wangmir/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/wangmir/go"
GOPRIVATE=""
GOPROXY="direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.20.4"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/pl/hz15bk5d0cg7dnfqc59gw0k40000gn/T/go-build398561758=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Initially, it happens on our private project, but I realized that, it can be reproducable on one of the public module I'm using.

From this repo (at cmd/atlas directory),

https://github.com/ariga/atlas/tree/master/cmd/atlas

when I put go list -m all,

it hangs forever.

What did you expect to see?

list of modules.

What did you see instead?

nothing. stuck forever.

bcmills commented 1 year ago

I'm not able to reproduce a hang on linux/amd64 with default GOPROXY, nor with GOPROXY=direct:

/tmp$ git clone https://github.com/ariga/atlas.git
Cloning into 'atlas'...
remote: Enumerating objects: 16391, done.
remote: Counting objects: 100% (1838/1838), done.
remote: Compressing objects: 100% (325/325), done.
remote: Total 16391 (delta 1596), reused 1596 (delta 1504), pack-reused 14553
Receiving objects: 100% (16391/16391), 7.54 MiB | 31.89 MiB/s, done.
Resolving deltas: 100% (10465/10465), done.

/tmp$ cd atlas
/tmp/atlas$ git checkout 32477e01988f8004265c44e9d4c9cdeb58ad42d3
Note: switching to '32477e01988f8004265c44e9d4c9cdeb58ad42d3'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 32477e01 sql/migrate: remove Validate hack (#1465)

/tmp/atlas$ go1.20.4 list -m all
ariga.io/atlas
github.com/DATA-DOG/go-sqlmock v1.5.0
github.com/agext/levenshtein v1.2.1
github.com/apparentlymart/go-dump v0.0.0-20180507223929-23540a00eaa3
github.com/apparentlymart/go-textseg v1.0.0
github.com/apparentlymart/go-textseg/v13 v13.0.0
github.com/creack/pty v1.1.9
github.com/davecgh/go-spew v1.1.1
github.com/go-openapi/inflect v0.19.0
github.com/go-test/deep v1.0.3
github.com/golang/protobuf v1.3.4
github.com/google/go-cmp v0.5.6
github.com/hashicorp/hcl/v2 v2.10.0
github.com/kr/pretty v0.1.0
github.com/kr/pty v1.1.1
github.com/kr/text v0.2.0
github.com/kylelemons/godebug v0.0.0-20170820004349-d65d576e9348
github.com/mitchellh/go-wordwrap v0.0.0-20150314170334-ad45545899c7
github.com/pmezard/go-difflib v1.0.0
github.com/sergi/go-diff v1.0.0
github.com/spf13/pflag v1.0.2
github.com/stretchr/objx v0.1.0
github.com/stretchr/testify v1.7.1-0.20210427113832-6241f9ab9942
github.com/vmihailenco/msgpack v3.3.3+incompatible
github.com/vmihailenco/msgpack/v4 v4.3.12
github.com/vmihailenco/tagparser v0.1.1
github.com/zclconf/go-cty v1.8.0
github.com/zclconf/go-cty-debug v0.0.0-20191215020915-b22d67c1ba0b
golang.org/x/crypto v0.0.0-20190426145343-a29dc8fdc734
golang.org/x/mod v0.8.0
golang.org/x/net v0.0.0-20200301022130-244492dfa37a
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f
golang.org/x/sys v0.5.0
golang.org/x/text v0.8.0
golang.org/x/tools v0.6.0
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1
google.golang.org/appengine v1.6.5
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b

/tmp/atlas$ go clean -modcache

/tmp/atlas$ export GOPROXY=direct

/tmp/atlas$ go1.20.4 list -m all

Are you sure it is actually stuck and not just taking a long time to fetch upstream repos using git due to GOPROXY=direct?
Try sending SIGQUIT to the go command while it appears to be stuck, and please attach the goroutine dump that it produces.

bcmills commented 1 year ago

Ah, now I see that you are experiencing the problem in the cmd/atlas subdirectory.

Changing to that directory and setting GOPROXY=direct, and then using watch pgrep -a git, I see a lot of git activity fetching various large dependencies (notably google-api-go-client).

I also see that it is using a lot of older dependencies, with go.mod files declaring go versions that predate module graph pruning, which can cause a lot of extra git traffic to extract go.mod files for otherwise-irrelevant versions.

I suggest switching to the default GOPROXY (https://proxy.golang.org) and/or upgrading your dependencies to more recent releases to allow graph pruning to take effect. Perhaps also consider contributing upstream changes to enable graph pruning in the other modules you depend on; even after running go get -u ./... && go mod tidy in that module, I see an awful lot of go.sum entries for otherwise-irrelevant go.mod files.

bcmills commented 1 year ago

As far as I can tell this is an unfortunate situation where the dependency graph has become extremely large and direct fetching takes an inordinately long time.

The former problem is mitigated by graph pruning, which still needs to be enabled in certain large projects. The latter problem is mitigated by the default GOPROXY.

Beyond that, I don't think there is much we can do.

bcmills commented 1 year ago

For reference, here is the list of dependencies of that module that do not yet have graph pruning enabled:

go list -f '{{with .Module}}{{if lt .GoVersion "1.17"}}{{.Path}} {{.Version}}: go {{.GoVersion}}{{end}}{{end}}' all | sort -u

``` github.com/agext/levenshtein v1.2.3: go github.com/antlr/antlr4/runtime/Go/antlr v1.4.10: go 1.16 github.com/apparentlymart/go-textseg/v13 v13.0.0: go 1.16 github.com/auxten/postgresql-parser v1.0.1: go 1.15 github.com/aws/aws-sdk-go v1.44.269: go 1.11 github.com/aws/aws-sdk-go-v2/config v1.18.25: go 1.15 github.com/aws/aws-sdk-go-v2/credentials v1.13.24: go 1.15 github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.3: go 1.15 github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.33: go 1.15 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.27: go 1.15 github.com/aws/aws-sdk-go-v2/internal/ini v1.3.34: go 1.15 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.27: go 1.15 github.com/aws/aws-sdk-go-v2/service/secretsmanager v1.19.8: go 1.15 github.com/aws/aws-sdk-go-v2/service/ssooidc v1.14.10: go 1.15 github.com/aws/aws-sdk-go-v2/service/sso v1.12.10: go 1.15 github.com/aws/aws-sdk-go-v2/service/sts v1.19.0: go 1.15 github.com/aws/aws-sdk-go-v2 v1.18.0: go 1.15 github.com/aws/smithy-go v1.13.5: go 1.15 github.com/benbjohnson/clock v1.3.5: go 1.15 github.com/certifi/gocertifi v0.0.0-20210507211836-431795d63e8d: go 1.12 github.com/chzyer/readline v1.5.1: go 1.15 github.com/cockroachdb/apd v1.1.1-0.20181017181144-bced77f817b4: go github.com/cockroachdb/errors v1.9.1: go 1.13 github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b: go 1.16 github.com/cockroachdb/redact v1.1.4: go 1.14 github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548: go github.com/davecgh/go-spew v1.1.1: go github.com/dustin/go-humanize v1.0.1: go 1.16 github.com/fsnotify/fsnotify v1.6.0: go 1.16 github.com/getsentry/raven-go v0.2.0: go github.com/gogo/protobuf v1.3.2: go 1.15 github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da: go github.com/google/go-cmp v0.5.9: go 1.13 github.com/google/s2a-go v0.1.4: go 1.16 github.com/google/uuid v1.3.0: go github.com/google/wire v0.5.0: go 1.12 github.com/go-openapi/inflect v0.19.0: go github.com/go-sql-driver/mysql v1.7.1: go 1.13 github.com/grpc-ecosystem/grpc-gateway v1.16.0: go 1.14 github.com/jmespath/go-jmespath v0.4.0: go 1.14 github.com/kr/pretty v0.3.1: go 1.12 github.com/kr/text v0.2.0: go github.com/lib/pq v1.10.9: go 1.13 github.com/manifoldco/promptui v0.9.0: go 1.12 github.com/mattn/go-colorable v0.1.13: go 1.15 github.com/mattn/go-isatty v0.0.19: go 1.15 github.com/mattn/go-sqlite3 v1.14.16: go 1.16 github.com/mitchellh/go-homedir v1.1.0: go github.com/mitchellh/go-wordwrap v1.0.1: go 1.14 github.com/olekukonko/tablewriter v0.0.5: go 1.12 github.com/pingcap/errors v0.11.5-0.20210425183316-da1aaba5fb63: go 1.14 github.com/pingcap/log v1.1.0: go 1.16 github.com/pkg/errors v0.9.1: go github.com/pmezard/go-difflib v1.0.0: go github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec: go 1.12 github.com/sirupsen/logrus v1.9.2: go 1.13 github.com/spf13/cobra v1.7.0: go 1.15 github.com/spf13/pflag v1.0.5: go 1.12 github.com/stretchr/testify v1.8.2: go 1.13 github.com/vektah/gqlparser/v2 v2.5.1: go 1.16 gocloud.dev v0.29.0: go 1.12 golang.org/x/sync v0.2.0: go google.golang.org/appengine v1.6.7: go 1.11 google.golang.org/protobuf v1.30.0: go 1.11 go.opencensus.io v0.24.0: go 1.13 gopkg.in/natefinch/lumberjack.v2 v2.2.1: go 1.13 gopkg.in/yaml.v3 v3.0.1: go ```

golang / go