search

golang / go

The Go programming language
https://go.dev
BSD 3-Clause "New" or "Revised" License
124.23k stars 17.7k forks source link

x/exp/cmd/gorelease: hangs #56840

Open benmathews opened 2 years ago

benmathews commented 2 years ago

What version of Go are you using (go version)?

$ go version
go version go1.19.3 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ben/.cache/go-build"
GOENV="/home/ben/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/ben/.gvm/pkgsets/go1.19.3/global/pkg/mod"
GONOPROXY="source.vivint.com,gitlab.com/vivint"
GONOSUMDB="source.vivint.com,gitlab.com/vivint"
GOOS="linux"
GOPATH="/home/ben/.gvm/pkgsets/go1.19.3/global"
GOPRIVATE="source.vivint.com,gitlab.com/vivint"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/ben/.gvm/gos/go1.19.3"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/ben/.gvm/gos/go1.19.3/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.19.3"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/ben/vivint/PL/platformusers/go.mod"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1894158366=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Running gorelease on most of our internal software takes 3-5 minutes and works as expected. On one particular repo it will hang for an hour with no response. I recognize that this gives no information to solve the issue. Is there a way to get debug output that would be helpful?

bcmills commented 2 years ago

Send the stuck command SIGQUIT to obtain a goroutine dump.

benmathews commented 2 years ago 
➜ gorelease
^\SIGQUIT: quit
PC=0x465361 m=0 sigcode=128

goroutine 0 [idle]:
runtime.futex()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/sys_linux_amd64.s:559 +0x21 fp=0x7ffc57396f40 sp=0x7ffc57396f38 pc=0x465361
runtime.futexsleep(0x100000162ff?, 0x57396fc0?, 0x44322b?)
    /home/ben/.gvm/gos/go1.19.3/src/runtime/os_linux.go:69 +0x36 fp=0x7ffc57396f90 sp=0x7ffc57396f40 pc=0x430896
runtime.notesleep(0x893528)
    /home/ben/.gvm/gos/go1.19.3/src/runtime/lock_futex.go:160 +0x87 fp=0x7ffc57396fc8 sp=0x7ffc57396f90 pc=0x40b4c7
runtime.mPark(...)
    /home/ben/.gvm/gos/go1.19.3/src/runtime/proc.go:1457
runtime.stopm()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/proc.go:2247 +0x8c fp=0x7ffc57396ff8 sp=0x7ffc57396fc8 pc=0x43af6c
runtime.findRunnable()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/proc.go:2874 +0x9e8 fp=0x7ffc573970e8 sp=0x7ffc57396ff8 pc=0x43c608
runtime.schedule()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/proc.go:3214 +0xbe fp=0x7ffc57397120 sp=0x7ffc573970e8 pc=0x43d43e
runtime.park_m(0xc0005029c0?)
    /home/ben/.gvm/gos/go1.19.3/src/runtime/proc.go:3363 +0x14d fp=0x7ffc57397150 sp=0x7ffc57397120 pc=0x43d96d
runtime.mcall()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/asm_amd64.s:448 +0x43 fp=0x7ffc57397160 sp=0x7ffc57397150 pc=0x461383

goroutine 1 [syscall, 5 minutes]:
syscall.Syscall6(0x4?, 0x4?, 0x0?, 0xc0000dc598?, 0x49a4fa?, 0x0?, 0x85?)
    /home/ben/.gvm/gos/go1.19.3/src/syscall/syscall_linux.go:90 +0x36 fp=0xc0000dc508 sp=0xc0000dc480 pc=0x49a856
os.(*Process).blockUntilWaitable(0xc00045c900)
    /home/ben/.gvm/gos/go1.19.3/src/os/wait_waitid.go:32 +0x87 fp=0xc0000dc5e0 sp=0xc0000dc508 pc=0x4b5a87
os.(*Process).wait(0xc00045c900)
    /home/ben/.gvm/gos/go1.19.3/src/os/exec_unix.go:22 +0x28 fp=0xc0000dc640 sp=0xc0000dc5e0 pc=0x4b0c68
os.(*Process).Wait(...)
    /home/ben/.gvm/gos/go1.19.3/src/os/exec.go:132
os/exec.(*Cmd).Wait(0xc0000de580)
    /home/ben/.gvm/gos/go1.19.3/src/os/exec/exec.go:599 +0x4b fp=0xc0000dc6b8 sp=0xc0000dc640 pc=0x4d6b0b
os/exec.(*Cmd).Run(0xc000017900?)
    /home/ben/.gvm/gos/go1.19.3/src/os/exec/exec.go:437 +0x39 fp=0xc0000dc6d8 sp=0xc0000dc6b8 pc=0x4d5d19
os/exec.(*Cmd).Output(0xc0000de580)
    /home/ben/.gvm/gos/go1.19.3/src/os/exec/exec.go:686 +0xc7 fp=0xc0000dc728 sp=0xc0000dc6d8 pc=0x4d7187
main.prepareLoadDir({0x7276b8?, 0xc000094960}, 0xc00002e580, {0xc00001e4b0, 0x22}, {0xc00001c5b8, 0x17}, {0x6c4984, 0x10}, 0x0)
    /home/ben/.gvm/pkgsets/go1.19.3/global/pkg/mod/golang.org/x/exp@v0.0.0-20221114191408-850992195362/cmd/gorelease/gorelease.go:1129 +0xd25 fp=0xc0000dcc38 sp=0xc0000dc728 pc=0x6540c5
main.loadLocalModule({_, _}, {_, _}, {_, _}, {_, _})
    /home/ben/.gvm/pkgsets/go1.19.3/global/pkg/mod/golang.org/x/exp@v0.0.0-20221114191408-850992195362/cmd/gorelease/gorelease.go:398 +0x102b fp=0xc0000dcff0 sp=0xc0000dcc38 pc=0x64d14b
main.runRelease({0x7276b8, 0xc000094960}, {0x725380, 0xc000012018}, {0xc00001e124, 0x21}, {0xc000014220, 0x0, 0x0})
    /home/ben/.gvm/pkgsets/go1.19.3/global/pkg/mod/golang.org/x/exp@v0.0.0-20221114191408-850992195362/cmd/gorelease/gorelease.go:218 +0x92d fp=0xc0000ddef0 sp=0xc0000dcff0 pc=0x64b7cd
main.main()
    /home/ben/.gvm/pkgsets/go1.19.3/global/pkg/mod/golang.org/x/exp@v0.0.0-20221114191408-850992195362/cmd/gorelease/gorelease.go:146 +0x18d fp=0xc0000ddf80 sp=0xc0000ddef0 pc=0x64adad
runtime.main()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/proc.go:250 +0x212 fp=0xc0000ddfe0 sp=0xc0000ddf80 pc=0x436d92
runtime.goexit()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000ddfe8 sp=0xc0000ddfe0 pc=0x463481

goroutine 2 [force gc (idle), 3 minutes]:
runtime.gopark(0x52a7973f1c99?, 0x0?, 0x0?, 0x0?, 0x0?)
    /home/ben/.gvm/gos/go1.19.3/src/runtime/proc.go:363 +0xd6 fp=0xc00005efb0 sp=0xc00005ef90 pc=0x437156
runtime.goparkunlock(...)
    /home/ben/.gvm/gos/go1.19.3/src/runtime/proc.go:369
runtime.forcegchelper()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/proc.go:302 +0xad fp=0xc00005efe0 sp=0xc00005efb0 pc=0x436fed
runtime.goexit()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00005efe8 sp=0xc00005efe0 pc=0x463481
created by runtime.init.6
    /home/ben/.gvm/gos/go1.19.3/src/runtime/proc.go:290 +0x25

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
    /home/ben/.gvm/gos/go1.19.3/src/runtime/proc.go:363 +0xd6 fp=0xc00005f790 sp=0xc00005f770 pc=0x437156
runtime.goparkunlock(...)
    /home/ben/.gvm/gos/go1.19.3/src/runtime/proc.go:369
runtime.bgsweep(0x0?)
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgcsweep.go:297 +0xd7 fp=0xc00005f7c8 sp=0xc00005f790 pc=0x424037
runtime.gcenable.func1()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgc.go:178 +0x26 fp=0xc00005f7e0 sp=0xc00005f7c8 pc=0x418ea6
runtime.goexit()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00005f7e8 sp=0xc00005f7e0 pc=0x463481
created by runtime.gcenable
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgc.go:178 +0x6b

goroutine 4 [GC scavenge wait]:
runtime.gopark(0x52b58f9df38a?, 0x11cc52e2?, 0x0?, 0x0?, 0x0?)
    /home/ben/.gvm/gos/go1.19.3/src/runtime/proc.go:363 +0xd6 fp=0xc00005ff70 sp=0xc00005ff50 pc=0x437156
runtime.goparkunlock(...)
    /home/ben/.gvm/gos/go1.19.3/src/runtime/proc.go:369
runtime.(*scavengerState).park(0x8929a0)
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgcscavenge.go:389 +0x53 fp=0xc00005ffa0 sp=0xc00005ff70 pc=0x422093
runtime.bgscavenge(0x0?)
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgcscavenge.go:622 +0x65 fp=0xc00005ffc8 sp=0xc00005ffa0 pc=0x422685
runtime.gcenable.func2()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgc.go:179 +0x26 fp=0xc00005ffe0 sp=0xc00005ffc8 pc=0x418e46
runtime.goexit()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00005ffe8 sp=0xc00005ffe0 pc=0x463481
created by runtime.gcenable
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgc.go:179 +0xaa

goroutine 5 [finalizer wait, 5 minutes]:
runtime.gopark(0x0?, 0x6d5d78?, 0xe0?, 0x65?, 0x2000000020?)
    /home/ben/.gvm/gos/go1.19.3/src/runtime/proc.go:363 +0xd6 fp=0xc00005e628 sp=0xc00005e608 pc=0x437156
runtime.goparkunlock(...)
    /home/ben/.gvm/gos/go1.19.3/src/runtime/proc.go:369
runtime.runfinq()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mfinal.go:180 +0x10f fp=0xc00005e7e0 sp=0xc00005e628 pc=0x417faf
runtime.goexit()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00005e7e8 sp=0xc00005e7e0 pc=0x463481
created by runtime.createfing
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mfinal.go:157 +0x45

goroutine 19 [GC worker (idle), 5 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /home/ben/.gvm/gos/go1.19.3/src/runtime/proc.go:363 +0xd6 fp=0xc000154750 sp=0xc000154730 pc=0x437156
runtime.gcBgMarkWorker()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgc.go:1235 +0xf1 fp=0xc0001547e0 sp=0xc000154750 pc=0x41adf1
runtime.goexit()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0001547e8 sp=0xc0001547e0 pc=0x463481
created by runtime.gcBgMarkStartWorkers
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgc.go:1159 +0x25

goroutine 33 [GC worker (idle), 5 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0xc000134020?)
    /home/ben/.gvm/gos/go1.19.3/src/runtime/proc.go:363 +0xd6 fp=0xc000060750 sp=0xc000060730 pc=0x437156
runtime.gcBgMarkWorker()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgc.go:1235 +0xf1 fp=0xc0000607e0 sp=0xc000060750 pc=0x41adf1
runtime.goexit()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000607e8 sp=0xc0000607e0 pc=0x463481
created by runtime.gcBgMarkStartWorkers
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgc.go:1159 +0x25

goroutine 34 [GC worker (idle), 5 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /home/ben/.gvm/gos/go1.19.3/src/runtime/proc.go:363 +0xd6 fp=0xc00005a750 sp=0xc00005a730 pc=0x437156
runtime.gcBgMarkWorker()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgc.go:1235 +0xf1 fp=0xc00005a7e0 sp=0xc00005a750 pc=0x41adf1
runtime.goexit()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00005a7e8 sp=0xc00005a7e0 pc=0x463481
created by runtime.gcBgMarkStartWorkers
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgc.go:1159 +0x25

goroutine 8 [GC worker (idle), 5 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0xc000134008?)
    /home/ben/.gvm/gos/go1.19.3/src/runtime/proc.go:363 +0xd6 fp=0xc000060f50 sp=0xc000060f30 pc=0x437156
runtime.gcBgMarkWorker()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgc.go:1235 +0xf1 fp=0xc000060fe0 sp=0xc000060f50 pc=0x41adf1
runtime.goexit()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000060fe8 sp=0xc000060fe0 pc=0x463481
created by runtime.gcBgMarkStartWorkers
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgc.go:1159 +0x25

goroutine 9 [GC worker (idle), 5 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /home/ben/.gvm/gos/go1.19.3/src/runtime/proc.go:363 +0xd6 fp=0xc000061750 sp=0xc000061730 pc=0x437156
runtime.gcBgMarkWorker()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgc.go:1235 +0xf1 fp=0xc0000617e0 sp=0xc000061750 pc=0x41adf1
runtime.goexit()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000617e8 sp=0xc0000617e0 pc=0x463481
created by runtime.gcBgMarkStartWorkers
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgc.go:1159 +0x25

goroutine 20 [GC worker (idle), 5 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /home/ben/.gvm/gos/go1.19.3/src/runtime/proc.go:363 +0xd6 fp=0xc000154f50 sp=0xc000154f30 pc=0x437156
runtime.gcBgMarkWorker()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgc.go:1235 +0xf1 fp=0xc000154fe0 sp=0xc000154f50 pc=0x41adf1
runtime.goexit()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000154fe8 sp=0xc000154fe0 pc=0x463481
created by runtime.gcBgMarkStartWorkers
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgc.go:1159 +0x25

goroutine 21 [GC worker (idle), 5 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /home/ben/.gvm/gos/go1.19.3/src/runtime/proc.go:363 +0xd6 fp=0xc000155750 sp=0xc000155730 pc=0x437156
runtime.gcBgMarkWorker()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgc.go:1235 +0xf1 fp=0xc0001557e0 sp=0xc000155750 pc=0x41adf1
runtime.goexit()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0001557e8 sp=0xc0001557e0 pc=0x463481
created by runtime.gcBgMarkStartWorkers
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgc.go:1159 +0x25

goroutine 22 [GC worker (idle), 5 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /home/ben/.gvm/gos/go1.19.3/src/runtime/proc.go:363 +0xd6 fp=0xc000155f50 sp=0xc000155f30 pc=0x437156
runtime.gcBgMarkWorker()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgc.go:1235 +0xf1 fp=0xc000155fe0 sp=0xc000155f50 pc=0x41adf1
runtime.goexit()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000155fe8 sp=0xc000155fe0 pc=0x463481
created by runtime.gcBgMarkStartWorkers
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgc.go:1159 +0x25

goroutine 23 [GC worker (idle), 5 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /home/ben/.gvm/gos/go1.19.3/src/runtime/proc.go:363 +0xd6 fp=0xc000156750 sp=0xc000156730 pc=0x437156
runtime.gcBgMarkWorker()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgc.go:1235 +0xf1 fp=0xc0001567e0 sp=0xc000156750 pc=0x41adf1
runtime.goexit()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0001567e8 sp=0xc0001567e0 pc=0x463481
created by runtime.gcBgMarkStartWorkers
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgc.go:1159 +0x25

goroutine 49 [GC worker (idle), 5 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /home/ben/.gvm/gos/go1.19.3/src/runtime/proc.go:363 +0xd6 fp=0xc000150750 sp=0xc000150730 pc=0x437156
runtime.gcBgMarkWorker()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgc.go:1235 +0xf1 fp=0xc0001507e0 sp=0xc000150750 pc=0x41adf1
runtime.goexit()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0001507e8 sp=0xc0001507e0 pc=0x463481
created by runtime.gcBgMarkStartWorkers
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgc.go:1159 +0x25

goroutine 50 [GC worker (idle), 5 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /home/ben/.gvm/gos/go1.19.3/src/runtime/proc.go:363 +0xd6 fp=0xc000150f50 sp=0xc000150f30 pc=0x437156
runtime.gcBgMarkWorker()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgc.go:1235 +0xf1 fp=0xc000150fe0 sp=0xc000150f50 pc=0x41adf1
runtime.goexit()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000150fe8 sp=0xc000150fe0 pc=0x463481
created by runtime.gcBgMarkStartWorkers
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgc.go:1159 +0x25

goroutine 51 [GC worker (idle), 5 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /home/ben/.gvm/gos/go1.19.3/src/runtime/proc.go:363 +0xd6 fp=0xc000151750 sp=0xc000151730 pc=0x437156
runtime.gcBgMarkWorker()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgc.go:1235 +0xf1 fp=0xc0001517e0 sp=0xc000151750 pc=0x41adf1
runtime.goexit()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0001517e8 sp=0xc0001517e0 pc=0x463481
created by runtime.gcBgMarkStartWorkers
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgc.go:1159 +0x25

goroutine 52 [GC worker (idle)]:
runtime.gopark(0x52a7974cc50e?, 0x1?, 0xfb?, 0x64?, 0x0?)
    /home/ben/.gvm/gos/go1.19.3/src/runtime/proc.go:363 +0xd6 fp=0xc000151f50 sp=0xc000151f30 pc=0x437156
runtime.gcBgMarkWorker()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgc.go:1235 +0xf1 fp=0xc000151fe0 sp=0xc000151f50 pc=0x41adf1
runtime.goexit()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000151fe8 sp=0xc000151fe0 pc=0x463481
created by runtime.gcBgMarkStartWorkers
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgc.go:1159 +0x25

goroutine 53 [GC worker (idle), 3 minutes]:
runtime.gopark(0x52a7974d3bac?, 0x1?, 0xfe?, 0x2f?, 0x0?)
    /home/ben/.gvm/gos/go1.19.3/src/runtime/proc.go:363 +0xd6 fp=0xc000152750 sp=0xc000152730 pc=0x437156
runtime.gcBgMarkWorker()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgc.go:1235 +0xf1 fp=0xc0001527e0 sp=0xc000152750 pc=0x41adf1
runtime.goexit()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0001527e8 sp=0xc0001527e0 pc=0x463481
created by runtime.gcBgMarkStartWorkers
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgc.go:1159 +0x25

goroutine 54 [GC worker (idle)]:
runtime.gopark(0x8c3700?, 0x1?, 0xf5?, 0xe6?, 0x0?)
    /home/ben/.gvm/gos/go1.19.3/src/runtime/proc.go:363 +0xd6 fp=0xc000152f50 sp=0xc000152f30 pc=0x437156
runtime.gcBgMarkWorker()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgc.go:1235 +0xf1 fp=0xc000152fe0 sp=0xc000152f50 pc=0x41adf1
runtime.goexit()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000152fe8 sp=0xc000152fe0 pc=0x463481
created by runtime.gcBgMarkStartWorkers
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgc.go:1159 +0x25

goroutine 55 [GC worker (idle)]:
runtime.gopark(0x52a7974cc848?, 0x1?, 0xe3?, 0xba?, 0x0?)
    /home/ben/.gvm/gos/go1.19.3/src/runtime/proc.go:363 +0xd6 fp=0xc000153750 sp=0xc000153730 pc=0x437156
runtime.gcBgMarkWorker()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgc.go:1235 +0xf1 fp=0xc0001537e0 sp=0xc000153750 pc=0x41adf1
runtime.goexit()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0001537e8 sp=0xc0001537e0 pc=0x463481
created by runtime.gcBgMarkStartWorkers
    /home/ben/.gvm/gos/go1.19.3/src/runtime/mgc.go:1159 +0x25

goroutine 10 [IO wait, 5 minutes]:
runtime.gopark(0x7fd65c4bea68?, 0xb?, 0x0?, 0x0?, 0x7?)
    /home/ben/.gvm/gos/go1.19.3/src/runtime/proc.go:363 +0xd6 fp=0xc0004eed00 sp=0xc0004eece0 pc=0x437156
runtime.netpollblock(0x499ba5?, 0x200?, 0x0?)
    /home/ben/.gvm/gos/go1.19.3/src/runtime/netpoll.go:526 +0xf7 fp=0xc0004eed38 sp=0xc0004eed00 pc=0x4300b7
internal/poll.runtime_pollWait(0x7fd634ff3578, 0x72)
    /home/ben/.gvm/gos/go1.19.3/src/runtime/netpoll.go:305 +0x89 fp=0xc0004eed58 sp=0xc0004eed38 pc=0x45eea9
internal/poll.(*pollDesc).wait(0xc0004807e0?, 0xc0004f2000?, 0x1)
    /home/ben/.gvm/gos/go1.19.3/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc0004eed80 sp=0xc0004eed58 pc=0x4ad452
internal/poll.(*pollDesc).waitRead(...)
    /home/ben/.gvm/gos/go1.19.3/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0004807e0, {0xc0004f2000, 0x200, 0x200})
    /home/ben/.gvm/gos/go1.19.3/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc0004eee00 sp=0xc0004eed80 pc=0x4adbda
os.(*File).read(...)
    /home/ben/.gvm/gos/go1.19.3/src/os/file_posix.go:31
os.(*File).Read(0xc000012120, {0xc0004f2000?, 0x0?, 0xc00005cea0?})
    /home/ben/.gvm/gos/go1.19.3/src/os/file.go:118 +0x5e fp=0xc0004eee58 sp=0xc0004eee00 pc=0x4b11fe
bytes.(*Buffer).ReadFrom(0xc000476870, {0x725340, 0xc000012120})
    /home/ben/.gvm/gos/go1.19.3/src/bytes/buffer.go:202 +0x98 fp=0xc0004eeeb0 sp=0xc0004eee58 pc=0x4cde18
io.copyBuffer({0x724a60, 0xc000476870}, {0x725340, 0xc000012120}, {0x0, 0x0, 0x0})
    /home/ben/.gvm/gos/go1.19.3/src/io/io.go:413 +0x14b fp=0xc0004eef30 sp=0xc0004eeeb0 pc=0x49474b
io.Copy(...)
    /home/ben/.gvm/gos/go1.19.3/src/io/io.go:386
os/exec.(*Cmd).writerDescriptor.func1()
    /home/ben/.gvm/gos/go1.19.3/src/os/exec/exec.go:407 +0x3a fp=0xc0004eef90 sp=0xc0004eef30 pc=0x4d5c9a
os/exec.(*Cmd).Start.func1(0x0?)
    /home/ben/.gvm/gos/go1.19.3/src/os/exec/exec.go:544 +0x25 fp=0xc0004eefc8 sp=0xc0004eef90 pc=0x4d6a45
os/exec.(*Cmd).Start.func2()
    /home/ben/.gvm/gos/go1.19.3/src/os/exec/exec.go:545 +0x2a fp=0xc0004eefe0 sp=0xc0004eefc8 pc=0x4d69ea
runtime.goexit()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0004eefe8 sp=0xc0004eefe0 pc=0x463481
created by os/exec.(*Cmd).Start
    /home/ben/.gvm/gos/go1.19.3/src/os/exec/exec.go:543 +0x73e

goroutine 11 [runnable]:
runtime.asyncPreempt2()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/preempt.go:308 +0x3f fp=0xc00005d528 sp=0xc00005d508 pc=0x43587f
runtime.asyncPreempt()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/preempt_amd64.s:50 +0xdb fp=0xc00005d6b0 sp=0xc00005d528 pc=0x464abb
io.copyBuffer({0x725420, 0xc0005120a0}, {0x725340, 0xc000012138}, {0x0, 0x0, 0x0})
    /home/ben/.gvm/gos/go1.19.3/src/io/io.go:429 +0x1ea fp=0xc00005d730 sp=0xc00005d6b0 pc=0x4947ea
io.Copy(...)
    /home/ben/.gvm/gos/go1.19.3/src/io/io.go:386
os/exec.(*Cmd).writerDescriptor.func1()
    /home/ben/.gvm/gos/go1.19.3/src/os/exec/exec.go:407 +0x3a fp=0xc00005d790 sp=0xc00005d730 pc=0x4d5c9a
os/exec.(*Cmd).Start.func1(0x0?)
    /home/ben/.gvm/gos/go1.19.3/src/os/exec/exec.go:544 +0x25 fp=0xc00005d7c8 sp=0xc00005d790 pc=0x4d6a45
os/exec.(*Cmd).Start.func2()
    /home/ben/.gvm/gos/go1.19.3/src/os/exec/exec.go:545 +0x2a fp=0xc00005d7e0 sp=0xc00005d7c8 pc=0x4d69ea
runtime.goexit()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00005d7e8 sp=0xc00005d7e0 pc=0x463481
created by os/exec.(*Cmd).Start
    /home/ben/.gvm/gos/go1.19.3/src/os/exec/exec.go:543 +0x73e

goroutine 12 [select, 5 minutes]:
runtime.gopark(0xc00005df90?, 0x2?, 0x0?, 0x0?, 0xc00005df64?)
    /home/ben/.gvm/gos/go1.19.3/src/runtime/proc.go:363 +0xd6 fp=0xc00005dde8 sp=0xc00005ddc8 pc=0x437156
runtime.selectgo(0xc00005df90, 0xc00005df60, 0x0?, 0x1, 0x0?, 0x1)
    /home/ben/.gvm/gos/go1.19.3/src/runtime/select.go:328 +0x7bc fp=0xc00005df28 sp=0xc00005dde8 pc=0x4464dc
os/exec.(*Cmd).watchCtx.func1()
    /home/ben/.gvm/gos/go1.19.3/src/os/exec/exec.go:648 +0x96 fp=0xc00005dfe0 sp=0xc00005df28 pc=0x4d6f76
runtime.goexit()
    /home/ben/.gvm/gos/go1.19.3/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00005dfe8 sp=0xc00005dfe0 pc=0x463481
created by os/exec.(*Cmd).watchCtx
    /home/ben/.gvm/gos/go1.19.3/src/os/exec/exec.go:647 +0x8e

rax    0xca
rbx    0x0
rcx    0x465363
rdx    0x0
rdi    0x893528
rsi    0x80
rbp    0x7ffc57396f80
rsp    0x7ffc57396f38
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x1
r13    0x1
r14    0x892cc0
r15    0x0
rip    0x465361
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
jasonphi commented 2 years ago

I attached a debugger to the running gorelease application when this is happening. It's spinning finding missing dependencies here: https://github.com/golang/exp/blob/master/cmd/gorelease/gorelease.go#L1125-L1131

jasonphi commented 2 years ago

Here's a stack trace for the go get -d command:

PC=0x469f01 m=0 sigcode=128

goroutine 0 [idle]:
runtime.futex()
        /usr/local/go/src/runtime/sys_linux_amd64.s:559 +0x21 fp=0x7ffe82194a90 sp=0x7ffe82194a88 pc=0x469f01
runtime.futexsleep(0x7ffe82194b10?, 0x82194b10?, 0x4463eb?)
        /usr/local/go/src/runtime/os_linux.go:69 +0x36 fp=0x7ffe82194ae0 sp=0x7ffe82194a90 pc=0x433976
runtime.notesleep(0xe36f28)
        /usr/local/go/src/runtime/lock_futex.go:160 +0x87 fp=0x7ffe82194b18 sp=0x7ffe82194ae0 pc=0x40d6e7
runtime.mPark(...)
        /usr/local/go/src/runtime/proc.go:1457
runtime.stopm()
        /usr/local/go/src/runtime/proc.go:2247 +0x8c fp=0x7ffe82194b48 sp=0x7ffe82194b18 pc=0x43e04c
runtime.findRunnable()
        /usr/local/go/src/runtime/proc.go:2874 +0x9e8 fp=0x7ffe82194c38 sp=0x7ffe82194b48 pc=0x43f6e8
runtime.schedule()
        /usr/local/go/src/runtime/proc.go:3214 +0xbe fp=0x7ffe82194c70 sp=0x7ffe82194c38 pc=0x44051e
runtime.park_m(0xc0005caea0?)
        /usr/local/go/src/runtime/proc.go:3363 +0x14d fp=0x7ffe82194ca0 sp=0x7ffe82194c70 pc=0x440a4d
runtime.mcall()
        /usr/local/go/src/runtime/asm_amd64.s:448 +0x43 fp=0x7ffe82194cb0 sp=0x7ffe82194ca0 pc=0x465de3

goroutine 1 [chan receive]:
runtime.gopark(0xc000624360?, 0xc001abaf68?, 0xf4?, 0xb6?, 0x7eecf2?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc001abaec0 sp=0xc001abaea0 pc=0x43a236
runtime.chanrecv(0xc000624360, 0x0, 0x1)
        /usr/local/go/src/runtime/chan.go:583 +0x49b fp=0xc001abaf50 sp=0xc001abaec0 pc=0x4085fb
runtime.chanrecv1(0xc000fcd9d0?, 0xc00002c6c0?)
        /usr/local/go/src/runtime/chan.go:442 +0x18 fp=0xc001abaf78 sp=0xc001abaf50 pc=0x4080f8
cmd/go/internal/modload.readModGraph({0xc001abb590?, 0xb2e690?}, 0x0, {0xc00209c000?, 0x6a, 0x6b})
        /usr/local/go/src/cmd/go/internal/modload/buildlist.go:387 +0x354 fp=0xc001abb4a8 sp=0xc001abaf78 pc=0x7ee354
cmd/go/internal/modload.selectPotentiallyImportedModules({0xb2e690, 0xc000028088}, 0xc000425290, 0xc000f50000, {0xc001302b80?, 0x1?, 0xc001302be0?})
        /usr/local/go/src/cmd/go/internal/modload/edit.go:395 +0x4ff fp=0xc001abb600 sp=0xc001abb4a8 pc=0x7f721f
cmd/go/internal/modload.editRequirements({0xb2e690, 0xc000028088}, 0xc000f50000, {0xc001302b80, 0x1, 0x1}, {0xc001302be0, 0x0, 0xc0003fe420?})
        /usr/local/go/src/cmd/go/internal/modload/edit.go:66 +0x194 fp=0xc001abb798 sp=0xc001abb600 pc=0x7f4934
cmd/go/internal/modload.EditBuildList({0xb2e690, 0xc000028088}, {0xc001302b80, 0x1, 0x1}, {0xc001302be0, 0x0, 0x1})
        /usr/local/go/src/cmd/go/internal/modload/buildlist.go:603 +0x87 fp=0xc001abb7f0 sp=0xc001abb798 pc=0x7efc27
cmd/go/internal/modget.(*resolver).updateBuildList(0xc000842d00, {0xb2e690, 0xc000028088}, {0xc001302b80, 0x1, 0x1})
        /usr/local/go/src/cmd/go/internal/modget/get.go:1749 +0x2a8 fp=0xc001abb9f8 sp=0xc001abb7f0 pc=0x91f8c8
cmd/go/internal/modget.(*resolver).applyUpgrades(0xc000842d00, {0xb2e690, 0xc000028088}, {0xc00089d500?, 0x1?, 0x0?})
        /usr/local/go/src/cmd/go/internal/modget/get.go:1312 +0x105 fp=0xc001abbcc0 sp=0xc001abb9f8 pc=0x91b905
cmd/go/internal/modget.runGet({0xb2e690, 0xc000028088}, 0xc00002c618?, {0xc0000240b0, 0x1, 0x1})
        /usr/local/go/src/cmd/go/internal/modget/get.go:351 +0x45e fp=0xc001abbd88 sp=0xc001abbcc0 pc=0x913f9e
main.invoke(0xe28fa0, {0xc000024090, 0x3, 0x3})
        /usr/local/go/src/cmd/go/main.go:225 +0x34e fp=0xc001abbe60 sp=0xc001abbd88 pc=0x940cce
main.main()
        /usr/local/go/src/cmd/go/main.go:179 +0x7d1 fp=0xc001abbf80 sp=0xc001abbe60 pc=0x940651
runtime.main()
        /usr/local/go/src/runtime/proc.go:250 +0x212 fp=0xc001abbfe0 sp=0xc001abbf80 pc=0x439e72
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc001abbfe8 sp=0xc001abbfe0 pc=0x468021

goroutine 2 [force gc (idle), 15 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00006cfb0 sp=0xc00006cf90 pc=0x43a236
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:369
runtime.forcegchelper()
        /usr/local/go/src/runtime/proc.go:302 +0xad fp=0xc00006cfe0 sp=0xc00006cfb0 pc=0x43a0cd
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00006cfe8 sp=0xc00006cfe0 pc=0x468021
created by runtime.init.6
        /usr/local/go/src/runtime/proc.go:290 +0x25

goroutine 3 [GC sweep wait]:
runtime.gopark(0xe35101?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00006d790 sp=0xc00006d770 pc=0x43a236
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:369
runtime.bgsweep(0x0?)
        /usr/local/go/src/runtime/mgcsweep.go:297 +0xd7 fp=0xc00006d7c8 sp=0xc00006d790 pc=0x426917
runtime.gcenable.func1()
        /usr/local/go/src/runtime/mgc.go:178 +0x26 fp=0xc00006d7e0 sp=0xc00006d7c8 pc=0x41b586
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00006d7e8 sp=0xc00006d7e0 pc=0x468021
created by runtime.gcenable
        /usr/local/go/src/runtime/mgc.go:178 +0x6b

goroutine 4 [GC scavenge wait]:
runtime.gopark(0x155175429c39?, 0x4e7fa6?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00006df70 sp=0xc00006df50 pc=0x43a236
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:369
runtime.(*scavengerState).park(0xe35680)
        /usr/local/go/src/runtime/mgcscavenge.go:389 +0x53 fp=0xc00006dfa0 sp=0xc00006df70 pc=0x424973
runtime.bgscavenge(0x0?)
        /usr/local/go/src/runtime/mgcscavenge.go:622 +0x65 fp=0xc00006dfc8 sp=0xc00006dfa0 pc=0x424f65
runtime.gcenable.func2()
        /usr/local/go/src/runtime/mgc.go:179 +0x26 fp=0xc00006dfe0 sp=0xc00006dfc8 pc=0x41b526
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00006dfe8 sp=0xc00006dfe0 pc=0x468021
created by runtime.gcenable
        /usr/local/go/src/runtime/mgc.go:179 +0xaa

goroutine 5 [finalizer wait, 14 minutes]:
runtime.gopark(0x0?, 0xa8a9d0?, 0x0?, 0xa0?, 0x2000000020?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00006c628 sp=0xc00006c608 pc=0x43a236
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:369
runtime.runfinq()
        /usr/local/go/src/runtime/mfinal.go:180 +0x10f fp=0xc00006c7e0 sp=0xc00006c628 pc=0x41a68f
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00006c7e8 sp=0xc00006c7e0 pc=0x468021
created by runtime.createfing
        /usr/local/go/src/runtime/mfinal.go:157 +0x45

goroutine 91 [GC worker (idle)]:
runtime.gopark(0xe67800?, 0x3?, 0x64?, 0x42?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00006bf50 sp=0xc00006bf30 pc=0x43a236
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00006bfe0 sp=0xc00006bf50 pc=0x41d6d1
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0x468021
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 200 [GC worker (idle)]:
runtime.gopark(0x155177418e52?, 0x3?, 0xc5?, 0xd?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00006ef50 sp=0xc00006ef30 pc=0x43a236
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00006efe0 sp=0xc00006ef50 pc=0x41d6d1
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00006efe8 sp=0xc00006efe0 pc=0x468021
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 103 [GC worker (idle)]:
runtime.gopark(0x155177417e51?, 0x3?, 0xd7?, 0x29?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0001fd750 sp=0xc0001fd730 pc=0x43a236
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc0001fd7e0 sp=0xc0001fd750 pc=0x41d6d1
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0001fd7e8 sp=0xc0001fd7e0 pc=0x468021
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 135 [GC worker (idle)]:
runtime.gopark(0x155177418b26?, 0x3?, 0x8e?, 0xb?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00006ff50 sp=0xc00006ff30 pc=0x43a236
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00006ffe0 sp=0xc00006ff50 pc=0x41d6d1
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00006ffe8 sp=0xc00006ffe0 pc=0x468021
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 6473431 [chan receive]:
runtime.gopark(0x0?, 0xc0009f6d88?, 0xdf?, 0xee?, 0x8?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0009f6d08 sp=0xc0009f6ce8 pc=0x43a236
runtime.chanrecv(0xc000214780, 0xc0009f6e30, 0x1)
        /usr/local/go/src/runtime/chan.go:583 +0x49b fp=0xc0009f6d98 sp=0xc0009f6d08 pc=0x4085fb
runtime.chanrecv1(0x40eedf?, 0x99b180?)
        /usr/local/go/src/runtime/chan.go:442 +0x18 fp=0xc0009f6dc0 sp=0xc0009f6d98 pc=0x4080f8
cmd/go/internal/par.(*Queue).Add(0xc000fcd9d0, 0xc0011c55c0)
        /usr/local/go/src/cmd/go/internal/par/queue.go:43 +0xa5 fp=0xc0009f6e68 sp=0xc0009f6dc0 pc=0x5bb2a5
cmd/go/internal/modload.readModGraph.func3({{0xc00150f410?, 0xc0003f7010?}, {0xc00078cba6?, 0xc000214780?}}, 0x1)
        /usr/local/go/src/cmd/go/internal/modload/buildlist.go:361 +0x1b2 fp=0xc0009f6f00 sp=0xc0009f6e68 pc=0x7eecf2
cmd/go/internal/modload.readModGraph.func3.1()
        /usr/local/go/src/cmd/go/internal/modload/buildlist.go:378 +0xda fp=0xc0009f6f60 sp=0xc0009f6f00 pc=0x7eee3a
cmd/go/internal/par.(*Queue).Add.func1()
        /usr/local/go/src/cmd/go/internal/par/queue.go:58 +0x7d fp=0xc0009f6fe0 sp=0xc0009f6f60 pc=0x5bb4dd
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0009f6fe8 sp=0xc0009f6fe0 pc=0x468021
created by cmd/go/internal/par.(*Queue).Add
        /usr/local/go/src/cmd/go/internal/par/queue.go:56 +0x225

goroutine 178 [GC worker (idle)]:
runtime.gopark(0xe67800?, 0x1?, 0x33?, 0x2c?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0005b3750 sp=0xc0005b3730 pc=0x43a236
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc0005b37e0 sp=0xc0005b3750 pc=0x41d6d1
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0005b37e8 sp=0xc0005b37e0 pc=0x468021
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 119 [GC worker (idle)]:
runtime.gopark(0x155177418fd1?, 0x1?, 0x8b?, 0x49?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00023cf50 sp=0xc00023cf30 pc=0x43a236
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00023cfe0 sp=0xc00023cf50 pc=0x41d6d1
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00023cfe8 sp=0xc00023cfe0 pc=0x468021
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 162 [GC worker (idle)]:
runtime.gopark(0x155177418e44?, 0x1?, 0xa5?, 0x5a?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000806750 sp=0xc000806730 pc=0x43a236
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc0008067e0 sp=0xc000806750 pc=0x41d6d1
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0008067e8 sp=0xc0008067e0 pc=0x468021
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 120 [GC worker (idle)]:
runtime.gopark(0x15517741841b?, 0x3?, 0xed?, 0x9c?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00023d750 sp=0xc00023d730 pc=0x43a236
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00023d7e0 sp=0xc00023d750 pc=0x41d6d1
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00023d7e8 sp=0xc00023d7e0 pc=0x468021
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 194 [GC worker (idle)]:
runtime.gopark(0x1551774189ad?, 0x1?, 0x2a?, 0x27?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000802750 sp=0xc000802730 pc=0x43a236
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc0008027e0 sp=0xc000802750 pc=0x41d6d1
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0008027e8 sp=0xc0008027e0 pc=0x468021
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 195 [GC worker (idle)]:
runtime.gopark(0x155177418eea?, 0x1?, 0xbf?, 0x7b?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000802f50 sp=0xc000802f30 pc=0x43a236
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000802fe0 sp=0xc000802f50 pc=0x41d6d1
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000802fe8 sp=0xc000802fe0 pc=0x468021
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 196 [GC worker (idle)]:
runtime.gopark(0x15517741906b?, 0x3?, 0x5?, 0x7f?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000803750 sp=0xc000803730 pc=0x43a236
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc0008037e0 sp=0xc000803750 pc=0x41d6d1
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0008037e8 sp=0xc0008037e0 pc=0x468021
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 197 [GC worker (idle)]:
runtime.gopark(0x155177418f83?, 0x1?, 0x12?, 0x45?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000803f50 sp=0xc000803f30 pc=0x43a236
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000803fe0 sp=0xc000803f50 pc=0x41d6d1
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000803fe8 sp=0xc000803fe0 pc=0x468021
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 198 [GC worker (idle)]:
runtime.gopark(0x155177418e05?, 0x1?, 0xc2?, 0x76?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000804750 sp=0xc000804730 pc=0x43a236
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc0008047e0 sp=0xc000804750 pc=0x41d6d1
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0008047e8 sp=0xc0008047e0 pc=0x468021
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 199 [GC worker (idle)]:
runtime.gopark(0x1551774191f8?, 0x3?, 0x79?, 0xf9?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000804f50 sp=0xc000804f30 pc=0x43a236
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000804fe0 sp=0xc000804f50 pc=0x41d6d1
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000804fe8 sp=0xc000804fe0 pc=0x468021
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 46 [GC worker (idle)]:
runtime.gopark(0xe67800?, 0x3?, 0x96?, 0x80?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0005aef50 sp=0xc0005aef30 pc=0x43a236
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc0005aefe0 sp=0xc0005aef50 pc=0x41d6d1
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0005aefe8 sp=0xc0005aefe0 pc=0x468021
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 47 [GC worker (idle)]:
runtime.gopark(0x155177422035?, 0x3?, 0xa9?, 0x72?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0005af750 sp=0xc0005af730 pc=0x43a236
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc0005af7e0 sp=0xc0005af750 pc=0x41d6d1
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0005af7e8 sp=0xc0005af7e0 pc=0x468021
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 6473440 [chan receive]:
runtime.gopark(0x0?, 0xc0009f7d88?, 0xdf?, 0xee?, 0x8?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0009f7d08 sp=0xc0009f7ce8 pc=0x43a236
runtime.chanrecv(0xc000214780, 0xc0009f7e30, 0x1)
        /usr/local/go/src/runtime/chan.go:583 +0x49b fp=0xc0009f7d98 sp=0xc0009f7d08 pc=0x4085fb
runtime.chanrecv1(0x40eedf?, 0x99b180?)
        /usr/local/go/src/runtime/chan.go:442 +0x18 fp=0xc0009f7dc0 sp=0xc0009f7d98 pc=0x4080f8
cmd/go/internal/par.(*Queue).Add(0xc000fcd9d0, 0xc0011c5600)
        /usr/local/go/src/cmd/go/internal/par/queue.go:43 +0xa5 fp=0xc0009f7e68 sp=0xc0009f7dc0 pc=0x5bb2a5
cmd/go/internal/modload.readModGraph.func3({{0xc000633f40?, 0xc0003f7610?}, {0xc00089a6f0?, 0xc000214780?}}, 0x1)
        /usr/local/go/src/cmd/go/internal/modload/buildlist.go:361 +0x1b2 fp=0xc0009f7f00 sp=0xc0009f7e68 pc=0x7eecf2
cmd/go/internal/modload.readModGraph.func3.1()
        /usr/local/go/src/cmd/go/internal/modload/buildlist.go:378 +0xda fp=0xc0009f7f60 sp=0xc0009f7f00 pc=0x7eee3a
cmd/go/internal/par.(*Queue).Add.func1()
        /usr/local/go/src/cmd/go/internal/par/queue.go:58 +0x7d fp=0xc0009f7fe0 sp=0xc0009f7f60 pc=0x5bb4dd
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0009f7fe8 sp=0xc0009f7fe0 pc=0x468021
created by cmd/go/internal/par.(*Queue).Add
        /usr/local/go/src/cmd/go/internal/par/queue.go:56 +0x225

goroutine 6473435 [chan receive]:
runtime.gopark(0x6?, 0x7ef000?, 0xc0?, 0x1d?, 0x1a?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000bd5ea8 sp=0xc000bd5e88 pc=0x43a236
runtime.chanrecv(0xc000214780, 0xc000bd5fa8, 0x1)
        /usr/local/go/src/runtime/chan.go:583 +0x49b fp=0xc000bd5f38 sp=0xc000bd5ea8 pc=0x4085fb
runtime.chanrecv1(0x0?, 0x0?)
        /usr/local/go/src/runtime/chan.go:442 +0x18 fp=0xc000bd5f60 sp=0xc000bd5f38 pc=0x4080f8
cmd/go/internal/par.(*Queue).Add.func1()
        /usr/local/go/src/cmd/go/internal/par/queue.go:60 +0xa5 fp=0xc000bd5fe0 sp=0xc000bd5f60 pc=0x5bb505
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000bd5fe8 sp=0xc000bd5fe0 pc=0x468021
created by cmd/go/internal/par.(*Queue).Add
        /usr/local/go/src/cmd/go/internal/par/queue.go:56 +0x225

goroutine 6473428 [chan receive]:
runtime.gopark(0x22?, 0x7ef000?, 0xa0?, 0x1d?, 0x19?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc001718ea8 sp=0xc001718e88 pc=0x43a236
runtime.chanrecv(0xc000214780, 0xc001718fa8, 0x1)
        /usr/local/go/src/runtime/chan.go:583 +0x49b fp=0xc001718f38 sp=0xc001718ea8 pc=0x4085fb
runtime.chanrecv1(0x5?, 0x5?)
        /usr/local/go/src/runtime/chan.go:442 +0x18 fp=0xc001718f60 sp=0xc001718f38 pc=0x4080f8
cmd/go/internal/par.(*Queue).Add.func1()
        /usr/local/go/src/cmd/go/internal/par/queue.go:60 +0xa5 fp=0xc001718fe0 sp=0xc001718f60 pc=0x5bb505
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc001718fe8 sp=0xc001718fe0 pc=0x468021
created by cmd/go/internal/par.(*Queue).Add
        /usr/local/go/src/cmd/go/internal/par/queue.go:56 +0x225

goroutine 6473432 [runnable]:
runtime.advanceEvacuationMark(0xc0017ca5a0?, 0x99b180?, 0x40?)
        /usr/local/go/src/runtime/map.go:1266 +0xd7 fp=0xc000e45be0 sp=0xc000e45bd8 pc=0x412497
runtime.evacuate(0x99b180?, 0xc0017ca5a0?, 0x40eedf?)
        /usr/local/go/src/runtime/map.go:1262 +0x139 fp=0xc000e45c98 sp=0xc000e45be0 pc=0x411f59
runtime.growWork(0xc000e45d00?, 0xc0017ca5a0, 0x7f0be8260198?)
        /usr/local/go/src/runtime/map.go:1135 +0x66 fp=0xc000e45cc8 sp=0xc000e45c98 pc=0x411de6
runtime.mapassign(0x99b180, 0xc0017ca5a0, 0xc00279cea0?)
        /usr/local/go/src/runtime/map.go:610 +0x14c fp=0xc000e45d50 sp=0xc000e45cc8 pc=0x4109cc
sync.(*Map).LoadOrStore(0xc0002146c8, {0x9cd8a0, 0xc002183c80}, {0x9699a0, 0xc002183ca0})
        /usr/local/go/src/sync/map.go:234 +0x453 fp=0xc000e45df0 sp=0xc000e45d50 pc=0x471073
cmd/go/internal/par.(*Cache).Do(0xc0002146c8, {0x9cd8a0, 0xc002183c80}, 0xc000e45eb0)
        /usr/local/go/src/cmd/go/internal/par/work.go:122 +0x65 fp=0xc000e45e38 sp=0xc000e45df0 pc=0x5bbde5
cmd/go/internal/modload.readModGraph.func2({{0xc001fadb40?, 0xc000215310?}, {0xc000ec1b90?, 0xc000214780?}})
        /usr/local/go/src/cmd/go/internal/modload/buildlist.go:329 +0x132 fp=0xc000e45f00 sp=0xc000e45e38 pc=0x7eef92
cmd/go/internal/modload.readModGraph.func3.1()
        /usr/local/go/src/cmd/go/internal/modload/buildlist.go:362 +0x44 fp=0xc000e45f60 sp=0xc000e45f00 pc=0x7eeda4
cmd/go/internal/par.(*Queue).Add.func1()
        /usr/local/go/src/cmd/go/internal/par/queue.go:58 +0x7d fp=0xc000e45fe0 sp=0xc000e45f60 pc=0x5bb4dd
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000e45fe8 sp=0xc000e45fe0 pc=0x468021
created by cmd/go/internal/par.(*Queue).Add
        /usr/local/go/src/cmd/go/internal/par/queue.go:56 +0x225

goroutine 6473427 [chan receive]:
runtime.gopark(0x0?, 0xc0009fad88?, 0xdf?, 0xee?, 0x8?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0009fad08 sp=0xc0009face8 pc=0x43a236
runtime.chanrecv(0xc000214780, 0xc0009fae30, 0x1)
        /usr/local/go/src/runtime/chan.go:583 +0x49b fp=0xc0009fad98 sp=0xc0009fad08 pc=0x4085fb
runtime.chanrecv1(0x40eedf?, 0x99b180?)
        /usr/local/go/src/runtime/chan.go:442 +0x18 fp=0xc0009fadc0 sp=0xc0009fad98 pc=0x4080f8
cmd/go/internal/par.(*Queue).Add(0xc000fcd9d0, 0xc0011c56c0)
        /usr/local/go/src/cmd/go/internal/par/queue.go:43 +0xa5 fp=0xc0009fae68 sp=0xc0009fadc0 pc=0x5bb2a5
cmd/go/internal/modload.readModGraph.func3({{0xc000ed0cc0?, 0xc0015e1c70?}, {0xc000a50810?, 0xc000214780?}}, 0x1)
        /usr/local/go/src/cmd/go/internal/modload/buildlist.go:361 +0x1b2 fp=0xc0009faf00 sp=0xc0009fae68 pc=0x7eecf2
cmd/go/internal/modload.readModGraph.func3.1()
        /usr/local/go/src/cmd/go/internal/modload/buildlist.go:378 +0xda fp=0xc0009faf60 sp=0xc0009faf00 pc=0x7eee3a
cmd/go/internal/par.(*Queue).Add.func1()
        /usr/local/go/src/cmd/go/internal/par/queue.go:58 +0x7d fp=0xc0009fafe0 sp=0xc0009faf60 pc=0x5bb4dd
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0009fafe8 sp=0xc0009fafe0 pc=0x468021
created by cmd/go/internal/par.(*Queue).Add
        /usr/local/go/src/cmd/go/internal/par/queue.go:56 +0x225

goroutine 6473433 [runnable]:
runtime.gopark(0x0?, 0xc002c8ad88?, 0xdf?, 0xee?, 0x8?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc002c8ad08 sp=0xc002c8ace8 pc=0x43a236
runtime.chanrecv(0xc000214780, 0xc002c8ae30, 0x1)
        /usr/local/go/src/runtime/chan.go:583 +0x49b fp=0xc002c8ad98 sp=0xc002c8ad08 pc=0x4085fb
runtime.chanrecv1(0x40eedf?, 0x99b180?)
        /usr/local/go/src/runtime/chan.go:442 +0x18 fp=0xc002c8adc0 sp=0xc002c8ad98 pc=0x4080f8
cmd/go/internal/par.(*Queue).Add(0xc000fcd9d0, 0xc0011c5580)
        /usr/local/go/src/cmd/go/internal/par/queue.go:43 +0xa5 fp=0xc002c8ae68 sp=0xc002c8adc0 pc=0x5bb2a5
cmd/go/internal/modload.readModGraph.func3({{0xc00148b080?, 0xc0003b0830?}, {0xc0002accf0?, 0xc000214780?}}, 0x1)
        /usr/local/go/src/cmd/go/internal/modload/buildlist.go:361 +0x1b2 fp=0xc002c8af00 sp=0xc002c8ae68 pc=0x7eecf2
cmd/go/internal/modload.readModGraph.func3.1()
        /usr/local/go/src/cmd/go/internal/modload/buildlist.go:378 +0xda fp=0xc002c8af60 sp=0xc002c8af00 pc=0x7eee3a
cmd/go/internal/par.(*Queue).Add.func1()
        /usr/local/go/src/cmd/go/internal/par/queue.go:58 +0x7d fp=0xc002c8afe0 sp=0xc002c8af60 pc=0x5bb4dd
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc002c8afe8 sp=0xc002c8afe0 pc=0x468021
created by cmd/go/internal/par.(*Queue).Add
        /usr/local/go/src/cmd/go/internal/par/queue.go:56 +0x225

goroutine 6473439 [chan receive]:
runtime.gopark(0x6?, 0x7ef000?, 0x0?, 0x1d?, 0x1c?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc002c90ea8 sp=0xc002c90e88 pc=0x43a236
runtime.chanrecv(0xc000214780, 0xc002c90fa8, 0x1)
        /usr/local/go/src/runtime/chan.go:583 +0x49b fp=0xc002c90f38 sp=0xc002c90ea8 pc=0x4085fb
runtime.chanrecv1(0x0?, 0x0?)
        /usr/local/go/src/runtime/chan.go:442 +0x18 fp=0xc002c90f60 sp=0xc002c90f38 pc=0x4080f8
cmd/go/internal/par.(*Queue).Add.func1()
        /usr/local/go/src/cmd/go/internal/par/queue.go:60 +0xa5 fp=0xc002c90fe0 sp=0xc002c90f60 pc=0x5bb505
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc002c90fe8 sp=0xc002c90fe0 pc=0x468021
created by cmd/go/internal/par.(*Queue).Add
        /usr/local/go/src/cmd/go/internal/par/queue.go:56 +0x225

goroutine 6473430 [chan receive]:
runtime.gopark(0x0?, 0xc0009f9d88?, 0xdf?, 0xee?, 0x8?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0009f9d08 sp=0xc0009f9ce8 pc=0x43a236
runtime.chanrecv(0xc000214780, 0xc0009f9e30, 0x1)
        /usr/local/go/src/runtime/chan.go:583 +0x49b fp=0xc0009f9d98 sp=0xc0009f9d08 pc=0x4085fb
runtime.chanrecv1(0x40eedf?, 0x99b180?)
        /usr/local/go/src/runtime/chan.go:442 +0x18 fp=0xc0009f9dc0 sp=0xc0009f9d98 pc=0x4080f8
cmd/go/internal/par.(*Queue).Add(0xc000fcd9d0, 0xc0011c5640)
        /usr/local/go/src/cmd/go/internal/par/queue.go:43 +0xa5 fp=0xc0009f9e68 sp=0xc0009f9dc0 pc=0x5bb2a5
cmd/go/internal/modload.readModGraph.func3({{0xc0012e4440?, 0xc0017d0a70?}, {0xc0002a7770?, 0xc000214780?}}, 0x1)
        /usr/local/go/src/cmd/go/internal/modload/buildlist.go:361 +0x1b2 fp=0xc0009f9f00 sp=0xc0009f9e68 pc=0x7eecf2
cmd/go/internal/modload.readModGraph.func3.1()
        /usr/local/go/src/cmd/go/internal/modload/buildlist.go:378 +0xda fp=0xc0009f9f60 sp=0xc0009f9f00 pc=0x7eee3a
cmd/go/internal/par.(*Queue).Add.func1()
        /usr/local/go/src/cmd/go/internal/par/queue.go:58 +0x7d fp=0xc0009f9fe0 sp=0xc0009f9f60 pc=0x5bb4dd
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0009f9fe8 sp=0xc0009f9fe0 pc=0x468021
created by cmd/go/internal/par.(*Queue).Add
        /usr/local/go/src/cmd/go/internal/par/queue.go:56 +0x225

goroutine 6473429 [chan receive]:
runtime.gopark(0xc001eb3420?, 0xc0011dba10?, 0x78?, 0xcf?, 0xc000425530?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc001717ea8 sp=0xc001717e88 pc=0x43a236
runtime.chanrecv(0xc000214780, 0xc001717fa8, 0x1)
        /usr/local/go/src/runtime/chan.go:583 +0x49b fp=0xc001717f38 sp=0xc001717ea8 pc=0x4085fb
runtime.chanrecv1(0x0?, 0x0?)
        /usr/local/go/src/runtime/chan.go:442 +0x18 fp=0xc001717f60 sp=0xc001717f38 pc=0x4080f8
cmd/go/internal/par.(*Queue).Add.func1()
        /usr/local/go/src/cmd/go/internal/par/queue.go:60 +0xa5 fp=0xc001717fe0 sp=0xc001717f60 pc=0x5bb505
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc001717fe8 sp=0xc001717fe0 pc=0x468021
created by cmd/go/internal/par.(*Queue).Add
        /usr/local/go/src/cmd/go/internal/par/queue.go:56 +0x225

goroutine 6473434 [chan receive]:
runtime.gopark(0x6?, 0x7ef000?, 0xe0?, 0x1c?, 0x23?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0009f5ea8 sp=0xc0009f5e88 pc=0x43a236
runtime.chanrecv(0xc000214780, 0xc0009f5fa8, 0x1)
        /usr/local/go/src/runtime/chan.go:583 +0x49b fp=0xc0009f5f38 sp=0xc0009f5ea8 pc=0x4085fb
runtime.chanrecv1(0x0?, 0x0?)
        /usr/local/go/src/runtime/chan.go:442 +0x18 fp=0xc0009f5f60 sp=0xc0009f5f38 pc=0x4080f8
cmd/go/internal/par.(*Queue).Add.func1()
        /usr/local/go/src/cmd/go/internal/par/queue.go:60 +0xa5 fp=0xc0009f5fe0 sp=0xc0009f5f60 pc=0x5bb505
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0009f5fe8 sp=0xc0009f5fe0 pc=0x468021
created by cmd/go/internal/par.(*Queue).Add
        /usr/local/go/src/cmd/go/internal/par/queue.go:56 +0x225

goroutine 6473436 [chan receive]:
runtime.gopark(0xc001eb3420?, 0xc0002ac7d0?, 0xa0?, 0xf5?, 0xc000425530?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc002c8fea8 sp=0xc002c8fe88 pc=0x43a236
runtime.chanrecv(0xc000214780, 0xc002c8ffa8, 0x1)
        /usr/local/go/src/runtime/chan.go:583 +0x49b fp=0xc002c8ff38 sp=0xc002c8fea8 pc=0x4085fb
runtime.chanrecv1(0x0?, 0x0?)
        /usr/local/go/src/runtime/chan.go:442 +0x18 fp=0xc002c8ff60 sp=0xc002c8ff38 pc=0x4080f8
cmd/go/internal/par.(*Queue).Add.func1()
        /usr/local/go/src/cmd/go/internal/par/queue.go:60 +0xa5 fp=0xc002c8ffe0 sp=0xc002c8ff60 pc=0x5bb505
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc002c8ffe8 sp=0xc002c8ffe0 pc=0x468021
created by cmd/go/internal/par.(*Queue).Add
        /usr/local/go/src/cmd/go/internal/par/queue.go:56 +0x225

goroutine 6473426 [chan receive]:
runtime.gopark(0xc001eb3420?, 0xc0026e2930?, 0x0?, 0x46?, 0xc000425530?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000078ea8 sp=0xc000078e88 pc=0x43a236
runtime.chanrecv(0xc000214780, 0xc000078fa8, 0x1)
        /usr/local/go/src/runtime/chan.go:583 +0x49b fp=0xc000078f38 sp=0xc000078ea8 pc=0x4085fb
runtime.chanrecv1(0xb?, 0x2?)
        /usr/local/go/src/runtime/chan.go:442 +0x18 fp=0xc000078f60 sp=0xc000078f38 pc=0x4080f8
cmd/go/internal/par.(*Queue).Add.func1()
        /usr/local/go/src/cmd/go/internal/par/queue.go:60 +0xa5 fp=0xc000078fe0 sp=0xc000078f60 pc=0x5bb505
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000078fe8 sp=0xc000078fe0 pc=0x468021
created by cmd/go/internal/par.(*Queue).Add
        /usr/local/go/src/cmd/go/internal/par/queue.go:56 +0x225

goroutine 6473441 [chan receive]:
runtime.gopark(0x0?, 0xc0009fbd88?, 0xdf?, 0xee?, 0x8?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0009fbd08 sp=0xc0009fbce8 pc=0x43a236
runtime.chanrecv(0xc000214780, 0xc0009fbe30, 0x1)
        /usr/local/go/src/runtime/chan.go:583 +0x49b fp=0xc0009fbd98 sp=0xc0009fbd08 pc=0x4085fb
runtime.chanrecv1(0x40eedf?, 0x99b180?)
        /usr/local/go/src/runtime/chan.go:442 +0x18 fp=0xc0009fbdc0 sp=0xc0009fbd98 pc=0x4080f8
cmd/go/internal/par.(*Queue).Add(0xc000fcd9d0, 0xc0011c5680)
        /usr/local/go/src/cmd/go/internal/par/queue.go:43 +0xa5 fp=0xc0009fbe68 sp=0xc0009fbdc0 pc=0x5bb2a5
cmd/go/internal/modload.readModGraph.func3({{0xc00109d530?, 0xc0017d04d0?}, {0xc000223888?, 0xc000214780?}}, 0x1)
        /usr/local/go/src/cmd/go/internal/modload/buildlist.go:361 +0x1b2 fp=0xc0009fbf00 sp=0xc0009fbe68 pc=0x7eecf2
cmd/go/internal/modload.readModGraph.func3.1()
        /usr/local/go/src/cmd/go/internal/modload/buildlist.go:378 +0xda fp=0xc0009fbf60 sp=0xc0009fbf00 pc=0x7eee3a
cmd/go/internal/par.(*Queue).Add.func1()
        /usr/local/go/src/cmd/go/internal/par/queue.go:58 +0x7d fp=0xc0009fbfe0 sp=0xc0009fbf60 pc=0x5bb4dd
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0009fbfe8 sp=0xc0009fbfe0 pc=0x468021
created by cmd/go/internal/par.(*Queue).Add
        /usr/local/go/src/cmd/go/internal/par/queue.go:56 +0x225

goroutine 6473437 [runnable]:
type..eq.cmd/vendor/golang.org/x/mod/module.Version(0xc002c8cd08?, 0xc002793c88?)
        <autogenerated>:1 +0x85 fp=0xc002c8cc60 sp=0xc002c8cc58 pc=0x5b4bc5
runtime.mapaccess2(0x99a6a0, 0xc000425410, 0xc001b44d50?)
        /usr/local/go/src/runtime/map.go:505 +0x1a4 fp=0xc002c8cca0 sp=0xc002c8cc60 pc=0x4105e4
cmd/go/internal/mvs.(*Graph).Require(0xc000ee2040, {{0xc000787540?, 0xc001fadb38?}, {0xc001fadb38?, 0xc0017ca5a0?}}, {0xc00059af00, 0x3, 0xc002c8cde0?})
        /usr/local/go/src/cmd/go/internal/mvs/graph.go:76 +0x1f4 fp=0xc002c8cd38 sp=0xc002c8cca0 pc=0x7e2cd4
cmd/go/internal/modload.readModGraph.func2.1()
        /usr/local/go/src/cmd/go/internal/modload/buildlist.go:334 +0x105 fp=0xc002c8cdf0 sp=0xc002c8cd38 pc=0x7ef105
cmd/go/internal/par.(*Cache).Do(0xc0002146c8, {0x9cd8a0, 0xc00223e040}, 0xc002c8ceb0)
        /usr/local/go/src/cmd/go/internal/par/work.go:128 +0xc3 fp=0xc002c8ce38 sp=0xc002c8cdf0 pc=0x5bbe43
cmd/go/internal/modload.readModGraph.func2({{0xc000787540?, 0xc000485370?}, {0xc001fadb38?, 0xc000214780?}})
        /usr/local/go/src/cmd/go/internal/modload/buildlist.go:329 +0x132 fp=0xc002c8cf00 sp=0xc002c8ce38 pc=0x7eef92
cmd/go/internal/modload.readModGraph.func3.1()
        /usr/local/go/src/cmd/go/internal/modload/buildlist.go:362 +0x44 fp=0xc002c8cf60 sp=0xc002c8cf00 pc=0x7eeda4
cmd/go/internal/par.(*Queue).Add.func1()
        /usr/local/go/src/cmd/go/internal/par/queue.go:58 +0x7d fp=0xc002c8cfe0 sp=0xc002c8cf60 pc=0x5bb4dd
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc002c8cfe8 sp=0xc002c8cfe0 pc=0x468021
created by cmd/go/internal/par.(*Queue).Add
        /usr/local/go/src/cmd/go/internal/par/queue.go:56 +0x225

goroutine 6473438 [chan receive]:
runtime.gopark(0xc001eb3420?, 0xc000cc39e0?, 0x80?, 0xf1?, 0xc000425530?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00171bea8 sp=0xc00171be88 pc=0x43a236
runtime.chanrecv(0xc000214780, 0xc00171bfa8, 0x1)
        /usr/local/go/src/runtime/chan.go:583 +0x49b fp=0xc00171bf38 sp=0xc00171bea8 pc=0x4085fb
runtime.chanrecv1(0x0?, 0x0?)
        /usr/local/go/src/runtime/chan.go:442 +0x18 fp=0xc00171bf60 sp=0xc00171bf38 pc=0x4080f8
cmd/go/internal/par.(*Queue).Add.func1()
        /usr/local/go/src/cmd/go/internal/par/queue.go:60 +0xa5 fp=0xc00171bfe0 sp=0xc00171bf60 pc=0x5bb505
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00171bfe8 sp=0xc00171bfe0 pc=0x468021
created by cmd/go/internal/par.(*Queue).Add
        /usr/local/go/src/cmd/go/internal/par/queue.go:56 +0x225

rax    0xca
rbx    0x0
rcx    0x469f03
rdx    0x0
rdi    0xe36f28
rsi    0x80
rbp    0x7ffe82194ad0
rsp    0x7ffe82194a88
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x7ffe82194b00
r13    0x18
r14    0xe35c20
r15    0x7f0bec0ba5cf
rip    0x469f01
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
jasonphi commented 2 years ago

gorelease gets stuck if no arguments are passed or if just the latest -base version is given but it doesn't get stuck if the latest -base version and a new -version are given.

jasonphi commented 2 years ago

It appears we're getting stuck here: https://github.com/golang/go/blob/master/src/cmd/go/internal/modget/get.go#L350-L353

We never break out of this loop in the temporary module created by gorelease

jasonphi commented 2 years ago

I haven't had time to create a minimal repro, yet, but I think I've reduced what's going on a little bit.

When gorelease creates a fake version and uses that as a replace version in the temp Go module we hit a case in go get -d . where it oscillates between two states:

Assume the module gorelease is being run against is foo.example/bar and the latest version of that module is v1.2.3.

jasonphi commented 1 year ago

I still haven't been able to create a minimal reproducer but I did compile a debug build of the go command and stepped through it while this is happening. The root cause seems to involve a circular module dependency and module versions that are on either side of the module graph pruning feature. In other words, in the example above v1.1.0 has a go.mod with a go directive pre Go 1.17 and v1.2.3 has a go.mod with a go directive post Go 1.17.

Without graph pruning there seems to be some dependency constraint that causes the resolver onto an old version and with graph pruning that constraint doesn't exist and a newer version is selected. This appears to happen with different dependencies in the dependency graph but involving the same root (circular) dependency, which is what causes the infinite oscillation back and forth.

I just noticed #56494 which also has a circular module dependency as a contributing factor. While the symptoms are different (hang versus panic) I wonder if the underlying bug is the same?

Once there's a patch for #56494 available I'd be happy to also test it with this issue.