search

golang / go

The Go programming language
https://go.dev
BSD 3-Clause "New" or "Revised" License
120.1k stars 17.24k forks source link

runtime: fatal error: mcall called on m->g0 stack on Windows #67108

Open RichieSams opened 2 weeks ago

RichieSams commented 2 weeks ago

Go version

go1.21.4

Output of go env in your module/workspace:

NOTE: We compile inside docker. The output below is from running go env within the docker container that we use. We cross-compile for linux, windows, and darwin. The workers having a problem in this case are running windows/amd64.

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21.4'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build2751320750=/tmp/go-build -gno-record-gcc-switches'

What did you do?

We have a cli program that runs on many thousands of VMs / bare metal workers as part of our larger CI setup. Recently, we've been seeing a non-trivial number of panics on a specific set of workers. These workers are bare metal AMD Threadripper machines running Windows 10. Below is some info from DXDiag. (I can get additional information if needed).

      Time of this report: 4/23/2024, 09:36:37
             Machine name: XXXXXXXX
               Machine Id: {F429D65A-427C-4F90-A587-8AB68B33FB57}
         Operating System: Windows 10 Enterprise 64-bit (10.0, Build 19045) (19041.vb_release.191206-1406)
                 Language: English (Regional Setting: English)
      System Manufacturer: Equus Computer Systems
             System Model: Nobilis
                     BIOS: F5c (type: UEFI)
                Processor: AMD Ryzen Threadripper 3990X 64-Core Processor  (64 CPUs), ~2.9GHz
                   Memory: 262144MB RAM
      Available OS Memory: 262032MB RAM
                Page File: 38200MB used, 668832MB available
              Windows Dir: C:\Windows
          DirectX Version: DirectX 12
      DX Setup Parameters: Not found
         User DPI Setting: 96 DPI (100 percent)
       System DPI Setting: 96 DPI (100 percent)
          DWM DPI Scaling: Disabled
                 Miracast: Available, no HDCP
Microsoft Graphics Hybrid: Not Supported
 DirectX Database Version: 1.0.8
           DxDiag Version: 10.00.19041.3636 64bit Unicode

We're seeing two types of panics:

  1. fatal error: runtime: mcall called on m->g0 stack
  2. init order seemingly not following the spec: https://go.dev/ref/spec#Package_initialization
    • init() from a package is being called before the init() of a package it depends on
    • Meaning a global variable is not yet populated
    • Leading to a nil pointer dereference panic

I have included multiple call stacks of these panics below. NOTE: these callstacks all happen on different machines. But they only happen on the same "series" of machine, which is documented above.

All the panics seem to happen during the init() phase of the runtime start-up. The panics are not super reproduce-able. I'm only really seeing it due to our scale.

What did you see happen?

Below are the callstacks from a number of panics across multiple machines.

fatal error: runtime: mcall called on m->g0 stack
runtime stack:
runtime.throw({0x1801948?, 0x2f?})
    runtime/panic.go:1077 +0x65 fp=0xc00025bc20 sp=0xc00025bbf0 pc=0x60c7a5
runtime.badmcall(0xd0?)
    runtime/proc.go:503 +0x1f fp=0xc00025bc40 sp=0xc00025bc20 pc=0x60fa9f
runtime.badmcall(0x60db19)
    <autogenerated>:1 +0x25 fp=0xc00025bc58 sp=0xc00025bc40 pc=0x641ea5
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc00025bc70
stack: frame={sp:0xc00025bc40, fp:0xc00025bc58} stack=[0x6fc804000,0x6fc9ff7b0)
goroutine 1 [running, locked to thread]:
google.golang.org/genproto/googleapis/type/expr.init()
    google.golang.org/genproto@v0.0.0-20240227224415-6ceb2ff114de/googleapis/type/expr/expr.pb.go:149 +0x69 fp=0xc00025be10 sp=0xc00025be08 pc=0xc11629
runtime.doInit1(0x243c990)
    runtime/proc.go:6740 +0xdd fp=0xc00025bf40 sp=0xc00025be10 pc=0x61bf1d
runtime.doInit(...)
    runtime/proc.go:6707
runtime.main()
    runtime/proc.go:249 +0x356 fp=0xc00025bfe0 sp=0xc00025bf40 pc=0x60f096
runtime.goexit()
    runtime/asm_amd64.s:1650 +0x1 fp=0xc00025bfe8 sp=0xc00025bfe0 pc=0x63d241
goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    runtime/proc.go:398 +0xce fp=0xc0000f5fa8 sp=0xc0000f5f88 pc=0x60f40e
runtime.goparkunlock(...)
    runtime/proc.go:404
runtime.forcegchelper()
    runtime/proc.go:322 +0xb8 fp=0xc0000f5fe0 sp=0xc0000f5fa8 pc=0x60f298
runtime.goexit()
    runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f5fe8 sp=0xc0000f5fe0 pc=0x63d241
created by runtime.init.6 in goroutine 1
    runtime/proc.go:310 +0x1a
goroutine 3 [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    runtime/proc.go:398 +0xce fp=0xc0000f7f78 sp=0xc0000f7f58 pc=0x60f40e
runtime.goparkunlock(...)
    runtime/proc.go:404
runtime.bgsweep(0x0?)
    runtime/mgcsweep.go:280 +0x94 fp=0xc0000f7fc8 sp=0xc0000f7f78 pc=0x5f8ab4
runtime.gcenable.func1()
    runtime/mgc.go:200 +0x25 fp=0xc0000f7fe0 sp=0xc0000f7fc8 pc=0x5edc25
runtime.goexit()
    runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f7fe8 sp=0xc0000f7fe0 pc=0x63d241
created by runtime.gcenable in goroutine 1
    runtime/mgc.go:200 +0x66
goroutine 18 [GC scavenge wait]:
runtime.gopark(0xc0000440e0?, 0x1a65f48?, 0x1?, 0x0?, 0xc0002041a0?)
    runtime/proc.go:398 +0xce fp=0xc00020df70 sp=0xc00020df50 pc=0x60f40e
runtime.goparkunlock(...)
    runtime/proc.go:404
runtime.(*scavengerState).park(0x2768440)
    runtime/mgcscavenge.go:425 +0x49 fp=0xc00020dfa0 sp=0xc00020df70 pc=0x5f6369
runtime.bgscavenge(0x0?)
    runtime/mgcscavenge.go:653 +0x3c fp=0xc00020dfc8 sp=0xc00020dfa0 pc=0x5f68fc
runtime.gcenable.func2()
    runtime/mgc.go:201 +0x25 fp=0xc00020dfe0 sp=0xc00020dfc8 pc=0x5edbc5
runtime.goexit()
    runtime/asm_amd64.s:1650 +0x1 fp=0xc00020dfe8 sp=0xc00020dfe0 pc=0x63d241
created by runtime.gcenable in goroutine 1
    runtime/mgc.go:201 +0xa5
goroutine 19 [finalizer wait]:
runtime.gopark(0x400000?, 0x1000f9e70?, 0x0?, 0x0?, 0x27cba60?)
    runtime/proc.go:398 +0xce fp=0xc0000f9e28 sp=0xc0000f9e08 pc=0x60f40e
runtime.runfinq()
    runtime/mfinal.go:193 +0x107 fp=0xc0000f9fe0 sp=0xc0000f9e28 pc=0x5ecce7
runtime.goexit()
    runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f9fe8 sp=0xc0000f9fe0 pc=0x63d241
created by runtime.createfing in goroutine 1
    runtime/mfinal.go:163 +0x3d
fatal error: runtime: mcall called on m->g0 stack
runtime stack:
runtime.throw({0x1522440?, 0x20?})
    /usr/local/go/src/runtime/panic.go:1047 +0x65 fp=0xc00083f748 sp=0xc00083f718 pc=0x2bc045
runtime.badmcall(0xa?)
    /usr/local/go/src/runtime/proc.go:468 +0x25 fp=0xc00083f768 sp=0xc00083f748 pc=0x2bf2c5
runtime.badmcall(0x0)
    <autogenerated>:1 +0x2c fp=0xc00083f780 sp=0xc00083f768 pc=0x2ec86c
runtime.asyncPreempt2()
    /usr/local/go/src/runtime/preempt.go:308 +0x3f fp=0xc00083f7a0 sp=0xc00083f780 pc=0x2bd27f
runtime.asyncPreempt()
    /usr/local/go/src/runtime/preempt_amd64.s:50 +0xdb fp=0xc00083f928 sp=0xc00083f7a0 pc=0x2ebbbb
github.com/dustin/go-humanize.revfmap(0xc00024f680)
    /builds/my-project/vendor/github.com/dustin/go-humanize/si.go:33 fp=0xc00083f930 sp=0xc00083f928 pc=0xe36ac0
github.com/dustin/go-humanize.init()
    /builds/my-project/vendor/github.com/dustin/go-humanize/si.go:30 +0x142b fp=0xc00083f990 sp=0xc00083f930 pc=0xe3872b
runtime.doInit(0x2156de0)
    /usr/local/go/src/runtime/proc.go:6329 +0x12d fp=0xc00083fac0 sp=0xc00083f990 pc=0x2cb2ed
runtime.doInit(0x2157c80)
    /usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083fbf0 sp=0xc00083fac0 pc=0x2cb231
runtime.doInit(0x215a020)
    /usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083fd20 sp=0xc00083fbf0 pc=0x2cb231
runtime.doInit(0x2164660)
    /usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083fe50 sp=0xc00083fd20 pc=0x2cb231
runtime.doInit(0x2151b00)
    /usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083ff80 sp=0xc00083fe50 pc=0x2cb231
runtime.main()
    /usr/local/go/src/runtime/proc.go:233 +0x1bf fp=0xc00083ffe0 sp=0xc00083ff80 pc=0x2be73f
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00083ffe8 sp=0xc00083ffe0 pc=0x2ea561
goroutine 1 [runnable, locked to thread]:
github.com/dustin/go-humanize.revfmap(0xc00024f680?)
    /builds/my-project/vendor/github.com/dustin/go-humanize/si.go:33 +0xf0 fp=0xc00083f930 sp=0xc00083f928 pc=0xe36bb0
github.com/dustin/go-humanize.init()
    /builds/my-project/vendor/github.com/dustin/go-humanize/si.go:30 +0x142b fp=0xc00083f990 sp=0xc00083f930 pc=0xe3872b
runtime.doInit(0x2156de0)
    /usr/local/go/src/runtime/proc.go:6329 +0x12d fp=0xc00083fac0 sp=0xc00083f990 pc=0x2cb2ed
runtime.doInit(0x2157c80)
    /usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083fbf0 sp=0xc00083fac0 pc=0x2cb231
runtime.doInit(0x215a020)
    /usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083fd20 sp=0xc00083fbf0 pc=0x2cb231
runtime.doInit(0x2164660)
    /usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083fe50 sp=0xc00083fd20 pc=0x2cb231
runtime.doInit(0x2151b00)
    /usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083ff80 sp=0xc00083fe50 pc=0x2cb231
runtime.main()
    /usr/local/go/src/runtime/proc.go:233 +0x1bf fp=0xc00083ffe0 sp=0xc00083ff80 pc=0x2be73f
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00083ffe8 sp=0xc00083ffe0 pc=0x2ea561
goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0000f3fb0 sp=0xc0000f3f90 pc=0x2beb16
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:369
runtime.forcegchelper()
    /usr/local/go/src/runtime/proc.go:302 +0xb1 fp=0xc0000f3fe0 sp=0xc0000f3fb0 pc=0x2be9b1
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000f3fe8 sp=0xc0000f3fe0 pc=0x2ea561
created by runtime.init.6
    /usr/local/go/src/runtime/proc.go:290 +0x25
goroutine 18 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00010ff90 sp=0xc00010ff70 pc=0x2beb16
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:369
runtime.bgsweep(0x0?)
    /usr/local/go/src/runtime/mgcsweep.go:297 +0xd7 fp=0xc00010ffc8 sp=0xc00010ff90 pc=0x2a7c37
runtime.gcenable.func1()
    /usr/local/go/src/runtime/mgc.go:178 +0x26 fp=0xc00010ffe0 sp=0xc00010ffc8 pc=0x29c786
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00010ffe8 sp=0xc00010ffe0 pc=0x2ea561
created by runtime.gcenable
    /usr/local/go/src/runtime/mgc.go:178 +0x6b
goroutine 19 [GC scavenge wait]:
runtime.gopark(0xc000106000?, 0x179b290?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000111f70 sp=0xc000111f50 pc=0x2beb16
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:369
runtime.(*scavengerState).park(0x2432600)
    /usr/local/go/src/runtime/mgcscavenge.go:389 +0x53 fp=0xc000111fa0 sp=0xc000111f70 pc=0x2a5c73
runtime.bgscavenge(0x0?)
    /usr/local/go/src/runtime/mgcscavenge.go:622 +0x65 fp=0xc000111fc8 sp=0xc000111fa0 pc=0x2a6285
runtime.gcenable.func2()
    /usr/local/go/src/runtime/mgc.go:179 +0x26 fp=0xc000111fe0 sp=0xc000111fc8 pc=0x29c726
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000111fe8 sp=0xc000111fe0 pc=0x2ea561
created by runtime.gcenable
    /usr/local/go/src/runtime/mgc.go:179 +0xaa
goroutine 34 [finalizer wait]:
runtime.gopark(0x0?, 0xc0000f7e70?, 0x6b?, 0x7d?, 0xc0000f7f70?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0000f7e28 sp=0xc0000f7e08 pc=0x2beb16
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:369
runtime.runfinq()
    /usr/local/go/src/runtime/mfinal.go:180 +0x10f fp=0xc0000f7fe0 sp=0xc0000f7e28 pc=0x29b88f
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000f7fe8 sp=0xc0000f7fe0 pc=0x2ea561
created by runtime.createfing
    /usr/local/go/src/runtime/mfinal.go:157 +0x45
goroutine 35 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00010bf50 sp=0xc00010bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00010bfe0 sp=0xc00010bf50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00010bfe8 sp=0xc00010bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 3 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0000f5f50 sp=0xc0000f5f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc0000f5fe0 sp=0xc0000f5f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000f5fe8 sp=0xc0000f5fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 20 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00011df50 sp=0xc00011df30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00011dfe0 sp=0xc00011df50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00011dfe8 sp=0xc00011dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 21 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00011ff50 sp=0xc00011ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00011ffe0 sp=0xc00011ff50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00011ffe8 sp=0xc00011ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 36 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00010df50 sp=0xc00010df30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00010dfe0 sp=0xc00010df50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00010dfe8 sp=0xc00010dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 37 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000119f50 sp=0xc000119f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000119fe0 sp=0xc000119f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000119fe8 sp=0xc000119fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 4 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000505f50 sp=0xc000505f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000505fe0 sp=0xc000505f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000505fe8 sp=0xc000505fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 5 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000507f50 sp=0xc000507f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000507fe0 sp=0xc000507f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000507fe8 sp=0xc000507fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 6 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000501f50 sp=0xc000501f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000501fe0 sp=0xc000501f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000501fe8 sp=0xc000501fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 22 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000127f50 sp=0xc000127f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000127fe0 sp=0xc000127f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000127fe8 sp=0xc000127fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 23 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000129f50 sp=0xc000129f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000129fe0 sp=0xc000129f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000129fe8 sp=0xc000129fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 38 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00011bf50 sp=0xc00011bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00011bfe0 sp=0xc00011bf50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00011bfe8 sp=0xc00011bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 7 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000503f50 sp=0xc000503f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000503fe0 sp=0xc000503f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000503fe8 sp=0xc000503fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 24 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000123f50 sp=0xc000123f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000123fe0 sp=0xc000123f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000123fe8 sp=0xc000123fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 25 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000125f50 sp=0xc000125f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000125fe0 sp=0xc000125f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000125fe8 sp=0xc000125fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 8 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00050df50 sp=0xc00050df30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00050dfe0 sp=0xc00050df50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00050dfe8 sp=0xc00050dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 26 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000509f50 sp=0xc000509f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000509fe0 sp=0xc000509f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000509fe8 sp=0xc000509fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 9 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00050ff50 sp=0xc00050ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00050ffe0 sp=0xc00050ff50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00050ffe8 sp=0xc00050ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 27 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00050bf50 sp=0xc00050bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00050bfe0 sp=0xc00050bf50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00050bfe8 sp=0xc00050bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 10 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000515f50 sp=0xc000515f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000515fe0 sp=0xc000515f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000515fe8 sp=0xc000515fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 28 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000511f50 sp=0xc000511f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000511fe0 sp=0xc000511f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000511fe8 sp=0xc000511fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 11 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000517f50 sp=0xc000517f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000517fe0 sp=0xc000517f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000517fe8 sp=0xc000517fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 29 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000513f50 sp=0xc000513f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000513fe0 sp=0xc000513f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000513fe8 sp=0xc000513fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 12 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00051df50 sp=0xc00051df30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00051dfe0 sp=0xc00051df50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00051dfe8 sp=0xc00051dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 30 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000519f50 sp=0xc000519f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000519fe0 sp=0xc000519f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000519fe8 sp=0xc000519fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 13 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00051ff50 sp=0xc00051ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00051ffe0 sp=0xc00051ff50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00051ffe8 sp=0xc00051ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 31 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00051bf50 sp=0xc00051bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00051bfe0 sp=0xc00051bf50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00051bfe8 sp=0xc00051bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 14 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000525f50 sp=0xc000525f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000525fe0 sp=0xc000525f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000525fe8 sp=0xc000525fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 39 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000521f50 sp=0xc000521f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000521fe0 sp=0xc000521f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000521fe8 sp=0xc000521fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 15 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000527f50 sp=0xc000527f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000527fe0 sp=0xc000527f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000527fe8 sp=0xc000527fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 40 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000523f50 sp=0xc000523f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000523fe0 sp=0xc000523f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000523fe8 sp=0xc000523fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 16 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00052ff50 sp=0xc00052ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00052ffe0 sp=0xc00052ff50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00052ffe8 sp=0xc00052ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 41 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00052bf50 sp=0xc00052bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00052bfe0 sp=0xc00052bf50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00052bfe8 sp=0xc00052bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 50 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000531f50 sp=0xc000531f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000531fe0 sp=0xc000531f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000531fe8 sp=0xc000531fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 42 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00052df50 sp=0xc00052df30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00052dfe0 sp=0xc00052df50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00052dfe8 sp=0xc00052dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 51 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000537f50 sp=0xc000537f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000537fe0 sp=0xc000537f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000537fe8 sp=0xc000537fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 43 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000533f50 sp=0xc000533f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000533fe0 sp=0xc000533f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000533fe8 sp=0xc000533fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 52 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000539f50 sp=0xc000539f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000539fe0 sp=0xc000539f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000539fe8 sp=0xc000539fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 44 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000535f50 sp=0xc000535f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000535fe0 sp=0xc000535f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000535fe8 sp=0xc000535fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 53 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00053ff50 sp=0xc00053ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00053ffe0 sp=0xc00053ff50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00053ffe8 sp=0xc00053ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 45 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00053bf50 sp=0xc00053bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00053bfe0 sp=0xc00053bf50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00053bfe8 sp=0xc00053bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 54 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000541f50 sp=0xc000541f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000541fe0 sp=0xc000541f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000541fe8 sp=0xc000541fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 46 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00053df50 sp=0xc00053df30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00053dfe0 sp=0xc00053df50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00053dfe8 sp=0xc00053dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 55 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000547f50 sp=0xc000547f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000547fe0 sp=0xc000547f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000547fe8 sp=0xc000547fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 47 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000543f50 sp=0xc000543f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000543fe0 sp=0xc000543f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000543fe8 sp=0xc000543fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 56 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000549f50 sp=0xc000549f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000549fe0 sp=0xc000549f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000549fe8 sp=0xc000549fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 48 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000545f50 sp=0xc000545f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000545fe0 sp=0xc000545f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000545fe8 sp=0xc000545fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 57 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00054ff50 sp=0xc00054ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00054ffe0 sp=0xc00054ff50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00054ffe8 sp=0xc00054ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 49 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00054bf50 sp=0xc00054bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00054bfe0 sp=0xc00054bf50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00054bfe8 sp=0xc00054bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 58 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000551f50 sp=0xc000551f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000551fe0 sp=0xc000551f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000551fe8 sp=0xc000551fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 66 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00054df50 sp=0xc00054df30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00054dfe0 sp=0xc00054df50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00054dfe8 sp=0xc00054dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 59 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000557f50 sp=0xc000557f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000557fe0 sp=0xc000557f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000557fe8 sp=0xc000557fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 67 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000553f50 sp=0xc000553f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000553fe0 sp=0xc000553f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000553fe8 sp=0xc000553fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 60 [GC worker (idle)]:
runtime.gopark(0x1d7554aaaaf8c?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000559f50 sp=0xc000559f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000559fe0 sp=0xc000559f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000559fe8 sp=0xc000559fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 68 [GC worker (idle)]:
runtime.gopark(0x1d7554aaaaf8c?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000555f50 sp=0xc000555f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000555fe0 sp=0xc000555f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000555fe8 sp=0xc000555fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 61 [GC worker (idle)]:
runtime.gopark(0x2496880?, 0x1?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00055ff50 sp=0xc00055ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00055ffe0 sp=0xc00055ff50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00055ffe8 sp=0xc00055ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 69 [GC worker (idle)]:
runtime.gopark(0x1d7554aaaaf8c?, 0x1?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00055bf50 sp=0xc00055bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00055bfe0 sp=0xc00055bf50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00055bfe8 sp=0xc00055bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 62 [GC worker (idle)]:
runtime.gopark(0x2496880?, 0x1?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000561f50 sp=0xc000561f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000561fe0 sp=0xc000561f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000561fe8 sp=0xc000561fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 70 [GC worker (idle)]:
runtime.gopark(0x1d7554aaaaf8c?, 0x1?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00055df50 sp=0xc00055df30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00055dfe0 sp=0xc00055df50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00055dfe8 sp=0xc00055dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 63 [GC worker (idle)]:
runtime.gopark(0x1d7554aaaaf8c?, 0x1?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000567f50 sp=0xc000567f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000567fe0 sp=0xc000567f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000567fe8 sp=0xc000567fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 71 [GC worker (idle)]:
runtime.gopark(0x2496880?, 0x1?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000563f50 sp=0xc000563f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000563fe0 sp=0xc000563f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000563fe8 sp=0xc000563fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 64 [GC worker (idle)]:
runtime.gopark(0x2496880?, 0x1?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000569f50 sp=0xc000569f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000569fe0 sp=0xc000569f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000569fe8 sp=0xc000569fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 72 [GC worker (idle)]:
runtime.gopark(0x2496880?, 0x1?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000565f50 sp=0xc000565f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000565fe0 sp=0xc000565f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000565fe8 sp=0xc000565fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 65 [GC worker (idle)]:
runtime.gopark(0x2496880?, 0x1?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00056ff50 sp=0xc00056ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00056ffe0 sp=0xc00056ff50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00056ffe8 sp=0xc00056ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 83 [select]:
runtime.gopark(0xc000571f88?, 0x3?, 0xc8?, 0xc?, 0xc000571f72?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000571df8 sp=0xc000571dd8 pc=0x2beb16
runtime.selectgo(0xc000571f88, 0xc000571f6c, 0xc0001d2500?, 0x0, 0x0?, 0x1)
    /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc000571f38 sp=0xc000571df8 pc=0x2ccebc
go.opencensus.io/stats/view.(*worker).start(0xc0001d2500)
    /builds/my-project/vendor/go.opencensus.io/stats/view/worker.go:292 +0xad fp=0xc000571fc8 sp=0xc000571f38 pc=0x977c2d
go.opencensus.io/stats/view.init.0.func1()
    /builds/my-project/vendor/go.opencensus.io/stats/view/worker.go:34 +0x26 fp=0xc000571fe0 sp=0xc000571fc8 pc=0x976e66
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000571fe8 sp=0xc000571fe0 pc=0x2ea561
created by go.opencensus.io/stats/view.init.0
    /builds/my-project/vendor/go.opencensus.io/stats/view/worker.go:34 +0x8d

Then here are some callstacks from init's that are seemingly not following the right init order:

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x5 pc=0xb84adc]
goroutine 1 [running]:
golang.org/x/text/internal/language.addTags({0x139, 0x0, 0x0, 0x0, 0x0, {0x0, 0x0}})
    golang.org/x/text@v0.14.0/internal/language/match.go:99 +0x27c
golang.org/x/text/internal/language.Tag.Maximize(...)
    golang.org/x/text@v0.14.0/internal/language/match.go:68
golang.org/x/text/language.init.0()
    golang.org/x/text@v0.14.0/language/match.go:727 +0x26d
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x9 pc=0x132a0c0]
goroutine 1 [running]:
google.golang.org/protobuf/internal/filedesc.(*Enums).Len(0x3023620?)
    /builds/my-project/vendor/google.golang.org/protobuf/internal/filedesc/desc_list_gen.go:25
google.golang.org/protobuf/reflect/protoregistry.rangeTopLevelDescriptors({0x23b2710, 0xc0003fc000}, 0xc00035f288)
    /builds/my-project/vendor/google.golang.org/protobuf/reflect/protoregistry/registry.go:408 +0x5d
google.golang.org/protobuf/reflect/protoregistry.(*Files).RegisterFile(0xc0000081f8, {0x23b2710?, 0xc0003fc000?})
    /builds/my-project/vendor/google.golang.org/protobuf/reflect/protoregistry/registry.go:149 +0x739
google.golang.org/protobuf/internal/filedesc.Builder.Build({{0x1dafbd0, 0x2e}, {0x2d53d20, 0x1bf, 0x1bf}, 0x1, 0x0, 0x0, 0x0, {0x239d908, ...}, ...})
    /builds/my-project/vendor/google.golang.org/protobuf/internal/filedesc/build.go:112 +0x1d6
google.golang.org/protobuf/internal/filetype.Builder.Build({{{0x1dafbd0, 0x2e}, {0x2d53d20, 0x1bf, 0x1bf}, 0x1, 0x0, 0x0, 0x0, {0x0, ...}, ...}, ...})
    /builds/my-project/vendor/google.golang.org/protobuf/internal/filetype/build.go:138 +0x1b8
google.golang.org/genproto/googleapis/rpc/code.file_google_rpc_code_proto_init()
    /builds/my-project/vendor/google.golang.org/genproto/googleapis/rpc/code/code.pb.go:331 +0x198
google.golang.org/genproto/googleapis/rpc/code.init.0()
    /builds/my-project/vendor/google.golang.org/genproto/googleapis/rpc/code/code.pb.go:313 +0x17

What did you expect to see?

The program finishes the init() phase without panics

RichieSams commented 2 weeks ago

Below is my go.mod file (with a few redactions)

module my-project/v3

require (
    github.com/AzureAD/microsoft-authentication-library-for-go v1.2.2
    github.com/antlr4-go/antlr/v4 v4.13.0
    github.com/bmatcuk/doublestar v1.3.4
    github.com/bsipos/thist v1.0.0
    github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc
    github.com/dustin/go-humanize v1.0.1
    github.com/getsentry/sentry-go v0.27.0
    github.com/go-ole/go-ole v1.3.0
    github.com/gobwas/glob v0.2.3
    github.com/gofrs/flock v0.8.1
    github.com/gonum/stat v0.0.0-20181125101827-41a0da705a5b
    github.com/gorilla/handlers v1.5.2
    github.com/gorilla/mux v1.8.1
    github.com/hashicorp/go-multierror v1.1.1
    github.com/jotfs/fastcdc-go v0.2.0
    github.com/json-iterator/go v1.1.12
    github.com/juju/ratelimit v1.0.2
    github.com/karrick/godirwalk v1.16.2
    github.com/olekukonko/tablewriter v0.0.5
    github.com/pkg/errors v0.9.1
    github.com/restic/chunker v0.4.0
    github.com/satori/go.uuid v1.2.0
    github.com/schollz/progressbar/v3 v3.14.2
    github.com/sirupsen/logrus v1.9.3
    github.com/skratchdot/open-golang v0.0.0-20190104022628-a2dfa6d0dab6
    github.com/spf13/cobra v1.8.0
    github.com/spf13/viper v1.18.2
    github.com/stretchr/testify v1.9.0
    github.com/thediveo/enumflag v0.10.1
    github.com/tinylib/msgp v1.1.9
    github.com/xhit/go-str2duration/v2 v2.1.0
    internal-a v1.7.0
    internal-b v1.2.0
    internal-c v1.7.0
    internal-d/v2 v2.70.0
    go.opentelemetry.io/otel v1.25.0
    go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.25.0
    go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.25.0
    go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.25.0
    go.opentelemetry.io/otel/metric v1.25.0
    go.opentelemetry.io/otel/sdk v1.25.0
    go.opentelemetry.io/otel/sdk/metric v1.25.0
    go.opentelemetry.io/otel/trace v1.25.0
    golang.org/x/sync v0.7.0
    golang.org/x/sys v0.19.0
    golang.org/x/term v0.19.0
    golang.org/x/text v0.14.0
    google.golang.org/grpc v1.63.2
    gopkg.in/yaml.v2 v2.4.0
)

require (
    cloud.google.com/go v0.112.0 // indirect
    cloud.google.com/go/compute v1.24.0 // indirect
    cloud.google.com/go/compute/metadata v0.2.3 // indirect
    cloud.google.com/go/iam v1.1.6 // indirect
    cloud.google.com/go/storage v1.36.0 // indirect
    github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78 // indirect
    github.com/Microsoft/go-winio v0.6.0 // indirect
    github.com/Nvveen/Gotty v0.0.0-20120604004816-cd527374f1e5 // indirect
    github.com/Shopify/toxiproxy/v2 v2.4.0 // indirect
    github.com/ajstarks/svgo v0.0.0-20180226025133-644b8db467af // indirect
    github.com/aws/aws-sdk-go v1.28.9 // indirect
    github.com/beorn7/perks v1.0.1 // indirect
    github.com/cenkalti/backoff/v4 v4.3.0 // indirect
    github.com/cespare/xxhash/v2 v2.2.0 // indirect
    github.com/containerd/continuity v0.3.0 // indirect
    github.com/cpuguy83/go-md2man/v2 v2.0.3 // indirect
    github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
    github.com/docker/cli v20.10.17+incompatible // indirect
    github.com/docker/docker v20.10.7+incompatible // indirect
    github.com/docker/go-connections v0.4.0 // indirect
    github.com/docker/go-units v0.4.0 // indirect
    github.com/fatih/color v1.14.1 // indirect
    github.com/felixge/httpsnoop v1.0.4 // indirect
    github.com/fogleman/gg v1.2.1-0.20190220221249-0403632d5b90 // indirect
    github.com/fsnotify/fsnotify v1.7.0 // indirect
    github.com/go-logr/logr v1.4.1 // indirect
    github.com/go-logr/stdr v1.2.2 // indirect
    github.com/gogo/protobuf v1.3.2 // indirect
    github.com/golang-jwt/jwt/v5 v5.0.0 // indirect
    github.com/golang/freetype v0.0.0-20170609003504-e2365dfdc4a0 // indirect
    github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
    github.com/golang/protobuf v1.5.4 // indirect
    github.com/gonum/blas v0.0.0-20181208220705-f22b278b28ac // indirect
    github.com/gonum/floats v0.0.0-20181209220543-c233463c7e82 // indirect
    github.com/gonum/integrate v0.0.0-20181209220457-a422b5c0fdf2 // indirect
    github.com/gonum/internal v0.0.0-20181124074243-f884aa714029 // indirect
    github.com/gonum/lapack v0.0.0-20181123203213-e4cdc5a0bff9 // indirect
    github.com/gonum/matrix v0.0.0-20181209220409-c518dec07be9 // indirect
    github.com/google/s2a-go v0.1.7 // indirect
    github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect
    github.com/google/uuid v1.6.0 // indirect
    github.com/googleapis/enterprise-certificate-proxy v0.3.2 // indirect
    github.com/googleapis/gax-go/v2 v2.12.0 // indirect
    github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.0 // indirect
    github.com/hashicorp/errwrap v1.0.0 // indirect
    github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
    github.com/hashicorp/go-hclog v1.5.0 // indirect
    github.com/hashicorp/go-retryablehttp v0.7.0 // indirect
    github.com/hashicorp/hcl v1.0.0 // indirect
    github.com/imdario/mergo v0.3.12 // indirect
    github.com/inconshreveable/mousetrap v1.1.0 // indirect
    github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af // indirect
    github.com/jung-kurt/gofpdf v1.0.3-0.20190309125859-24315acbbda5 // indirect
    github.com/kylelemons/godebug v1.1.0 // indirect
    github.com/magiconair/properties v1.8.7 // indirect
    github.com/mattn/go-colorable v0.1.13 // indirect
    github.com/mattn/go-isatty v0.0.20 // indirect
    github.com/mattn/go-runewidth v0.0.14 // indirect
    github.com/matttproud/golang_protobuf_extensions v1.0.1 // indirect
    github.com/minio/minio-go/v6 v6.0.46 // indirect
    github.com/minio/sha256-simd v0.1.1 // indirect
    github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db // indirect
    github.com/mitchellh/go-homedir v1.1.0 // indirect
    github.com/mitchellh/mapstructure v1.5.0 // indirect
    github.com/moby/term v0.0.0-20201216013528-df9cb8a40635 // indirect
    github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
    github.com/modern-go/reflect2 v1.0.2 // indirect
    github.com/opencontainers/go-digest v1.0.0 // indirect
    github.com/opencontainers/image-spec v1.0.2 // indirect
    github.com/opencontainers/runc v1.1.5 // indirect
    github.com/ory/dockertest/v3 v3.10.0 // indirect
    github.com/pelletier/go-toml/v2 v2.1.0 // indirect
    github.com/philhofer/fwd v1.1.2 // indirect
    github.com/pierrec/lz4 v2.0.5+incompatible // indirect
    github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 // indirect
    github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
    github.com/prometheus/client_golang v1.12.1 // indirect
    github.com/prometheus/client_model v0.2.0 // indirect
    github.com/prometheus/common v0.32.1 // indirect
    github.com/prometheus/procfs v0.7.3 // indirect
    github.com/rivo/uniseg v0.4.7 // indirect
    github.com/russross/blackfriday/v2 v2.1.0 // indirect
    github.com/sagikazarmark/locafero v0.4.0 // indirect
    github.com/sagikazarmark/slog-shim v0.1.0 // indirect
    github.com/sourcegraph/conc v0.3.0 // indirect
    github.com/spf13/afero v1.11.0 // indirect
    github.com/spf13/cast v1.6.0 // indirect
    github.com/spf13/pflag v1.0.5 // indirect
    github.com/subosito/gotenv v1.6.0 // indirect
    github.com/xeipuuv/gojsonpointer v0.0.0-20180127040702-4e3ac2762d5f // indirect
    github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 // indirect
    github.com/xeipuuv/gojsonschema v1.2.0 // indirect
    go.opencensus.io v0.24.0 // indirect
    go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.47.0 // indirect
    go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptrace v0.30.0 // indirect
    go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.47.0 // indirect
    go.opentelemetry.io/proto/otlp v1.1.0 // indirect
    go.uber.org/atomic v1.9.0 // indirect
    go.uber.org/multierr v1.9.0 // indirect
    golang.org/x/crypto v0.21.0 // indirect
    golang.org/x/exp v0.0.0-20230905200255-921286631fa9 // indirect
    golang.org/x/image v0.0.0-20190802002840-cff245a6509b // indirect
    golang.org/x/mod v0.13.0 // indirect
    golang.org/x/net v0.23.0 // indirect
    golang.org/x/oauth2 v0.17.0 // indirect
    golang.org/x/time v0.5.0 // indirect
    golang.org/x/tools v0.14.0 // indirect
    gonum.org/v1/plot v0.0.0-20190515093506-e2840ee46a6b // indirect
    google.golang.org/api v0.162.0 // indirect
    google.golang.org/appengine v1.6.8 // indirect
    google.golang.org/genproto v0.0.0-20240227224415-6ceb2ff114de // indirect
    google.golang.org/genproto/googleapis/api v0.0.0-20240227224415-6ceb2ff114de // indirect
    google.golang.org/genproto/googleapis/rpc v0.0.0-20240401170217-c3f982113cda // indirect
    google.golang.org/protobuf v1.33.0 // indirect
    gopkg.in/ini.v1 v1.67.0 // indirect
    gopkg.in/yaml.v3 v3.0.1 // indirect
    nhooyr.io/websocket v1.8.10 // indirect
)

replace github.com/antlr/antlr4 v0.0.0-20180905200841-432022fc1ca0 => github.com/michilu/antlr4 v0.0.0-20180803091604-411960b1878f

replace github.com/uber/jaeger-lib v2.0.0+incompatible => github.com/uber/jaeger-lib v1.5.0

replace github.com/skratchdot/open-golang => github.com/seanhoughton/open-golang v0.0.0-20190322203133-7972e18a724d

replace git.apache.org/thrift.git => github.com/apache/thrift v0.12.0

go 1.21
RichieSams commented 2 weeks ago

This seems to be related to: https://github.com/golang/go/issues/56774

But in that issue, the poster said updating to 1.21.3 fixed it. However, we're using 1.21.4, and still seeing the issue.

RichieSams commented 2 weeks ago

Slightly different panic:

unexpected fault address 0x411a89e
fatal error: fault
[signal 0xc0000005 code=0x0 addr=0x411a89e pc=0xe003e1]
goroutine 1 [running, locked to thread]:
runtime.throw({0x1f23e6f?, 0x0?})
    runtime/panic.go:1077 +0x65 fp=0xc0000fbd58 sp=0xc0000fbd28 pc=0xd9c7a5
runtime.sigpanic()
    runtime/signal_windows.go:377 +0xd0 fp=0xc0000fbda0 sp=0xc0000fbd58 pc=0xdb06b0
time.map.init.1()
    time/zoneinfo_abbrs_windows.go:154 +0x81 fp=0xc0000fbe00 sp=0xc0000fbda0 pc=0xe003e1
time.init()
    time/zoneinfo_abbrs_windows.go:15 +0x25 fp=0xc0000fbe10 sp=0xc0000fbe00 pc=0xe001a5
runtime.doInit1(0x2b81dc0)
    runtime/proc.go:6740 +0xdd fp=0xc0000fbf40 sp=0xc0000fbe10 pc=0xdabf1d
runtime.doInit(...)
    runtime/proc.go:6707
runtime.main()
    runtime/proc.go:249 +0x356 fp=0xc0000fbfe0 sp=0xc0000fbf40 pc=0xd9f096
runtime.goexit()
    runtime/asm_amd64.s:1650 +0x1 fp=0xc0000fbfe8 sp=0xc0000fbfe0 pc=0xdcd141
goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    runtime/proc.go:398 +0xce fp=0xc0000f7fa8 sp=0xc0000f7f88 pc=0xd9f40e
runtime.goparkunlock(...)
    runtime/proc.go:404
runtime.forcegchelper()
    runtime/proc.go:322 +0xb8 fp=0xc0000f7fe0 sp=0xc0000f7fa8 pc=0xd9f298
runtime.goexit()
    runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f7fe8 sp=0xc0000f7fe0 pc=0xdcd141
created by runtime.init.6 in goroutine 1
    runtime/proc.go:310 +0x1a
goroutine 3 [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    runtime/proc.go:398 +0xce fp=0xc0000f9f78 sp=0xc0000f9f58 pc=0xd9f40e
runtime.goparkunlock(...)
    runtime/proc.go:404
runtime.bgsweep(0x0?)
    runtime/mgcsweep.go:280 +0x94 fp=0xc0000f9fc8 sp=0xc0000f9f78 pc=0xd88ab4
runtime.gcenable.func1()
    runtime/mgc.go:200 +0x25 fp=0xc0000f9fe0 sp=0xc0000f9fc8 pc=0xd7dc25
runtime.goexit()
    runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f9fe8 sp=0xc0000f9fe0 pc=0xdcd141
created by runtime.gcenable in goroutine 1
    runtime/mgc.go:200 +0x66
goroutine 4 [GC scavenge wait]:
runtime.gopark(0xc0000480e0?, 0x21b8428?, 0x1?, 0x0?, 0xc0000f4b60?)
    runtime/proc.go:398 +0xce fp=0xc000109f70 sp=0xc000109f50 pc=0xd9f40e
runtime.goparkunlock(...)
    runtime/proc.go:404
runtime.(*scavengerState).park(0x2e63e20)
    runtime/mgcscavenge.go:425 +0x49 fp=0xc000109fa0 sp=0xc000109f70 pc=0xd86369
runtime.bgscavenge(0x0?)
    runtime/mgcscavenge.go:653 +0x3c fp=0xc000109fc8 sp=0xc000109fa0 pc=0xd868fc
runtime.gcenable.func2()
    runtime/mgc.go:201 +0x25 fp=0xc000109fe0 sp=0xc000109fc8 pc=0xd7dbc5
runtime.goexit()
    runtime/asm_amd64.s:1650 +0x1 fp=0xc000109fe8 sp=0xc000109fe0 pc=0xdcd141
created by runtime.gcenable in goroutine 1
    runtime/mgc.go:201 +0xa5
RichieSams commented 1 week ago

A new type of panic:

fatal error: runtime: name offset out of range
goroutine 1 [running, locked to thread]:
runtime.throw({0x187d65c?, 0x1850bb5?})
    runtime/panic.go:1077 +0x65 fp=0xc0000fbcf8 sp=0xc0000fbcc8 pc=0x68c7a5
runtime.resolveNameOff(0x284d14a?, 0x6c5160)
    runtime/type.go:119 +0x245 fp=0xc0000fbd50 sp=0xc0000fbcf8 pc=0x6b1765
internal/reflectlite.resolveNameOff(0x0?, 0x0?)
    runtime/runtime1.go:626 +0x13 fp=0xc0000fbd70 sp=0xc0000fbd50 pc=0x6b82d3
internal/reflectlite.rtype.nameOff(...)
    internal/reflectlite/type.go:244
internal/reflectlite.rtype.String({0xc000218030?})
    internal/reflectlite/type.go:256 +0x1b fp=0xc0000fbd98 sp=0xc0000fbd70 pc=0x6c4d7b
internal/reflectlite.elem(0xc0000fbdf0?)
    internal/reflectlite/type.go:320 +0x73 fp=0xc0000fbdd0 sp=0xc0000fbd98 pc=0x6c5113
internal/reflectlite.rtype.Elem({0xc000218000?})
    internal/reflectlite/type.go:324 +0x13 fp=0xc0000fbde8 sp=0xc0000fbdd0 pc=0x6c5173
errors.init()
    errors/wrap.go:136 +0x43 fp=0xc0000fbe10 sp=0xc0000fbde8 pc=0x6c90c3
runtime.doInit1(0x24bccc0)
    runtime/proc.go:6740 +0xdd fp=0xc0000fbf40 sp=0xc0000fbe10 pc=0x69bf1d
runtime.doInit(...)
    runtime/proc.go:6707
runtime.main()
    runtime/proc.go:249 +0x356 fp=0xc0000fbfe0 sp=0xc0000fbf40 pc=0x68f096
runtime.goexit()
    runtime/asm_amd64.s:1650 +0x1 fp=0xc0000fbfe8 sp=0xc0000fbfe0 pc=0x6bd241
goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    runtime/proc.go:398 +0xce fp=0xc0000f7fa8 sp=0xc0000f7f88 pc=0x68f40e
runtime.goparkunlock(...)
    runtime/proc.go:404
runtime.forcegchelper()
    runtime/proc.go:322 +0xb8 fp=0xc0000f7fe0 sp=0xc0000f7fa8 pc=0x68f298
runtime.goexit()
    runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f7fe8 sp=0xc0000f7fe0 pc=0x6bd241
created by runtime.init.6 in goroutine 1
    runtime/proc.go:310 +0x1a
goroutine 3 [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    runtime/proc.go:398 +0xce fp=0xc0000f9f78 sp=0xc0000f9f58 pc=0x68f40e
runtime.goparkunlock(...)
    runtime/proc.go:404
runtime.bgsweep(0x0?)
    runtime/mgcsweep.go:280 +0x94 fp=0xc0000f9fc8 sp=0xc0000f9f78 pc=0x678ab4
runtime.gcenable.func1()
    runtime/mgc.go:200 +0x25 fp=0xc0000f9fe0 sp=0xc0000f9fc8 pc=0x66dc25
runtime.goexit()
    runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f9fe8 sp=0xc0000f9fe0 pc=0x6bd241
created by runtime.gcenable in goroutine 1
    runtime/mgc.go:200 +0x66
goroutine 18 [GC scavenge wait]:
runtime.gopark(0xc0000480e0?, 0x1ae7600?, 0x1?, 0x0?, 0xc0002041a0?)
    runtime/proc.go:398 +0xce fp=0xc00020df70 sp=0xc00020df50 pc=0x68f40e
runtime.goparkunlock(...)
    runtime/proc.go:404
runtime.(*scavengerState).park(0x27ea440)
    runtime/mgcscavenge.go:425 +0x49 fp=0xc00020dfa0 sp=0xc00020df70 pc=0x676369
runtime.bgscavenge(0x0?)
    runtime/mgcscavenge.go:653 +0x3c fp=0xc00020dfc8 sp=0xc00020dfa0 pc=0x6768fc
runtime.gcenable.func2()
    runtime/mgc.go:201 +0x25 fp=0xc00020dfe0 sp=0xc00020dfc8 pc=0x66dbc5
runtime.goexit()
    runtime/asm_amd64.s:1650 +0x1 fp=0xc00020dfe8 sp=0xc00020dfe0 pc=0x6bd241
created by runtime.gcenable in goroutine 1
    runtime/mgc.go:201 +0xa5
cherrymui commented 1 week ago

NOTE: We compile inside docker. The output below is from running go env within the docker container that we use. We cross-compile for linux, windows, and darwin. The workers having a problem in this case are running windows/amd64.

These workers are bare metal AMD Threadripper machines running Windows 10.

Just to be clear, you're cross compiling inside docker, but running the program on real Windows machines? Do you see it only on a specific type of Windows machines? Thanks.

RichieSams commented 1 week ago

Just to be clear, you're cross compiling inside docker, but running the program on real Windows machines?

Correct. We cross compile in docker. But we're running in native Windows.

Do you see it only on a specific type of Windows machines? Thanks.

Yes. We only see it on our AMD servers. Example specs of the machines is listed above in the original post. None of our Intel-based servers are seeing these panics

Let me know if you have any other questions. :)

cherrymui commented 1 week ago

Thanks. Have you tried running the program under the race detector? Is the program using cgo? Is there a way we can reproduce the issue ourselves?

cherrymui commented 1 week ago

cc @golang/windows

RichieSams commented 1 week ago

Have you tried running the program under the race detector?

No, but that's a good idea. I'll give that a shot today

Is the program using cgo?

No, we disable it with CGO_ENABLED=0 when compiling

Is there a way we can reproduce the issue ourselves?

Unfortunately, it's a proprietary program. But given it's panic'ing on init() and in standard libraries, I'm going to see if I can create a small "hello world" test program and just run it continuously on the machines to see if I can reproduce it.

RichieSams commented 1 week ago

I'm wondering if there is some small incorrect assumption with AMD hardware in the golang runtime code? I see this other issue, which isn't the same stack trace, but is also limited to AMD hardware: https://github.com/golang/go/issues/62440

RichieSams commented 1 week ago

Some more callstacks from our servers:

fatal: morestack on g0
Exception 0x80000003 0x0 0x0 0xccfa1
PC=0xccfa1
runtime.abort()
    runtime/asm_amd64.s:1113 +0x1 fp=0xc0000f9bc0 sp=0xc0000f9bb8 pc=0xccfa1
traceback: unexpected SPWRITE function runtime.morestack
runtime.morestack()
    runtime/asm_amd64.s:560 +0x24 fp=0xc0000f9bc8 sp=0xc0000f9bc0 pc=0xcb204
goroutine 1 [running, locked to thread]:
runtime.doInit1(0x1e82a00?)
    runtime/proc.go:6711 +0x4c5 fp=0xc0000f9f40 sp=0xc0000f9f38 pc=0xac305
runtime.doInit(...)
    runtime/proc.go:6707
runtime.main()
    runtime/proc.go:249 +0x356 fp=0xc0000f9fe0 sp=0xc0000f9f40 pc=0x9f096
runtime.goexit()
    runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f9fe8 sp=0xc0000f9fe0 pc=0xcd141
goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    runtime/proc.go:398 +0xce fp=0xc0000f5fa8 sp=0xc0000f5f88 pc=0x9f40e
runtime.goparkunlock(...)
    runtime/proc.go:404
runtime.forcegchelper()
    runtime/proc.go:322 +0xb8 fp=0xc0000f5fe0 sp=0xc0000f5fa8 pc=0x9f298
runtime.goexit()
    runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f5fe8 sp=0xc0000f5fe0 pc=0xcd141
created by runtime.init.6 in goroutine 1
    runtime/proc.go:310 +0x1a
goroutine 3 [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    runtime/proc.go:398 +0xce fp=0xc0000f7f78 sp=0xc0000f7f58 pc=0x9f40e
runtime.goparkunlock(...)
    runtime/proc.go:404
runtime.bgsweep(0x0?)
    runtime/mgcsweep.go:280 +0x94 fp=0xc0000f7fc8 sp=0xc0000f7f78 pc=0x88ab4
runtime.gcenable.func1()
    runtime/mgc.go:200 +0x25 fp=0xc0000f7fe0 sp=0xc0000f7fc8 pc=0x7dc25
runtime.goexit()
    runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f7fe8 sp=0xc0000f7fe0 pc=0xcd141
created by runtime.gcenable in goroutine 1
    runtime/mgc.go:200 +0x66
goroutine 4 [GC scavenge wait]:
runtime.gopark(0xc0000440e0?, 0x14b8428?, 0x1?, 0x0?, 0xc0000f2b60?)
    runtime/proc.go:398 +0xce fp=0xc000107f70 sp=0xc000107f50 pc=0x9f40e
runtime.goparkunlock(...)
    runtime/proc.go:404
runtime.(*scavengerState).park(0x2163e20)
    runtime/mgcscavenge.go:425 +0x49 fp=0xc000107fa0 sp=0xc000107f70 pc=0x86369
runtime.bgscavenge(0x0?)
    runtime/mgcscavenge.go:653 +0x3c fp=0xc000107fc8 sp=0xc000107fa0 pc=0x868fc
runtime.gcenable.func2()
    runtime/mgc.go:201 +0x25 fp=0xc000107fe0 sp=0xc000107fc8 pc=0x7dbc5
runtime.goexit()
    runtime/asm_amd64.s:1650 +0x1 fp=0xc000107fe8 sp=0xc000107fe0 pc=0xcd141
created by runtime.gcenable in goroutine 1
    runtime/mgc.go:201 +0xa5
rax     0x17
rbx     0x2164e80
rcx     0x2164c00
rdi     0xcc34a2e000
rsi     0xc0000f9ab8
rbp     0xc0000f9bd8
rsp     0xc0000f9bb8
r8      0xc0000f9888
r9      0xc0000f9a60
r10     0x0
r11     0x246
r12     0xcc34dff6d0
r13     0x1000
r14     0x21645e0
r15     0xb2c
rip     0xccfa1
rflags  0x212
cs      0x33
fs      0x53
gs      0x2b
panic: godebug: Value of name not listed in godebugs.All: 
goroutine 1 [running]:
internal/godebug.(*Setting).Value.func1()
    internal/godebug/godebug.go:141 +0xb9
sync.(*Once).doSlow(0x28ba410?, 0xc0000f9dd8?)
    sync/once.go:74 +0xbf
sync.(*Once).Do(...)
    sync/once.go:65
internal/godebug.(*Setting).Value(0x28ba410)
    internal/godebug/godebug.go:138 +0x48
internal/intern.safeMap(...)
    internal/intern/intern.go:74
internal/intern.init()
    internal/intern/intern.go:66 +0x48
RichieSams commented 1 week ago

Another callstack:

panic: runtime error: makeslice: cap out of range
goroutine 1 [running]:
runtime/metrics.init.0()
    runtime/metrics/description.go:457 +0xa6
RichieSams commented 1 week ago 
fatal error: fault
[signal 0xc0000005 code=0x0 addr=0x1f800608 pc=0x142df7e]
goroutine 1 gp=0xc0000fc000 m=0 mp=0x3240880 [running, locked to thread]:
runtime.throw({0x22296ae?, 0xfe15f40ffe15f40f?})
    runtime/panic.go:1023 +0x65 fp=0xc00019d798 sp=0xc00019d768 pc=0x102ebc5
runtime.sigpanic()
    runtime/signal_windows.go:414 +0xd0 fp=0xc00019d7e0 sp=0xc00019d798 pc=0x1044670
golang.org/x/text/internal/language/compact.getCoreIndex.func1(0x249?)
    golang.org/x/text@v0.14.0/internal/language/compact/compact.go:32 +0x1e fp=0xc00019d800 sp=0xc00019d7e0 pc=0x142df7e
sort.Search(0xc0001ee060?, 0xc00019d848)
    sort/search.go:65 +0x46 fp=0xc00019d830 sp=0xc00019d800 pc=0x109db26
golang.org/x/text/internal/language/compact.getCoreIndex({0x0, 0x0, 0x0, 0x0, 0x0, {0x0, 0x0}})
    golang.org/x/text@v0.14.0/internal/language/compact/compact.go:31 +0x9f fp=0xc00019d888 sp=0xc00019d830 pc=0x142debf
golang.org/x/text/internal/language/compact.FromTag({0x0, 0x0, 0x0, 0x0, 0x0, {0x0, 0x0}})
    golang.org/x/text@v0.14.0/internal/language/compact/language.go:238 +0x5a5 fp=0xc00019db90 sp=0xc00019d888 pc=0x142f145
golang.org/x/text/internal/language/compact.Make({0x0, 0x0, 0x0, 0x0, 0x0, {0x0, 0x0}})
    golang.org/x/text@v0.14.0/internal/language/compact/language.go:53 +0x285 fp=0xc00019dc30 sp=0xc00019db90 pc=0x142e505
golang.org/x/text/language.makeTag(...)
    golang.org/x/text@v0.14.0/language/language.go:25
golang.org/x/text/language.CanonType.Parse(0x17, {0x2241259?, 0xc0001bc070?})
    golang.org/x/text@v0.14.0/language/parse.go:62 +0x1f7 fp=0xc00019dd58 sp=0xc00019dc30 pc=0x14343d7
golang.org/x/text/language.Parse(...)
    golang.org/x/text@v0.14.0/language/parse.go:35
golang.org/x/text/language.MustParse({0x2241259?, 0x15?})
    golang.org/x/text@v0.14.0/language/tags.go:14 +0x25 fp=0xc00019dd80 sp=0xc00019dd58 pc=0x1434645
golang.org/x/text/cases.init.0()
    golang.org/x/text@v0.14.0/cases/map.go:43 +0xa8 fp=0xc00019de20 sp=0xc00019dd80 pc=0x143aaa8
runtime.doInit1(0x2f02670)
    runtime/proc.go:7176 +0xed fp=0xc00019df50 sp=0xc00019de20 pc=0x103f82d
runtime.doInit(...)
    runtime/proc.go:7143
runtime.main()
    runtime/proc.go:253 +0x327 fp=0xc00019dfe0 sp=0xc00019df50 pc=0x1031587
runtime.goexit({})
    runtime/asm_amd64.s:1695 +0x1 fp=0xc00019dfe8 sp=0xc00019dfe0 pc=0x1062801
goroutine 18 gp=0xc0001841c0 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    runtime/proc.go:402 +0xce fp=0xc000101fa8 sp=0xc000101f88 pc=0x10318ee
runtime.goparkunlock(...)
    runtime/proc.go:408
runtime.forcegchelper()
    runtime/proc.go:326 +0xb8 fp=0xc000101fe0 sp=0xc000101fa8 pc=0x1031778
runtime.goexit({})
    runtime/asm_amd64.s:1695 +0x1 fp=0xc000101fe8 sp=0xc000101fe0 pc=0x1062801
created by runtime.init.6 in goroutine 1
    runtime/proc.go:314 +0x1a
goroutine 2 gp=0xc0000fca80 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    runtime/proc.go:402 +0xce fp=0xc00010ff80 sp=0xc00010ff60 pc=0x10318ee
runtime.goparkunlock(...)
    runtime/proc.go:408
runtime.bgsweep(0xc0000480e0)
    runtime/mgcsweep.go:278 +0x94 fp=0xc00010ffc8 sp=0xc00010ff80 pc=0x101ab14
runtime.gcenable.gowrap1()
    runtime/mgc.go:203 +0x25 fp=0xc00010ffe0 sp=0xc00010ffc8 pc=0x100f405
runtime.goexit({})
    runtime/asm_amd64.s:1695 +0x1 fp=0xc00010ffe8 sp=0xc00010ffe0 pc=0x1062801
created by runtime.gcenable in goroutine 1
    runtime/mgc.go:203 +0x66
goroutine 3 gp=0xc0000fcc40 m=nil [GC scavenge wait]:
runtime.gopark(0xc0000480e0?, 0x2508298?, 0x1?, 0x0?, 0xc0000fcc40?)
    runtime/proc.go:402 +0xce fp=0xc000111f78 sp=0xc000111f58 pc=0x10318ee
runtime.goparkunlock(...)
    runtime/proc.go:408
runtime.(*scavengerState).park(0x323ec40)
    runtime/mgcscavenge.go:425 +0x49 fp=0xc000111fa8 sp=0xc000111f78 pc=0x10184e9
runtime.bgscavenge(0xc0000480e0)
    runtime/mgcscavenge.go:653 +0x3c fp=0xc000111fc8 sp=0xc000111fa8 pc=0x1018a7c
runtime.gcenable.gowrap2()
    runtime/mgc.go:204 +0x25 fp=0xc000111fe0 sp=0xc000111fc8 pc=0x100f3a5
runtime.goexit({})
    runtime/asm_amd64.s:1695 +0x1 fp=0xc000111fe8 sp=0xc000111fe0 pc=0x1062801
created by runtime.gcenable in goroutine 1
    runtime/mgc.go:204 +0xa5
goroutine 4 gp=0xc0000fce00 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    runtime/proc.go:402 +0xce fp=0xc00010be20 sp=0xc00010be00 pc=0x10318ee
runtime.runfinq()
    runtime/mfinal.go:194 +0x107 fp=0xc00010bfe0 sp=0xc00010be20 pc=0x100e487
runtime.goexit({})
    runtime/asm_amd64.s:1695 +0x1 fp=0xc00010bfe8 sp=0xc00010bfe0 pc=0x1062801
created by runtime.createfing in goroutine 1
    runtime/mfinal.go:164 +0x3d
cherrymui commented 6 days ago

I'm wondering if there is some small incorrect assumption with AMD hardware in the golang runtime code? I see this other issue, which isn't the same stack trace, but is also limited to AMD hardware: https://github.com/golang/go/issues/62440

It could be. It could also be a kernel issue. A reproducer would be very helpful. Thanks.