runtime: fatal error: mcall called on m->g0 stack on Windows

[RichieSams]

Go version

go1.21.4

Output of go env in your module/workspace:

NOTE: We compile inside docker. The output below is from running go env within the docker container that we use. We cross-compile for linux, windows, and darwin. The workers having a problem in this case are running windows/amd64.

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21.4'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build2751320750=/tmp/go-build -gno-record-gcc-switches'

What did you do?

We have a cli program that runs on many thousands of VMs / bare metal workers as part of our larger CI setup. Recently, we've been seeing a non-trivial number of panics on a specific set of workers. These workers are bare metal AMD Threadripper machines running Windows 10. Below is some info from DXDiag. (I can get additional information if needed).

      Time of this report: 4/23/2024, 09:36:37
             Machine name: XXXXXXXX
               Machine Id: {F429D65A-427C-4F90-A587-8AB68B33FB57}
         Operating System: Windows 10 Enterprise 64-bit (10.0, Build 19045) (19041.vb_release.191206-1406)
                 Language: English (Regional Setting: English)
      System Manufacturer: Equus Computer Systems
             System Model: Nobilis
                     BIOS: F5c (type: UEFI)
                Processor: AMD Ryzen Threadripper 3990X 64-Core Processor  (64 CPUs), ~2.9GHz
                   Memory: 262144MB RAM
      Available OS Memory: 262032MB RAM
                Page File: 38200MB used, 668832MB available
              Windows Dir: C:\Windows
          DirectX Version: DirectX 12
      DX Setup Parameters: Not found
         User DPI Setting: 96 DPI (100 percent)
       System DPI Setting: 96 DPI (100 percent)
          DWM DPI Scaling: Disabled
                 Miracast: Available, no HDCP
Microsoft Graphics Hybrid: Not Supported
 DirectX Database Version: 1.0.8
           DxDiag Version: 10.00.19041.3636 64bit Unicode

We're seeing two types of panics:

1. fatal error: runtime: mcall called on m->g0 stack
2. init order seemingly not following the spec: https://go.dev/ref/spec#Package_initialization
   • init() from a package is being called before the init() of a package it depends on
   • Meaning a global variable is not yet populated
   • Leading to a nil pointer dereference panic

I have included multiple call stacks of these panics below. NOTE: these callstacks all happen on different machines. But they only happen on the same "series" of machine, which is documented above.

All the panics seem to happen during the init() phase of the runtime start-up. The panics are not super reproduce-able. I'm only really seeing it due to our scale.

What did you see happen?

Below are the callstacks from a number of panics across multiple machines.

fatal error: runtime: mcall called on m->g0 stack
runtime stack:
runtime.throw({0x1801948?, 0x2f?})
    runtime/panic.go:1077 +0x65 fp=0xc00025bc20 sp=0xc00025bbf0 pc=0x60c7a5
runtime.badmcall(0xd0?)
    runtime/proc.go:503 +0x1f fp=0xc00025bc40 sp=0xc00025bc20 pc=0x60fa9f
runtime.badmcall(0x60db19)
    <autogenerated>:1 +0x25 fp=0xc00025bc58 sp=0xc00025bc40 pc=0x641ea5
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc00025bc70
stack: frame={sp:0xc00025bc40, fp:0xc00025bc58} stack=[0x6fc804000,0x6fc9ff7b0)
goroutine 1 [running, locked to thread]:
google.golang.org/genproto/googleapis/type/expr.init()
    google.golang.org/genproto@v0.0.0-20240227224415-6ceb2ff114de/googleapis/type/expr/expr.pb.go:149 +0x69 fp=0xc00025be10 sp=0xc00025be08 pc=0xc11629
runtime.doInit1(0x243c990)
    runtime/proc.go:6740 +0xdd fp=0xc00025bf40 sp=0xc00025be10 pc=0x61bf1d
runtime.doInit(...)
    runtime/proc.go:6707
runtime.main()
    runtime/proc.go:249 +0x356 fp=0xc00025bfe0 sp=0xc00025bf40 pc=0x60f096
runtime.goexit()
    runtime/asm_amd64.s:1650 +0x1 fp=0xc00025bfe8 sp=0xc00025bfe0 pc=0x63d241
goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    runtime/proc.go:398 +0xce fp=0xc0000f5fa8 sp=0xc0000f5f88 pc=0x60f40e
runtime.goparkunlock(...)
    runtime/proc.go:404
runtime.forcegchelper()
    runtime/proc.go:322 +0xb8 fp=0xc0000f5fe0 sp=0xc0000f5fa8 pc=0x60f298
runtime.goexit()
    runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f5fe8 sp=0xc0000f5fe0 pc=0x63d241
created by runtime.init.6 in goroutine 1
    runtime/proc.go:310 +0x1a
goroutine 3 [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    runtime/proc.go:398 +0xce fp=0xc0000f7f78 sp=0xc0000f7f58 pc=0x60f40e
runtime.goparkunlock(...)
    runtime/proc.go:404
runtime.bgsweep(0x0?)
    runtime/mgcsweep.go:280 +0x94 fp=0xc0000f7fc8 sp=0xc0000f7f78 pc=0x5f8ab4
runtime.gcenable.func1()
    runtime/mgc.go:200 +0x25 fp=0xc0000f7fe0 sp=0xc0000f7fc8 pc=0x5edc25
runtime.goexit()
    runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f7fe8 sp=0xc0000f7fe0 pc=0x63d241
created by runtime.gcenable in goroutine 1
    runtime/mgc.go:200 +0x66
goroutine 18 [GC scavenge wait]:
runtime.gopark(0xc0000440e0?, 0x1a65f48?, 0x1?, 0x0?, 0xc0002041a0?)
    runtime/proc.go:398 +0xce fp=0xc00020df70 sp=0xc00020df50 pc=0x60f40e
runtime.goparkunlock(...)
    runtime/proc.go:404
runtime.(*scavengerState).park(0x2768440)
    runtime/mgcscavenge.go:425 +0x49 fp=0xc00020dfa0 sp=0xc00020df70 pc=0x5f6369
runtime.bgscavenge(0x0?)
    runtime/mgcscavenge.go:653 +0x3c fp=0xc00020dfc8 sp=0xc00020dfa0 pc=0x5f68fc
runtime.gcenable.func2()
    runtime/mgc.go:201 +0x25 fp=0xc00020dfe0 sp=0xc00020dfc8 pc=0x5edbc5
runtime.goexit()
    runtime/asm_amd64.s:1650 +0x1 fp=0xc00020dfe8 sp=0xc00020dfe0 pc=0x63d241
created by runtime.gcenable in goroutine 1
    runtime/mgc.go:201 +0xa5
goroutine 19 [finalizer wait]:
runtime.gopark(0x400000?, 0x1000f9e70?, 0x0?, 0x0?, 0x27cba60?)
    runtime/proc.go:398 +0xce fp=0xc0000f9e28 sp=0xc0000f9e08 pc=0x60f40e
runtime.runfinq()
    runtime/mfinal.go:193 +0x107 fp=0xc0000f9fe0 sp=0xc0000f9e28 pc=0x5ecce7
runtime.goexit()
    runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f9fe8 sp=0xc0000f9fe0 pc=0x63d241
created by runtime.createfing in goroutine 1
    runtime/mfinal.go:163 +0x3d

fatal error: runtime: mcall called on m->g0 stack
runtime stack:
runtime.throw({0x1522440?, 0x20?})
    /usr/local/go/src/runtime/panic.go:1047 +0x65 fp=0xc00083f748 sp=0xc00083f718 pc=0x2bc045
runtime.badmcall(0xa?)
    /usr/local/go/src/runtime/proc.go:468 +0x25 fp=0xc00083f768 sp=0xc00083f748 pc=0x2bf2c5
runtime.badmcall(0x0)
    <autogenerated>:1 +0x2c fp=0xc00083f780 sp=0xc00083f768 pc=0x2ec86c
runtime.asyncPreempt2()
    /usr/local/go/src/runtime/preempt.go:308 +0x3f fp=0xc00083f7a0 sp=0xc00083f780 pc=0x2bd27f
runtime.asyncPreempt()
    /usr/local/go/src/runtime/preempt_amd64.s:50 +0xdb fp=0xc00083f928 sp=0xc00083f7a0 pc=0x2ebbbb
github.com/dustin/go-humanize.revfmap(0xc00024f680)
    /builds/my-project/vendor/github.com/dustin/go-humanize/si.go:33 fp=0xc00083f930 sp=0xc00083f928 pc=0xe36ac0
github.com/dustin/go-humanize.init()
    /builds/my-project/vendor/github.com/dustin/go-humanize/si.go:30 +0x142b fp=0xc00083f990 sp=0xc00083f930 pc=0xe3872b
runtime.doInit(0x2156de0)
    /usr/local/go/src/runtime/proc.go:6329 +0x12d fp=0xc00083fac0 sp=0xc00083f990 pc=0x2cb2ed
runtime.doInit(0x2157c80)
    /usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083fbf0 sp=0xc00083fac0 pc=0x2cb231
runtime.doInit(0x215a020)
    /usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083fd20 sp=0xc00083fbf0 pc=0x2cb231
runtime.doInit(0x2164660)
    /usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083fe50 sp=0xc00083fd20 pc=0x2cb231
runtime.doInit(0x2151b00)
    /usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083ff80 sp=0xc00083fe50 pc=0x2cb231
runtime.main()
    /usr/local/go/src/runtime/proc.go:233 +0x1bf fp=0xc00083ffe0 sp=0xc00083ff80 pc=0x2be73f
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00083ffe8 sp=0xc00083ffe0 pc=0x2ea561
goroutine 1 [runnable, locked to thread]:
github.com/dustin/go-humanize.revfmap(0xc00024f680?)
    /builds/my-project/vendor/github.com/dustin/go-humanize/si.go:33 +0xf0 fp=0xc00083f930 sp=0xc00083f928 pc=0xe36bb0
github.com/dustin/go-humanize.init()
    /builds/my-project/vendor/github.com/dustin/go-humanize/si.go:30 +0x142b fp=0xc00083f990 sp=0xc00083f930 pc=0xe3872b
runtime.doInit(0x2156de0)
    /usr/local/go/src/runtime/proc.go:6329 +0x12d fp=0xc00083fac0 sp=0xc00083f990 pc=0x2cb2ed
runtime.doInit(0x2157c80)
    /usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083fbf0 sp=0xc00083fac0 pc=0x2cb231
runtime.doInit(0x215a020)
    /usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083fd20 sp=0xc00083fbf0 pc=0x2cb231
runtime.doInit(0x2164660)
    /usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083fe50 sp=0xc00083fd20 pc=0x2cb231
runtime.doInit(0x2151b00)
    /usr/local/go/src/runtime/proc.go:6306 +0x71 fp=0xc00083ff80 sp=0xc00083fe50 pc=0x2cb231
runtime.main()
    /usr/local/go/src/runtime/proc.go:233 +0x1bf fp=0xc00083ffe0 sp=0xc00083ff80 pc=0x2be73f
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00083ffe8 sp=0xc00083ffe0 pc=0x2ea561
goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0000f3fb0 sp=0xc0000f3f90 pc=0x2beb16
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:369
runtime.forcegchelper()
    /usr/local/go/src/runtime/proc.go:302 +0xb1 fp=0xc0000f3fe0 sp=0xc0000f3fb0 pc=0x2be9b1
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000f3fe8 sp=0xc0000f3fe0 pc=0x2ea561
created by runtime.init.6
    /usr/local/go/src/runtime/proc.go:290 +0x25
goroutine 18 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00010ff90 sp=0xc00010ff70 pc=0x2beb16
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:369
runtime.bgsweep(0x0?)
    /usr/local/go/src/runtime/mgcsweep.go:297 +0xd7 fp=0xc00010ffc8 sp=0xc00010ff90 pc=0x2a7c37
runtime.gcenable.func1()
    /usr/local/go/src/runtime/mgc.go:178 +0x26 fp=0xc00010ffe0 sp=0xc00010ffc8 pc=0x29c786
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00010ffe8 sp=0xc00010ffe0 pc=0x2ea561
created by runtime.gcenable
    /usr/local/go/src/runtime/mgc.go:178 +0x6b
goroutine 19 [GC scavenge wait]:
runtime.gopark(0xc000106000?, 0x179b290?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000111f70 sp=0xc000111f50 pc=0x2beb16
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:369
runtime.(*scavengerState).park(0x2432600)
    /usr/local/go/src/runtime/mgcscavenge.go:389 +0x53 fp=0xc000111fa0 sp=0xc000111f70 pc=0x2a5c73
runtime.bgscavenge(0x0?)
    /usr/local/go/src/runtime/mgcscavenge.go:622 +0x65 fp=0xc000111fc8 sp=0xc000111fa0 pc=0x2a6285
runtime.gcenable.func2()
    /usr/local/go/src/runtime/mgc.go:179 +0x26 fp=0xc000111fe0 sp=0xc000111fc8 pc=0x29c726
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000111fe8 sp=0xc000111fe0 pc=0x2ea561
created by runtime.gcenable
    /usr/local/go/src/runtime/mgc.go:179 +0xaa
goroutine 34 [finalizer wait]:
runtime.gopark(0x0?, 0xc0000f7e70?, 0x6b?, 0x7d?, 0xc0000f7f70?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0000f7e28 sp=0xc0000f7e08 pc=0x2beb16
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:369
runtime.runfinq()
    /usr/local/go/src/runtime/mfinal.go:180 +0x10f fp=0xc0000f7fe0 sp=0xc0000f7e28 pc=0x29b88f
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000f7fe8 sp=0xc0000f7fe0 pc=0x2ea561
created by runtime.createfing
    /usr/local/go/src/runtime/mfinal.go:157 +0x45
goroutine 35 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00010bf50 sp=0xc00010bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00010bfe0 sp=0xc00010bf50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00010bfe8 sp=0xc00010bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 3 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0000f5f50 sp=0xc0000f5f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc0000f5fe0 sp=0xc0000f5f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000f5fe8 sp=0xc0000f5fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 20 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00011df50 sp=0xc00011df30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00011dfe0 sp=0xc00011df50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00011dfe8 sp=0xc00011dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 21 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00011ff50 sp=0xc00011ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00011ffe0 sp=0xc00011ff50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00011ffe8 sp=0xc00011ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 36 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00010df50 sp=0xc00010df30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00010dfe0 sp=0xc00010df50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00010dfe8 sp=0xc00010dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 37 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000119f50 sp=0xc000119f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000119fe0 sp=0xc000119f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000119fe8 sp=0xc000119fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 4 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000505f50 sp=0xc000505f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000505fe0 sp=0xc000505f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000505fe8 sp=0xc000505fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 5 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000507f50 sp=0xc000507f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000507fe0 sp=0xc000507f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000507fe8 sp=0xc000507fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 6 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000501f50 sp=0xc000501f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000501fe0 sp=0xc000501f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000501fe8 sp=0xc000501fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 22 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000127f50 sp=0xc000127f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000127fe0 sp=0xc000127f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000127fe8 sp=0xc000127fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 23 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000129f50 sp=0xc000129f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000129fe0 sp=0xc000129f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000129fe8 sp=0xc000129fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 38 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00011bf50 sp=0xc00011bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00011bfe0 sp=0xc00011bf50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00011bfe8 sp=0xc00011bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 7 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000503f50 sp=0xc000503f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000503fe0 sp=0xc000503f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000503fe8 sp=0xc000503fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 24 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000123f50 sp=0xc000123f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000123fe0 sp=0xc000123f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000123fe8 sp=0xc000123fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 25 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000125f50 sp=0xc000125f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000125fe0 sp=0xc000125f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000125fe8 sp=0xc000125fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 8 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00050df50 sp=0xc00050df30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00050dfe0 sp=0xc00050df50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00050dfe8 sp=0xc00050dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 26 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000509f50 sp=0xc000509f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000509fe0 sp=0xc000509f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000509fe8 sp=0xc000509fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 9 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00050ff50 sp=0xc00050ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00050ffe0 sp=0xc00050ff50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00050ffe8 sp=0xc00050ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 27 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00050bf50 sp=0xc00050bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00050bfe0 sp=0xc00050bf50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00050bfe8 sp=0xc00050bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 10 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000515f50 sp=0xc000515f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000515fe0 sp=0xc000515f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000515fe8 sp=0xc000515fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 28 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000511f50 sp=0xc000511f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000511fe0 sp=0xc000511f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000511fe8 sp=0xc000511fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 11 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000517f50 sp=0xc000517f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000517fe0 sp=0xc000517f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000517fe8 sp=0xc000517fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 29 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000513f50 sp=0xc000513f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000513fe0 sp=0xc000513f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000513fe8 sp=0xc000513fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 12 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00051df50 sp=0xc00051df30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00051dfe0 sp=0xc00051df50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00051dfe8 sp=0xc00051dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 30 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000519f50 sp=0xc000519f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000519fe0 sp=0xc000519f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000519fe8 sp=0xc000519fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 13 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00051ff50 sp=0xc00051ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00051ffe0 sp=0xc00051ff50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00051ffe8 sp=0xc00051ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 31 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00051bf50 sp=0xc00051bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00051bfe0 sp=0xc00051bf50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00051bfe8 sp=0xc00051bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 14 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000525f50 sp=0xc000525f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000525fe0 sp=0xc000525f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000525fe8 sp=0xc000525fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 39 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000521f50 sp=0xc000521f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000521fe0 sp=0xc000521f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000521fe8 sp=0xc000521fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 15 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000527f50 sp=0xc000527f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000527fe0 sp=0xc000527f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000527fe8 sp=0xc000527fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 40 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000523f50 sp=0xc000523f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000523fe0 sp=0xc000523f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000523fe8 sp=0xc000523fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 16 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00052ff50 sp=0xc00052ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00052ffe0 sp=0xc00052ff50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00052ffe8 sp=0xc00052ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 41 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00052bf50 sp=0xc00052bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00052bfe0 sp=0xc00052bf50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00052bfe8 sp=0xc00052bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 50 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000531f50 sp=0xc000531f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000531fe0 sp=0xc000531f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000531fe8 sp=0xc000531fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 42 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00052df50 sp=0xc00052df30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00052dfe0 sp=0xc00052df50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00052dfe8 sp=0xc00052dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 51 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000537f50 sp=0xc000537f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000537fe0 sp=0xc000537f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000537fe8 sp=0xc000537fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 43 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000533f50 sp=0xc000533f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000533fe0 sp=0xc000533f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000533fe8 sp=0xc000533fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 52 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000539f50 sp=0xc000539f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000539fe0 sp=0xc000539f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000539fe8 sp=0xc000539fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 44 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000535f50 sp=0xc000535f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000535fe0 sp=0xc000535f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000535fe8 sp=0xc000535fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 53 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00053ff50 sp=0xc00053ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00053ffe0 sp=0xc00053ff50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00053ffe8 sp=0xc00053ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 45 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00053bf50 sp=0xc00053bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00053bfe0 sp=0xc00053bf50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00053bfe8 sp=0xc00053bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 54 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000541f50 sp=0xc000541f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000541fe0 sp=0xc000541f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000541fe8 sp=0xc000541fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 46 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00053df50 sp=0xc00053df30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00053dfe0 sp=0xc00053df50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00053dfe8 sp=0xc00053dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 55 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000547f50 sp=0xc000547f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000547fe0 sp=0xc000547f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000547fe8 sp=0xc000547fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 47 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000543f50 sp=0xc000543f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000543fe0 sp=0xc000543f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000543fe8 sp=0xc000543fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 56 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000549f50 sp=0xc000549f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000549fe0 sp=0xc000549f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000549fe8 sp=0xc000549fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 48 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000545f50 sp=0xc000545f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000545fe0 sp=0xc000545f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000545fe8 sp=0xc000545fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 57 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00054ff50 sp=0xc00054ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00054ffe0 sp=0xc00054ff50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00054ffe8 sp=0xc00054ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 49 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00054bf50 sp=0xc00054bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00054bfe0 sp=0xc00054bf50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00054bfe8 sp=0xc00054bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 58 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000551f50 sp=0xc000551f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000551fe0 sp=0xc000551f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000551fe8 sp=0xc000551fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 66 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00054df50 sp=0xc00054df30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00054dfe0 sp=0xc00054df50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00054dfe8 sp=0xc00054dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 59 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000557f50 sp=0xc000557f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000557fe0 sp=0xc000557f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000557fe8 sp=0xc000557fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 67 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000553f50 sp=0xc000553f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000553fe0 sp=0xc000553f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000553fe8 sp=0xc000553fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 60 [GC worker (idle)]:
runtime.gopark(0x1d7554aaaaf8c?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000559f50 sp=0xc000559f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000559fe0 sp=0xc000559f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000559fe8 sp=0xc000559fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 68 [GC worker (idle)]:
runtime.gopark(0x1d7554aaaaf8c?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000555f50 sp=0xc000555f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000555fe0 sp=0xc000555f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000555fe8 sp=0xc000555fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 61 [GC worker (idle)]:
runtime.gopark(0x2496880?, 0x1?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00055ff50 sp=0xc00055ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00055ffe0 sp=0xc00055ff50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00055ffe8 sp=0xc00055ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 69 [GC worker (idle)]:
runtime.gopark(0x1d7554aaaaf8c?, 0x1?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00055bf50 sp=0xc00055bf30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00055bfe0 sp=0xc00055bf50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00055bfe8 sp=0xc00055bfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 62 [GC worker (idle)]:
runtime.gopark(0x2496880?, 0x1?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000561f50 sp=0xc000561f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000561fe0 sp=0xc000561f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000561fe8 sp=0xc000561fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 70 [GC worker (idle)]:
runtime.gopark(0x1d7554aaaaf8c?, 0x1?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00055df50 sp=0xc00055df30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00055dfe0 sp=0xc00055df50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00055dfe8 sp=0xc00055dfe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 63 [GC worker (idle)]:
runtime.gopark(0x1d7554aaaaf8c?, 0x1?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000567f50 sp=0xc000567f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000567fe0 sp=0xc000567f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000567fe8 sp=0xc000567fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 71 [GC worker (idle)]:
runtime.gopark(0x2496880?, 0x1?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000563f50 sp=0xc000563f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000563fe0 sp=0xc000563f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000563fe8 sp=0xc000563fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 64 [GC worker (idle)]:
runtime.gopark(0x2496880?, 0x1?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000569f50 sp=0xc000569f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000569fe0 sp=0xc000569f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000569fe8 sp=0xc000569fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 72 [GC worker (idle)]:
runtime.gopark(0x2496880?, 0x1?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000565f50 sp=0xc000565f30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000565fe0 sp=0xc000565f50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000565fe8 sp=0xc000565fe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 65 [GC worker (idle)]:
runtime.gopark(0x2496880?, 0x1?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00056ff50 sp=0xc00056ff30 pc=0x2beb16
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00056ffe0 sp=0xc00056ff50 pc=0x29e991
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00056ffe8 sp=0xc00056ffe0 pc=0x2ea561
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1159 +0x25
goroutine 83 [select]:
runtime.gopark(0xc000571f88?, 0x3?, 0xc8?, 0xc?, 0xc000571f72?)
    /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000571df8 sp=0xc000571dd8 pc=0x2beb16
runtime.selectgo(0xc000571f88, 0xc000571f6c, 0xc0001d2500?, 0x0, 0x0?, 0x1)
    /usr/local/go/src/runtime/select.go:328 +0x7dc fp=0xc000571f38 sp=0xc000571df8 pc=0x2ccebc
go.opencensus.io/stats/view.(*worker).start(0xc0001d2500)
    /builds/my-project/vendor/go.opencensus.io/stats/view/worker.go:292 +0xad fp=0xc000571fc8 sp=0xc000571f38 pc=0x977c2d
go.opencensus.io/stats/view.init.0.func1()
    /builds/my-project/vendor/go.opencensus.io/stats/view/worker.go:34 +0x26 fp=0xc000571fe0 sp=0xc000571fc8 pc=0x976e66
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000571fe8 sp=0xc000571fe0 pc=0x2ea561
created by go.opencensus.io/stats/view.init.0
    /builds/my-project/vendor/go.opencensus.io/stats/view/worker.go:34 +0x8d

Then here are some callstacks from init's that are seemingly not following the right init order:

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x5 pc=0xb84adc]
goroutine 1 [running]:
golang.org/x/text/internal/language.addTags({0x139, 0x0, 0x0, 0x0, 0x0, {0x0, 0x0}})
    golang.org/x/text@v0.14.0/internal/language/match.go:99 +0x27c
golang.org/x/text/internal/language.Tag.Maximize(...)
    golang.org/x/text@v0.14.0/internal/language/match.go:68
golang.org/x/text/language.init.0()
    golang.org/x/text@v0.14.0/language/match.go:727 +0x26d

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x9 pc=0x132a0c0]
goroutine 1 [running]:
google.golang.org/protobuf/internal/filedesc.(*Enums).Len(0x3023620?)
    /builds/my-project/vendor/google.golang.org/protobuf/internal/filedesc/desc_list_gen.go:25
google.golang.org/protobuf/reflect/protoregistry.rangeTopLevelDescriptors({0x23b2710, 0xc0003fc000}, 0xc00035f288)
    /builds/my-project/vendor/google.golang.org/protobuf/reflect/protoregistry/registry.go:408 +0x5d
google.golang.org/protobuf/reflect/protoregistry.(*Files).RegisterFile(0xc0000081f8, {0x23b2710?, 0xc0003fc000?})
    /builds/my-project/vendor/google.golang.org/protobuf/reflect/protoregistry/registry.go:149 +0x739
google.golang.org/protobuf/internal/filedesc.Builder.Build({{0x1dafbd0, 0x2e}, {0x2d53d20, 0x1bf, 0x1bf}, 0x1, 0x0, 0x0, 0x0, {0x239d908, ...}, ...})
    /builds/my-project/vendor/google.golang.org/protobuf/internal/filedesc/build.go:112 +0x1d6
google.golang.org/protobuf/internal/filetype.Builder.Build({{{0x1dafbd0, 0x2e}, {0x2d53d20, 0x1bf, 0x1bf}, 0x1, 0x0, 0x0, 0x0, {0x0, ...}, ...}, ...})
    /builds/my-project/vendor/google.golang.org/protobuf/internal/filetype/build.go:138 +0x1b8
google.golang.org/genproto/googleapis/rpc/code.file_google_rpc_code_proto_init()
    /builds/my-project/vendor/google.golang.org/genproto/googleapis/rpc/code/code.pb.go:331 +0x198
google.golang.org/genproto/googleapis/rpc/code.init.0()
    /builds/my-project/vendor/google.golang.org/genproto/googleapis/rpc/code/code.pb.go:313 +0x17

What did you expect to see?

The program finishes the init() phase without panics

[RichieSams]

Below is my go.mod file (with a few redactions)

module my-project/v3

require (
    github.com/AzureAD/microsoft-authentication-library-for-go v1.2.2
    github.com/antlr4-go/antlr/v4 v4.13.0
    github.com/bmatcuk/doublestar v1.3.4
    github.com/bsipos/thist v1.0.0
    github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc
    github.com/dustin/go-humanize v1.0.1
    github.com/getsentry/sentry-go v0.27.0
    github.com/go-ole/go-ole v1.3.0
    github.com/gobwas/glob v0.2.3
    github.com/gofrs/flock v0.8.1
    github.com/gonum/stat v0.0.0-20181125101827-41a0da705a5b
    github.com/gorilla/handlers v1.5.2
    github.com/gorilla/mux v1.8.1
    github.com/hashicorp/go-multierror v1.1.1
    github.com/jotfs/fastcdc-go v0.2.0
    github.com/json-iterator/go v1.1.12
    github.com/juju/ratelimit v1.0.2
    github.com/karrick/godirwalk v1.16.2
    github.com/olekukonko/tablewriter v0.0.5
    github.com/pkg/errors v0.9.1
    github.com/restic/chunker v0.4.0
    github.com/satori/go.uuid v1.2.0
    github.com/schollz/progressbar/v3 v3.14.2
    github.com/sirupsen/logrus v1.9.3
    github.com/skratchdot/open-golang v0.0.0-20190104022628-a2dfa6d0dab6
    github.com/spf13/cobra v1.8.0
    github.com/spf13/viper v1.18.2
    github.com/stretchr/testify v1.9.0
    github.com/thediveo/enumflag v0.10.1
    github.com/tinylib/msgp v1.1.9
    github.com/xhit/go-str2duration/v2 v2.1.0
    internal-a v1.7.0
    internal-b v1.2.0
    internal-c v1.7.0
    internal-d/v2 v2.70.0
    go.opentelemetry.io/otel v1.25.0
    go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.25.0
    go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.25.0
    go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.25.0
    go.opentelemetry.io/otel/metric v1.25.0
    go.opentelemetry.io/otel/sdk v1.25.0
    go.opentelemetry.io/otel/sdk/metric v1.25.0
    go.opentelemetry.io/otel/trace v1.25.0
    golang.org/x/sync v0.7.0
    golang.org/x/sys v0.19.0
    golang.org/x/term v0.19.0
    golang.org/x/text v0.14.0
    google.golang.org/grpc v1.63.2
    gopkg.in/yaml.v2 v2.4.0
)

require (
    cloud.google.com/go v0.112.0 // indirect
    cloud.google.com/go/compute v1.24.0 // indirect
    cloud.google.com/go/compute/metadata v0.2.3 // indirect
    cloud.google.com/go/iam v1.1.6 // indirect
    cloud.google.com/go/storage v1.36.0 // indirect
    github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78 // indirect
    github.com/Microsoft/go-winio v0.6.0 // indirect
    github.com/Nvveen/Gotty v0.0.0-20120604004816-cd527374f1e5 // indirect
    github.com/Shopify/toxiproxy/v2 v2.4.0 // indirect
    github.com/ajstarks/svgo v0.0.0-20180226025133-644b8db467af // indirect
    github.com/aws/aws-sdk-go v1.28.9 // indirect
    github.com/beorn7/perks v1.0.1 // indirect
    github.com/cenkalti/backoff/v4 v4.3.0 // indirect
    github.com/cespare/xxhash/v2 v2.2.0 // indirect
    github.com/containerd/continuity v0.3.0 // indirect
    github.com/cpuguy83/go-md2man/v2 v2.0.3 // indirect
    github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
    github.com/docker/cli v20.10.17+incompatible // indirect
    github.com/docker/docker v20.10.7+incompatible // indirect
    github.com/docker/go-connections v0.4.0 // indirect
    github.com/docker/go-units v0.4.0 // indirect
    github.com/fatih/color v1.14.1 // indirect
    github.com/felixge/httpsnoop v1.0.4 // indirect
    github.com/fogleman/gg v1.2.1-0.20190220221249-0403632d5b90 // indirect
    github.com/fsnotify/fsnotify v1.7.0 // indirect
    github.com/go-logr/logr v1.4.1 // indirect
    github.com/go-logr/stdr v1.2.2 // indirect
    github.com/gogo/protobuf v1.3.2 // indirect
    github.com/golang-jwt/jwt/v5 v5.0.0 // indirect
    github.com/golang/freetype v0.0.0-20170609003504-e2365dfdc4a0 // indirect
    github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
    github.com/golang/protobuf v1.5.4 // indirect
    github.com/gonum/blas v0.0.0-20181208220705-f22b278b28ac // indirect
    github.com/gonum/floats v0.0.0-20181209220543-c233463c7e82 // indirect
    github.com/gonum/integrate v0.0.0-20181209220457-a422b5c0fdf2 // indirect
    github.com/gonum/internal v0.0.0-20181124074243-f884aa714029 // indirect
    github.com/gonum/lapack v0.0.0-20181123203213-e4cdc5a0bff9 // indirect
    github.com/gonum/matrix v0.0.0-20181209220409-c518dec07be9 // indirect
    github.com/google/s2a-go v0.1.7 // indirect
    github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect
    github.com/google/uuid v1.6.0 // indirect
    github.com/googleapis/enterprise-certificate-proxy v0.3.2 // indirect
    github.com/googleapis/gax-go/v2 v2.12.0 // indirect
    github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.0 // indirect
    github.com/hashicorp/errwrap v1.0.0 // indirect
    github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
    github.com/hashicorp/go-hclog v1.5.0 // indirect
    github.com/hashicorp/go-retryablehttp v0.7.0 // indirect
    github.com/hashicorp/hcl v1.0.0 // indirect
    github.com/imdario/mergo v0.3.12 // indirect
    github.com/inconshreveable/mousetrap v1.1.0 // indirect
    github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af // indirect
    github.com/jung-kurt/gofpdf v1.0.3-0.20190309125859-24315acbbda5 // indirect
    github.com/kylelemons/godebug v1.1.0 // indirect
    github.com/magiconair/properties v1.8.7 // indirect
    github.com/mattn/go-colorable v0.1.13 // indirect
    github.com/mattn/go-isatty v0.0.20 // indirect
    github.com/mattn/go-runewidth v0.0.14 // indirect
    github.com/matttproud/golang_protobuf_extensions v1.0.1 // indirect
    github.com/minio/minio-go/v6 v6.0.46 // indirect
    github.com/minio/sha256-simd v0.1.1 // indirect
    github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db // indirect
    github.com/mitchellh/go-homedir v1.1.0 // indirect
    github.com/mitchellh/mapstructure v1.5.0 // indirect
    github.com/moby/term v0.0.0-20201216013528-df9cb8a40635 // indirect
    github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
    github.com/modern-go/reflect2 v1.0.2 // indirect
    github.com/opencontainers/go-digest v1.0.0 // indirect
    github.com/opencontainers/image-spec v1.0.2 // indirect
    github.com/opencontainers/runc v1.1.5 // indirect
    github.com/ory/dockertest/v3 v3.10.0 // indirect
    github.com/pelletier/go-toml/v2 v2.1.0 // indirect
    github.com/philhofer/fwd v1.1.2 // indirect
    github.com/pierrec/lz4 v2.0.5+incompatible // indirect
    github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 // indirect
    github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
    github.com/prometheus/client_golang v1.12.1 // indirect
    github.com/prometheus/client_model v0.2.0 // indirect
    github.com/prometheus/common v0.32.1 // indirect
    github.com/prometheus/procfs v0.7.3 // indirect
    github.com/rivo/uniseg v0.4.7 // indirect
    github.com/russross/blackfriday/v2 v2.1.0 // indirect
    github.com/sagikazarmark/locafero v0.4.0 // indirect
    github.com/sagikazarmark/slog-shim v0.1.0 // indirect
    github.com/sourcegraph/conc v0.3.0 // indirect
    github.com/spf13/afero v1.11.0 // indirect
    github.com/spf13/cast v1.6.0 // indirect
    github.com/spf13/pflag v1.0.5 // indirect
    github.com/subosito/gotenv v1.6.0 // indirect
    github.com/xeipuuv/gojsonpointer v0.0.0-20180127040702-4e3ac2762d5f // indirect
    github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 // indirect
    github.com/xeipuuv/gojsonschema v1.2.0 // indirect
    go.opencensus.io v0.24.0 // indirect
    go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.47.0 // indirect
    go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptrace v0.30.0 // indirect
    go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.47.0 // indirect
    go.opentelemetry.io/proto/otlp v1.1.0 // indirect
    go.uber.org/atomic v1.9.0 // indirect
    go.uber.org/multierr v1.9.0 // indirect
    golang.org/x/crypto v0.21.0 // indirect
    golang.org/x/exp v0.0.0-20230905200255-921286631fa9 // indirect
    golang.org/x/image v0.0.0-20190802002840-cff245a6509b // indirect
    golang.org/x/mod v0.13.0 // indirect
    golang.org/x/net v0.23.0 // indirect
    golang.org/x/oauth2 v0.17.0 // indirect
    golang.org/x/time v0.5.0 // indirect
    golang.org/x/tools v0.14.0 // indirect
    gonum.org/v1/plot v0.0.0-20190515093506-e2840ee46a6b // indirect
    google.golang.org/api v0.162.0 // indirect
    google.golang.org/appengine v1.6.8 // indirect
    google.golang.org/genproto v0.0.0-20240227224415-6ceb2ff114de // indirect
    google.golang.org/genproto/googleapis/api v0.0.0-20240227224415-6ceb2ff114de // indirect
    google.golang.org/genproto/googleapis/rpc v0.0.0-20240401170217-c3f982113cda // indirect
    google.golang.org/protobuf v1.33.0 // indirect
    gopkg.in/ini.v1 v1.67.0 // indirect
    gopkg.in/yaml.v3 v3.0.1 // indirect
    nhooyr.io/websocket v1.8.10 // indirect
)

replace github.com/antlr/antlr4 v0.0.0-20180905200841-432022fc1ca0 => github.com/michilu/antlr4 v0.0.0-20180803091604-411960b1878f

replace github.com/uber/jaeger-lib v2.0.0+incompatible => github.com/uber/jaeger-lib v1.5.0

replace github.com/skratchdot/open-golang => github.com/seanhoughton/open-golang v0.0.0-20190322203133-7972e18a724d

replace git.apache.org/thrift.git => github.com/apache/thrift v0.12.0

go 1.21

[RichieSams]

This seems to be related to: https://github.com/golang/go/issues/56774

But in that issue, the poster said updating to 1.21.3 fixed it. However, we're using 1.21.4, and still seeing the issue.

[RichieSams]

Slightly different panic:

unexpected fault address 0x411a89e
fatal error: fault
[signal 0xc0000005 code=0x0 addr=0x411a89e pc=0xe003e1]
goroutine 1 [running, locked to thread]:
runtime.throw({0x1f23e6f?, 0x0?})
    runtime/panic.go:1077 +0x65 fp=0xc0000fbd58 sp=0xc0000fbd28 pc=0xd9c7a5
runtime.sigpanic()
    runtime/signal_windows.go:377 +0xd0 fp=0xc0000fbda0 sp=0xc0000fbd58 pc=0xdb06b0
time.map.init.1()
    time/zoneinfo_abbrs_windows.go:154 +0x81 fp=0xc0000fbe00 sp=0xc0000fbda0 pc=0xe003e1
time.init()
    time/zoneinfo_abbrs_windows.go:15 +0x25 fp=0xc0000fbe10 sp=0xc0000fbe00 pc=0xe001a5
runtime.doInit1(0x2b81dc0)
    runtime/proc.go:6740 +0xdd fp=0xc0000fbf40 sp=0xc0000fbe10 pc=0xdabf1d
runtime.doInit(...)
    runtime/proc.go:6707
runtime.main()
    runtime/proc.go:249 +0x356 fp=0xc0000fbfe0 sp=0xc0000fbf40 pc=0xd9f096
runtime.goexit()
    runtime/asm_amd64.s:1650 +0x1 fp=0xc0000fbfe8 sp=0xc0000fbfe0 pc=0xdcd141
goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    runtime/proc.go:398 +0xce fp=0xc0000f7fa8 sp=0xc0000f7f88 pc=0xd9f40e
runtime.goparkunlock(...)
    runtime/proc.go:404
runtime.forcegchelper()
    runtime/proc.go:322 +0xb8 fp=0xc0000f7fe0 sp=0xc0000f7fa8 pc=0xd9f298
runtime.goexit()
    runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f7fe8 sp=0xc0000f7fe0 pc=0xdcd141
created by runtime.init.6 in goroutine 1
    runtime/proc.go:310 +0x1a
goroutine 3 [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    runtime/proc.go:398 +0xce fp=0xc0000f9f78 sp=0xc0000f9f58 pc=0xd9f40e
runtime.goparkunlock(...)
    runtime/proc.go:404
runtime.bgsweep(0x0?)
    runtime/mgcsweep.go:280 +0x94 fp=0xc0000f9fc8 sp=0xc0000f9f78 pc=0xd88ab4
runtime.gcenable.func1()
    runtime/mgc.go:200 +0x25 fp=0xc0000f9fe0 sp=0xc0000f9fc8 pc=0xd7dc25
runtime.goexit()
    runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f9fe8 sp=0xc0000f9fe0 pc=0xdcd141
created by runtime.gcenable in goroutine 1
    runtime/mgc.go:200 +0x66
goroutine 4 [GC scavenge wait]:
runtime.gopark(0xc0000480e0?, 0x21b8428?, 0x1?, 0x0?, 0xc0000f4b60?)
    runtime/proc.go:398 +0xce fp=0xc000109f70 sp=0xc000109f50 pc=0xd9f40e
runtime.goparkunlock(...)
    runtime/proc.go:404
runtime.(*scavengerState).park(0x2e63e20)
    runtime/mgcscavenge.go:425 +0x49 fp=0xc000109fa0 sp=0xc000109f70 pc=0xd86369
runtime.bgscavenge(0x0?)
    runtime/mgcscavenge.go:653 +0x3c fp=0xc000109fc8 sp=0xc000109fa0 pc=0xd868fc
runtime.gcenable.func2()
    runtime/mgc.go:201 +0x25 fp=0xc000109fe0 sp=0xc000109fc8 pc=0xd7dbc5
runtime.goexit()
    runtime/asm_amd64.s:1650 +0x1 fp=0xc000109fe8 sp=0xc000109fe0 pc=0xdcd141
created by runtime.gcenable in goroutine 1
    runtime/mgc.go:201 +0xa5

[RichieSams]

A new type of panic:

fatal error: runtime: name offset out of range
goroutine 1 [running, locked to thread]:
runtime.throw({0x187d65c?, 0x1850bb5?})
    runtime/panic.go:1077 +0x65 fp=0xc0000fbcf8 sp=0xc0000fbcc8 pc=0x68c7a5
runtime.resolveNameOff(0x284d14a?, 0x6c5160)
    runtime/type.go:119 +0x245 fp=0xc0000fbd50 sp=0xc0000fbcf8 pc=0x6b1765
internal/reflectlite.resolveNameOff(0x0?, 0x0?)
    runtime/runtime1.go:626 +0x13 fp=0xc0000fbd70 sp=0xc0000fbd50 pc=0x6b82d3
internal/reflectlite.rtype.nameOff(...)
    internal/reflectlite/type.go:244
internal/reflectlite.rtype.String({0xc000218030?})
    internal/reflectlite/type.go:256 +0x1b fp=0xc0000fbd98 sp=0xc0000fbd70 pc=0x6c4d7b
internal/reflectlite.elem(0xc0000fbdf0?)
    internal/reflectlite/type.go:320 +0x73 fp=0xc0000fbdd0 sp=0xc0000fbd98 pc=0x6c5113
internal/reflectlite.rtype.Elem({0xc000218000?})
    internal/reflectlite/type.go:324 +0x13 fp=0xc0000fbde8 sp=0xc0000fbdd0 pc=0x6c5173
errors.init()
    errors/wrap.go:136 +0x43 fp=0xc0000fbe10 sp=0xc0000fbde8 pc=0x6c90c3
runtime.doInit1(0x24bccc0)
    runtime/proc.go:6740 +0xdd fp=0xc0000fbf40 sp=0xc0000fbe10 pc=0x69bf1d
runtime.doInit(...)
    runtime/proc.go:6707
runtime.main()
    runtime/proc.go:249 +0x356 fp=0xc0000fbfe0 sp=0xc0000fbf40 pc=0x68f096
runtime.goexit()
    runtime/asm_amd64.s:1650 +0x1 fp=0xc0000fbfe8 sp=0xc0000fbfe0 pc=0x6bd241
goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    runtime/proc.go:398 +0xce fp=0xc0000f7fa8 sp=0xc0000f7f88 pc=0x68f40e
runtime.goparkunlock(...)
    runtime/proc.go:404
runtime.forcegchelper()
    runtime/proc.go:322 +0xb8 fp=0xc0000f7fe0 sp=0xc0000f7fa8 pc=0x68f298
runtime.goexit()
    runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f7fe8 sp=0xc0000f7fe0 pc=0x6bd241
created by runtime.init.6 in goroutine 1
    runtime/proc.go:310 +0x1a
goroutine 3 [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    runtime/proc.go:398 +0xce fp=0xc0000f9f78 sp=0xc0000f9f58 pc=0x68f40e
runtime.goparkunlock(...)
    runtime/proc.go:404
runtime.bgsweep(0x0?)
    runtime/mgcsweep.go:280 +0x94 fp=0xc0000f9fc8 sp=0xc0000f9f78 pc=0x678ab4
runtime.gcenable.func1()
    runtime/mgc.go:200 +0x25 fp=0xc0000f9fe0 sp=0xc0000f9fc8 pc=0x66dc25
runtime.goexit()
    runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f9fe8 sp=0xc0000f9fe0 pc=0x6bd241
created by runtime.gcenable in goroutine 1
    runtime/mgc.go:200 +0x66
goroutine 18 [GC scavenge wait]:
runtime.gopark(0xc0000480e0?, 0x1ae7600?, 0x1?, 0x0?, 0xc0002041a0?)
    runtime/proc.go:398 +0xce fp=0xc00020df70 sp=0xc00020df50 pc=0x68f40e
runtime.goparkunlock(...)
    runtime/proc.go:404
runtime.(*scavengerState).park(0x27ea440)
    runtime/mgcscavenge.go:425 +0x49 fp=0xc00020dfa0 sp=0xc00020df70 pc=0x676369
runtime.bgscavenge(0x0?)
    runtime/mgcscavenge.go:653 +0x3c fp=0xc00020dfc8 sp=0xc00020dfa0 pc=0x6768fc
runtime.gcenable.func2()
    runtime/mgc.go:201 +0x25 fp=0xc00020dfe0 sp=0xc00020dfc8 pc=0x66dbc5
runtime.goexit()
    runtime/asm_amd64.s:1650 +0x1 fp=0xc00020dfe8 sp=0xc00020dfe0 pc=0x6bd241
created by runtime.gcenable in goroutine 1
    runtime/mgc.go:201 +0xa5

[cherrymui]

NOTE: We compile inside docker. The output below is from running go env within the docker container that we use. We cross-compile for linux, windows, and darwin. The workers having a problem in this case are running windows/amd64.

These workers are bare metal AMD Threadripper machines running Windows 10.

Just to be clear, you're cross compiling inside docker, but running the program on real Windows machines? Do you see it only on a specific type of Windows machines? Thanks.

[RichieSams]

Just to be clear, you're cross compiling inside docker, but running the program on real Windows machines?

Correct. We cross compile in docker. But we're running in native Windows.

Do you see it only on a specific type of Windows machines? Thanks.

Yes. We only see it on our AMD servers. Example specs of the machines is listed above in the original post. None of our Intel-based servers are seeing these panics

Let me know if you have any other questions. :)

[cherrymui]

Thanks. Have you tried running the program under the race detector? Is the program using cgo? Is there a way we can reproduce the issue ourselves?

[cherrymui]

cc @golang/windows

[RichieSams]

Have you tried running the program under the race detector?

No, but that's a good idea. I'll give that a shot today

Is the program using cgo?

No, we disable it with CGO_ENABLED=0 when compiling

Is there a way we can reproduce the issue ourselves?

Unfortunately, it's a proprietary program. But given it's panic'ing on init() and in standard libraries, I'm going to see if I can create a small "hello world" test program and just run it continuously on the machines to see if I can reproduce it.

[RichieSams]

I'm wondering if there is some small incorrect assumption with AMD hardware in the golang runtime code? I see this other issue, which isn't the same stack trace, but is also limited to AMD hardware: https://github.com/golang/go/issues/62440

[RichieSams]

Some more callstacks from our servers:

fatal: morestack on g0
Exception 0x80000003 0x0 0x0 0xccfa1
PC=0xccfa1
runtime.abort()
    runtime/asm_amd64.s:1113 +0x1 fp=0xc0000f9bc0 sp=0xc0000f9bb8 pc=0xccfa1
traceback: unexpected SPWRITE function runtime.morestack
runtime.morestack()
    runtime/asm_amd64.s:560 +0x24 fp=0xc0000f9bc8 sp=0xc0000f9bc0 pc=0xcb204
goroutine 1 [running, locked to thread]:
runtime.doInit1(0x1e82a00?)
    runtime/proc.go:6711 +0x4c5 fp=0xc0000f9f40 sp=0xc0000f9f38 pc=0xac305
runtime.doInit(...)
    runtime/proc.go:6707
runtime.main()
    runtime/proc.go:249 +0x356 fp=0xc0000f9fe0 sp=0xc0000f9f40 pc=0x9f096
runtime.goexit()
    runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f9fe8 sp=0xc0000f9fe0 pc=0xcd141
goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    runtime/proc.go:398 +0xce fp=0xc0000f5fa8 sp=0xc0000f5f88 pc=0x9f40e
runtime.goparkunlock(...)
    runtime/proc.go:404
runtime.forcegchelper()
    runtime/proc.go:322 +0xb8 fp=0xc0000f5fe0 sp=0xc0000f5fa8 pc=0x9f298
runtime.goexit()
    runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f5fe8 sp=0xc0000f5fe0 pc=0xcd141
created by runtime.init.6 in goroutine 1
    runtime/proc.go:310 +0x1a
goroutine 3 [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    runtime/proc.go:398 +0xce fp=0xc0000f7f78 sp=0xc0000f7f58 pc=0x9f40e
runtime.goparkunlock(...)
    runtime/proc.go:404
runtime.bgsweep(0x0?)
    runtime/mgcsweep.go:280 +0x94 fp=0xc0000f7fc8 sp=0xc0000f7f78 pc=0x88ab4
runtime.gcenable.func1()
    runtime/mgc.go:200 +0x25 fp=0xc0000f7fe0 sp=0xc0000f7fc8 pc=0x7dc25
runtime.goexit()
    runtime/asm_amd64.s:1650 +0x1 fp=0xc0000f7fe8 sp=0xc0000f7fe0 pc=0xcd141
created by runtime.gcenable in goroutine 1
    runtime/mgc.go:200 +0x66
goroutine 4 [GC scavenge wait]:
runtime.gopark(0xc0000440e0?, 0x14b8428?, 0x1?, 0x0?, 0xc0000f2b60?)
    runtime/proc.go:398 +0xce fp=0xc000107f70 sp=0xc000107f50 pc=0x9f40e
runtime.goparkunlock(...)
    runtime/proc.go:404
runtime.(*scavengerState).park(0x2163e20)
    runtime/mgcscavenge.go:425 +0x49 fp=0xc000107fa0 sp=0xc000107f70 pc=0x86369
runtime.bgscavenge(0x0?)
    runtime/mgcscavenge.go:653 +0x3c fp=0xc000107fc8 sp=0xc000107fa0 pc=0x868fc
runtime.gcenable.func2()
    runtime/mgc.go:201 +0x25 fp=0xc000107fe0 sp=0xc000107fc8 pc=0x7dbc5
runtime.goexit()
    runtime/asm_amd64.s:1650 +0x1 fp=0xc000107fe8 sp=0xc000107fe0 pc=0xcd141
created by runtime.gcenable in goroutine 1
    runtime/mgc.go:201 +0xa5
rax     0x17
rbx     0x2164e80
rcx     0x2164c00
rdi     0xcc34a2e000
rsi     0xc0000f9ab8
rbp     0xc0000f9bd8
rsp     0xc0000f9bb8
r8      0xc0000f9888
r9      0xc0000f9a60
r10     0x0
r11     0x246
r12     0xcc34dff6d0
r13     0x1000
r14     0x21645e0
r15     0xb2c
rip     0xccfa1
rflags  0x212
cs      0x33
fs      0x53
gs      0x2b

panic: godebug: Value of name not listed in godebugs.All: 
goroutine 1 [running]:
internal/godebug.(*Setting).Value.func1()
    internal/godebug/godebug.go:141 +0xb9
sync.(*Once).doSlow(0x28ba410?, 0xc0000f9dd8?)
    sync/once.go:74 +0xbf
sync.(*Once).Do(...)
    sync/once.go:65
internal/godebug.(*Setting).Value(0x28ba410)
    internal/godebug/godebug.go:138 +0x48
internal/intern.safeMap(...)
    internal/intern/intern.go:74
internal/intern.init()
    internal/intern/intern.go:66 +0x48

[RichieSams]

Another callstack:

panic: runtime error: makeslice: cap out of range
goroutine 1 [running]:
runtime/metrics.init.0()
    runtime/metrics/description.go:457 +0xa6

[RichieSams]

fatal error: fault
[signal 0xc0000005 code=0x0 addr=0x1f800608 pc=0x142df7e]
goroutine 1 gp=0xc0000fc000 m=0 mp=0x3240880 [running, locked to thread]:
runtime.throw({0x22296ae?, 0xfe15f40ffe15f40f?})
    runtime/panic.go:1023 +0x65 fp=0xc00019d798 sp=0xc00019d768 pc=0x102ebc5
runtime.sigpanic()
    runtime/signal_windows.go:414 +0xd0 fp=0xc00019d7e0 sp=0xc00019d798 pc=0x1044670
golang.org/x/text/internal/language/compact.getCoreIndex.func1(0x249?)
    golang.org/x/text@v0.14.0/internal/language/compact/compact.go:32 +0x1e fp=0xc00019d800 sp=0xc00019d7e0 pc=0x142df7e
sort.Search(0xc0001ee060?, 0xc00019d848)
    sort/search.go:65 +0x46 fp=0xc00019d830 sp=0xc00019d800 pc=0x109db26
golang.org/x/text/internal/language/compact.getCoreIndex({0x0, 0x0, 0x0, 0x0, 0x0, {0x0, 0x0}})
    golang.org/x/text@v0.14.0/internal/language/compact/compact.go:31 +0x9f fp=0xc00019d888 sp=0xc00019d830 pc=0x142debf
golang.org/x/text/internal/language/compact.FromTag({0x0, 0x0, 0x0, 0x0, 0x0, {0x0, 0x0}})
    golang.org/x/text@v0.14.0/internal/language/compact/language.go:238 +0x5a5 fp=0xc00019db90 sp=0xc00019d888 pc=0x142f145
golang.org/x/text/internal/language/compact.Make({0x0, 0x0, 0x0, 0x0, 0x0, {0x0, 0x0}})
    golang.org/x/text@v0.14.0/internal/language/compact/language.go:53 +0x285 fp=0xc00019dc30 sp=0xc00019db90 pc=0x142e505
golang.org/x/text/language.makeTag(...)
    golang.org/x/text@v0.14.0/language/language.go:25
golang.org/x/text/language.CanonType.Parse(0x17, {0x2241259?, 0xc0001bc070?})
    golang.org/x/text@v0.14.0/language/parse.go:62 +0x1f7 fp=0xc00019dd58 sp=0xc00019dc30 pc=0x14343d7
golang.org/x/text/language.Parse(...)
    golang.org/x/text@v0.14.0/language/parse.go:35
golang.org/x/text/language.MustParse({0x2241259?, 0x15?})
    golang.org/x/text@v0.14.0/language/tags.go:14 +0x25 fp=0xc00019dd80 sp=0xc00019dd58 pc=0x1434645
golang.org/x/text/cases.init.0()
    golang.org/x/text@v0.14.0/cases/map.go:43 +0xa8 fp=0xc00019de20 sp=0xc00019dd80 pc=0x143aaa8
runtime.doInit1(0x2f02670)
    runtime/proc.go:7176 +0xed fp=0xc00019df50 sp=0xc00019de20 pc=0x103f82d
runtime.doInit(...)
    runtime/proc.go:7143
runtime.main()
    runtime/proc.go:253 +0x327 fp=0xc00019dfe0 sp=0xc00019df50 pc=0x1031587
runtime.goexit({})
    runtime/asm_amd64.s:1695 +0x1 fp=0xc00019dfe8 sp=0xc00019dfe0 pc=0x1062801
goroutine 18 gp=0xc0001841c0 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    runtime/proc.go:402 +0xce fp=0xc000101fa8 sp=0xc000101f88 pc=0x10318ee
runtime.goparkunlock(...)
    runtime/proc.go:408
runtime.forcegchelper()
    runtime/proc.go:326 +0xb8 fp=0xc000101fe0 sp=0xc000101fa8 pc=0x1031778
runtime.goexit({})
    runtime/asm_amd64.s:1695 +0x1 fp=0xc000101fe8 sp=0xc000101fe0 pc=0x1062801
created by runtime.init.6 in goroutine 1
    runtime/proc.go:314 +0x1a
goroutine 2 gp=0xc0000fca80 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    runtime/proc.go:402 +0xce fp=0xc00010ff80 sp=0xc00010ff60 pc=0x10318ee
runtime.goparkunlock(...)
    runtime/proc.go:408
runtime.bgsweep(0xc0000480e0)
    runtime/mgcsweep.go:278 +0x94 fp=0xc00010ffc8 sp=0xc00010ff80 pc=0x101ab14
runtime.gcenable.gowrap1()
    runtime/mgc.go:203 +0x25 fp=0xc00010ffe0 sp=0xc00010ffc8 pc=0x100f405
runtime.goexit({})
    runtime/asm_amd64.s:1695 +0x1 fp=0xc00010ffe8 sp=0xc00010ffe0 pc=0x1062801
created by runtime.gcenable in goroutine 1
    runtime/mgc.go:203 +0x66
goroutine 3 gp=0xc0000fcc40 m=nil [GC scavenge wait]:
runtime.gopark(0xc0000480e0?, 0x2508298?, 0x1?, 0x0?, 0xc0000fcc40?)
    runtime/proc.go:402 +0xce fp=0xc000111f78 sp=0xc000111f58 pc=0x10318ee
runtime.goparkunlock(...)
    runtime/proc.go:408
runtime.(*scavengerState).park(0x323ec40)
    runtime/mgcscavenge.go:425 +0x49 fp=0xc000111fa8 sp=0xc000111f78 pc=0x10184e9
runtime.bgscavenge(0xc0000480e0)
    runtime/mgcscavenge.go:653 +0x3c fp=0xc000111fc8 sp=0xc000111fa8 pc=0x1018a7c
runtime.gcenable.gowrap2()
    runtime/mgc.go:204 +0x25 fp=0xc000111fe0 sp=0xc000111fc8 pc=0x100f3a5
runtime.goexit({})
    runtime/asm_amd64.s:1695 +0x1 fp=0xc000111fe8 sp=0xc000111fe0 pc=0x1062801
created by runtime.gcenable in goroutine 1
    runtime/mgc.go:204 +0xa5
goroutine 4 gp=0xc0000fce00 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    runtime/proc.go:402 +0xce fp=0xc00010be20 sp=0xc00010be00 pc=0x10318ee
runtime.runfinq()
    runtime/mfinal.go:194 +0x107 fp=0xc00010bfe0 sp=0xc00010be20 pc=0x100e487
runtime.goexit({})
    runtime/asm_amd64.s:1695 +0x1 fp=0xc00010bfe8 sp=0xc00010bfe0 pc=0x1062801
created by runtime.createfing in goroutine 1
    runtime/mfinal.go:164 +0x3d

[cherrymui]

I'm wondering if there is some small incorrect assumption with AMD hardware in the golang runtime code? I see this other issue, which isn't the same stack trace, but is also limited to AMD hardware: https://github.com/golang/go/issues/62440

It could be. It could also be a kernel issue. A reproducer would be very helpful. Thanks.

[RichieSams]

I'm attempting to get a minimal repro. I created something and I'm running it on all our workers periodically to try to get a failure. I'll report back here if I can get it to trigger.

[gopherbot]

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

[distancesprinter]

I am experiencing this issue and can reproduce. It occurs intermittently when I attempt to use backup tool restic. Let me know what needs to be done and I will see if I can help.

My hardware is: AMD EPYC 7543P 32-Core Processor

PS C:\restic> .\restic_0.17.0_windows_amd64.exe
fatal error: runtime: mcall called on m->g0 stack

runtime stack:
runtime.throw({0x18d2ad1?, 0x16fd540?})
        /usr/local/go/src/runtime/panic.go:1023 +0x65 fp=0xc00035fba0 sp=0xc00035fb70 pc=0xa6d405
runtime.badmcall(0xc00035fbf8?)
        /usr/local/go/src/runtime/proc.go:507 +0x1f fp=0xc00035fbc0 sp=0xc00035fba0 pc=0xa707bf
runtime.badmcall(0xa6e899)
        <autogenerated>:1 +0x25 fp=0xc00035fbd8 sp=0xc00035fbc0 pc=0xaa5be5
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc00035fbf0
stack: frame={sp:0xc00035fbc0, fp:0xc00035fbd8} stack=[0x15ed004000,0x15ed1ff850)

goroutine 1 gp=0xc000066000 m=nil [runnable, locked to thread]:
time.NewTicker(0x2540be400?)
        /usr/local/go/src/time/tick.go:20 +0xf0 fp=0xc00035fd90 sp=0xc00035fd88 pc=0xaccc90
go.opencensus.io/stats/view.NewMeter()
        /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:266 +0x45 fp=0xc00035fdf0 sp=0xc00035fd90 pc=0x1444b05
go.opencensus.io/stats/view.init.0()
        /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:33 +0x17 fp=0xc00035fe20 sp=0xc00035fdf0 pc=0x1444037
runtime.doInit1(0x2253870)
        /usr/local/go/src/runtime/proc.go:7176 +0xed fp=0xc00035ff50 sp=0xc00035fe20 pc=0xa7e06d
runtime.doInit(...)
        /usr/local/go/src/runtime/proc.go:7143
runtime.main()
        /usr/local/go/src/runtime/proc.go:253 +0x327 fp=0xc00035ffe0 sp=0xc00035ff50 pc=0xa6fdc7
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00035ffe8 sp=0xc00035ffe0 pc=0xaa0ea1

goroutine 2 gp=0xc000066700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000069fa8 sp=0xc000069f88 pc=0xa7012e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:408
runtime.forcegchelper()
        /usr/local/go/src/runtime/proc.go:326 +0xb8 fp=0xc000069fe0 sp=0xc000069fa8 pc=0xa6ffb8
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000069fe8 sp=0xc000069fe0 pc=0xaa0ea1
created by runtime.init.6 in goroutine 1
        /usr/local/go/src/runtime/proc.go:314 +0x1a

goroutine 3 gp=0xc000066a80 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006bf80 sp=0xc00006bf60 pc=0xa7012e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:408
runtime.bgsweep(0xc000074000)
        /usr/local/go/src/runtime/mgcsweep.go:278 +0x94 fp=0xc00006bfc8 sp=0xc00006bf80 pc=0xa5a2d4
runtime.gcenable.gowrap1()
        /usr/local/go/src/runtime/mgc.go:203 +0x25 fp=0xc00006bfe0 sp=0xc00006bfc8 pc=0xa4eb45
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0xaa0ea1
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000066c40 m=nil [GC scavenge wait]:
runtime.gopark(0xc000074000?, 0x1a7dd08?, 0x1?, 0x0?, 0xc000066c40?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00007bf78 sp=0xc00007bf58 pc=0xa7012e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:408
runtime.(*scavengerState).park(0x22f43c0)
        /usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00007bfa8 sp=0xc00007bf78 pc=0xa57cc9
runtime.bgscavenge(0xc000074000)
        /usr/local/go/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc00007bfc8 sp=0xc00007bfa8 pc=0xa5825c
runtime.gcenable.gowrap2()
        /usr/local/go/src/runtime/mgc.go:204 +0x25 fp=0xc00007bfe0 sp=0xc00007bfc8 pc=0xa4eae5
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00007bfe8 sp=0xc00007bfe0 pc=0xaa0ea1
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:204 +0xa5

goroutine 18 gp=0xc0001041c0 m=nil [finalizer wait]:
runtime.gopark(0xc00006de60?, 0xa570fc?, 0xc0?, 0xe5?, 0x550011?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006de20 sp=0xc00006de00 pc=0xa7012e
runtime.runfinq()
        /usr/local/go/src/runtime/mfinal.go:194 +0x107 fp=0xc00006dfe0 sp=0xc00006de20 pc=0xa4dbc7
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006dfe8 sp=0xc00006dfe0 pc=0xaa0ea1
created by runtime.createfing in goroutine 1
        /usr/local/go/src/runtime/mfinal.go:164 +0x3d
PS C:\restic>

[ianlancetaylor]

@distancesprinter Is there a way that we can reproduce the problem ourselves?

[distancesprinter]

You will have to let me know what you mean by that. You could try to download restic_0.17.0_windows_amd64 and try to run it. It seems like my issues only occur after a certain hour of the day, but I am still trying to decipher a pattern.

On Tue, Aug 20, 2024, 11:16 PM Ian Lance Taylor @.***> wrote:

@distancesprinter https://github.com/distancesprinter Is there a way that we can reproduce the problem ourselves?

— Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/67108#issuecomment-2300496792, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADFATVEAYIHWNADJXUM4VVLZSQA67AVCNFSM6AAAAABG6TTIXGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMBQGQ4TMNZZGI . You are receiving this because you were mentioned.Message ID: @.***>

[ianlancetaylor]

What I mean is a precise set of instructions that let us reliably reproduce the problem on our own systems. It sounds like we are not there yet.

[distancesprinter]

Just so you are aware, I am not a Go developer, I only arrived here by Googling my error, but I will try to be as helpful as possible.

The software I am running is restic_0.17.0_windows_amd64.exe

restic 0.17.0 compiled with go1.22.5 on windows/amd64

There is build process information listed at the following locations: https://restic.readthedocs.io/en/stable/developer_information.html https://github.com/restic/builder

To be clear, I did not build the software. I am using the official release binary.

You can download the binary from: https://github.com/restic/restic/releases/tag/v0.17.0 https://github.com/restic/restic/releases/download/v0.17.0/restic_0.17.0_windows_amd64.zip

Extract the zip to a folder; I am using C:\restic

From Windows GUI, right-click Start and choose Windows PowerShell (Admin) to launch PowerShell as an Administrator.

Then cd to C:\restic and attempt to run the binary.

I have found that after booting the OS, if I repeatedly try to run the binary, eventually it works (usually on the 5th or 6th attempt).

I am running Windows Server 2019 Standard (Desktop Experience) in a bhyve VM.

Here is some console output, showing repeated attempts to execute the binary, and eventually the expected output. I also printed systeminfo so you have detailed build and hotfix information.

Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

PS C:\Windows\system32> cd C:\restic\
PS C:\restic> .\restic_0.17.0_windows_amd64.exe
fatal error: runtime: mcall called on m->g0 stack

runtime stack:
runtime.throw({0x1722ad1?, 0x10?})
        /usr/local/go/src/runtime/panic.go:1023 +0x65 fp=0xc000145c30 sp=0xc000145c00 pc=0x8bd405
runtime.badmcall(0x89a265?)
        /usr/local/go/src/runtime/proc.go:507 +0x1f fp=0xc000145c50 sp=0xc000145c30 pc=0x8c07bf
runtime.badmcall(0x8be899)
        <autogenerated>:1 +0x25 fp=0xc000145c68 sp=0xc000145c50 pc=0x8f5be5
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc000145c80
stack: frame={sp:0xc000145c50, fp:0xc000145c68} stack=[0x7d0ec04000,0x7d0edff710)

goroutine 1 gp=0xc000066000 m=nil [runnable, locked to thread]:
golang.org/x/text/encoding/unicode.init()
        <autogenerated>:1 +0x1c6 fp=0xc000145e20 sp=0xc000145e18 pc=0xe92906
runtime.doInit1(0x20a1ed0)
        /usr/local/go/src/runtime/proc.go:7176 +0xed fp=0xc000145f50 sp=0xc000145e20 pc=0x8ce06d
runtime.doInit(...)
        /usr/local/go/src/runtime/proc.go:7143
runtime.main()
        /usr/local/go/src/runtime/proc.go:253 +0x327 fp=0xc000145fe0 sp=0xc000145f50 pc=0x8bfdc7
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000145fe8 sp=0xc000145fe0 pc=0x8f0ea1

goroutine 2 gp=0xc000066700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000069fa8 sp=0xc000069f88 pc=0x8c012e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:408
runtime.forcegchelper()
        /usr/local/go/src/runtime/proc.go:326 +0xb8 fp=0xc000069fe0 sp=0xc000069fa8 pc=0x8bffb8
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000069fe8 sp=0xc000069fe0 pc=0x8f0ea1
created by runtime.init.6 in goroutine 1
        /usr/local/go/src/runtime/proc.go:314 +0x1a

goroutine 3 gp=0xc000066a80 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006bf80 sp=0xc00006bf60 pc=0x8c012e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:408
runtime.bgsweep(0xc000074000)
        /usr/local/go/src/runtime/mgcsweep.go:278 +0x94 fp=0xc00006bfc8 sp=0xc00006bf80 pc=0x8aa2d4
runtime.gcenable.gowrap1()
        /usr/local/go/src/runtime/mgc.go:203 +0x25 fp=0xc00006bfe0 sp=0xc00006bfc8 pc=0x89eb45
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0x8f0ea1
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:203 +0x66

goroutine 18 gp=0xc0001041c0 m=nil [GC scavenge wait]:
runtime.gopark(0xc000074000?, 0x18cdd08?, 0x1?, 0x0?, 0xc0001041c0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00010df78 sp=0xc00010df58 pc=0x8c012e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:408
runtime.(*scavengerState).park(0x21443c0)
        /usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00010dfa8 sp=0xc00010df78 pc=0x8a7cc9
runtime.bgscavenge(0xc000074000)
        /usr/local/go/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc00010dfc8 sp=0xc00010dfa8 pc=0x8a825c
runtime.gcenable.gowrap2()
        /usr/local/go/src/runtime/mgc.go:204 +0x25 fp=0xc00010dfe0 sp=0xc00010dfc8 pc=0x89eae5
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00010dfe8 sp=0xc00010dfe0 pc=0x8f0ea1
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:204 +0xa5

goroutine 19 gp=0xc000104380 m=nil [finalizer wait]:
runtime.gopark(0xc00006de60?, 0x8a70fc?, 0xc0?, 0xe5?, 0x550011?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006de20 sp=0xc00006de00 pc=0x8c012e
runtime.runfinq()
        /usr/local/go/src/runtime/mfinal.go:194 +0x107 fp=0xc00006dfe0 sp=0xc00006de20 pc=0x89dbc7
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006dfe8 sp=0xc00006dfe0 pc=0x8f0ea1
created by runtime.createfing in goroutine 1
        /usr/local/go/src/runtime/mfinal.go:164 +0x3d
PS C:\restic> .\restic_0.17.0_windows_amd64.exe
fatal error: runtime: mcall called on m->g0 stack

runtime stack:
runtime.throw({0x1722ad1?, 0xc0003dfc60?})
        /usr/local/go/src/runtime/panic.go:1023 +0x65 fp=0xc0003dfc30 sp=0xc0003dfc00 pc=0x8bd405
runtime.badmcall(0xa300000002145fc0?)
        /usr/local/go/src/runtime/proc.go:507 +0x1f fp=0xc0003dfc50 sp=0xc0003dfc30 pc=0x8c07bf
runtime.badmcall(0x8be899)
        <autogenerated>:1 +0x25 fp=0xc0003dfc68 sp=0xc0003dfc50 pc=0x8f5be5
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc0003dfc80
stack: frame={sp:0xc0003dfc50, fp:0xc0003dfc68} stack=[0x5de7604000,0x5de77ffd20)

goroutine 1 gp=0xc000066000 m=nil [runnable, locked to thread]:
github.com/restic/restic/internal/feature.init.0()
        /restic/internal/feature/registry.go:16 +0x53e fp=0xc0003dfe20 sp=0xc0003dfe18 pc=0xc3745e
runtime.doInit1(0x20a1c70)
        /usr/local/go/src/runtime/proc.go:7176 +0xed fp=0xc0003dff50 sp=0xc0003dfe20 pc=0x8ce06d
runtime.doInit(...)
        /usr/local/go/src/runtime/proc.go:7143
runtime.main()
        /usr/local/go/src/runtime/proc.go:253 +0x327 fp=0xc0003dffe0 sp=0xc0003dff50 pc=0x8bfdc7
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0003dffe8 sp=0xc0003dffe0 pc=0x8f0ea1

goroutine 2 gp=0xc000066700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000069fa8 sp=0xc000069f88 pc=0x8c012e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:408
runtime.forcegchelper()
        /usr/local/go/src/runtime/proc.go:326 +0xb8 fp=0xc000069fe0 sp=0xc000069fa8 pc=0x8bffb8
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000069fe8 sp=0xc000069fe0 pc=0x8f0ea1
created by runtime.init.6 in goroutine 1
        /usr/local/go/src/runtime/proc.go:314 +0x1a

goroutine 3 gp=0xc000066a80 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006bf80 sp=0xc00006bf60 pc=0x8c012e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:408
runtime.bgsweep(0xc000074000)
        /usr/local/go/src/runtime/mgcsweep.go:278 +0x94 fp=0xc00006bfc8 sp=0xc00006bf80 pc=0x8aa2d4
runtime.gcenable.gowrap1()
        /usr/local/go/src/runtime/mgc.go:203 +0x25 fp=0xc00006bfe0 sp=0xc00006bfc8 pc=0x89eb45
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0x8f0ea1
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000066c40 m=nil [GC scavenge wait]:
runtime.gopark(0xc000074000?, 0x18cdd08?, 0x1?, 0x0?, 0xc000066c40?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00007bf78 sp=0xc00007bf58 pc=0x8c012e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:408
runtime.(*scavengerState).park(0x21443c0)
        /usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00007bfa8 sp=0xc00007bf78 pc=0x8a7cc9
runtime.bgscavenge(0xc000074000)
        /usr/local/go/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc00007bfc8 sp=0xc00007bfa8 pc=0x8a825c
runtime.gcenable.gowrap2()
        /usr/local/go/src/runtime/mgc.go:204 +0x25 fp=0xc00007bfe0 sp=0xc00007bfc8 pc=0x89eae5
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00007bfe8 sp=0xc00007bfe0 pc=0x8f0ea1
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:204 +0xa5

goroutine 5 gp=0xc000066fc0 m=nil [finalizer wait]:
runtime.gopark(0xc00006de60?, 0x8a70fc?, 0xc0?, 0xe5?, 0x550011?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006de20 sp=0xc00006de00 pc=0x8c012e
runtime.runfinq()
        /usr/local/go/src/runtime/mfinal.go:194 +0x107 fp=0xc00006dfe0 sp=0xc00006de20 pc=0x89dbc7
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006dfe8 sp=0xc00006dfe0 pc=0x8f0ea1
created by runtime.createfing in goroutine 1
        /usr/local/go/src/runtime/mfinal.go:164 +0x3d

goroutine 18 gp=0xc000211500 m=nil [select]:
runtime.gopark(0xc000077f78?, 0x3?, 0x8?, 0x1?, 0xc000077f72?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000077e18 sp=0xc000077df8 pc=0x8c012e
runtime.selectgo(0xc000077f78, 0xc000077f6c, 0xc000220d00?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000077f38 sp=0xc000077e18 pc=0x8d0585
go.opencensus.io/stats/view.(*worker).start(0xc000220d00)
        /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0x9f fp=0xc000077fc8 sp=0xc000077f38 pc=0x1294dff
go.opencensus.io/stats/view.init.0.gowrap1()
        /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x25 fp=0xc000077fe0 sp=0xc000077fc8 pc=0x1294165
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000077fe8 sp=0xc000077fe0 pc=0x8f0ea1
created by go.opencensus.io/stats/view.init.0 in goroutine 1
        /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x8d
PS C:\restic> .\restic_0.17.0_windows_amd64.exe
fatal: morestack on g0
Exception 0x80000003 0x0 0x0 0x8f0d01
PC=0x8f0d01

runtime.abort()
        /usr/local/go/src/runtime/asm_amd64.s:1157 +0x1 fp=0xc00063fc20 sp=0xc00063fc18 pc=0x8f0d01
runtime.morestack()
        /usr/local/go/src/runtime/asm_amd64.s:593 +0x3d fp=0xc00063fc28 sp=0xc00063fc20 pc=0x8eef7d

goroutine 1 gp=0xc000066000 m=nil [runnable]:
github.com/spf13/cobra.(*Command).ExecuteC(0x21345e0)
        /home/build/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1054 +0x625 fp=0xc00063fe00 sp=0xc00063fdf8 pc=0xc28b05
github.com/spf13/cobra.(*Command).Execute(...)
        /home/build/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041
github.com/spf13/cobra.(*Command).ExecuteContext(...)
        /home/build/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1034
main.main()
        /restic/cmd/restic/main.go:125 +0x265 fp=0xc00063ff50 sp=0xc00063fe00 pc=0x13ca3e5
runtime.main()
        /usr/local/go/src/runtime/proc.go:271 +0x28b fp=0xc00063ffe0 sp=0xc00063ff50 pc=0x8bfd2b
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00063ffe8 sp=0xc00063ffe0 pc=0x8f0ea1

goroutine 2 gp=0xc000066700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000069fa8 sp=0xc000069f88 pc=0x8c012e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:408
runtime.forcegchelper()
        /usr/local/go/src/runtime/proc.go:326 +0xb8 fp=0xc000069fe0 sp=0xc000069fa8 pc=0x8bffb8
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000069fe8 sp=0xc000069fe0 pc=0x8f0ea1
created by runtime.init.6 in goroutine 1
        /usr/local/go/src/runtime/proc.go:314 +0x1a

goroutine 3 gp=0xc000066a80 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006bf80 sp=0xc00006bf60 pc=0x8c012e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:408
runtime.bgsweep(0xc000074000)
        /usr/local/go/src/runtime/mgcsweep.go:278 +0x94 fp=0xc00006bfc8 sp=0xc00006bf80 pc=0x8aa2d4
runtime.gcenable.gowrap1()
        /usr/local/go/src/runtime/mgc.go:203 +0x25 fp=0xc00006bfe0 sp=0xc00006bfc8 pc=0x89eb45
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0x8f0ea1
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000066c40 m=nil [GC scavenge wait]:
runtime.gopark(0x10000?, 0x18cdd08?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00007bf78 sp=0xc00007bf58 pc=0x8c012e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:408
runtime.(*scavengerState).park(0x21443c0)
        /usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00007bfa8 sp=0xc00007bf78 pc=0x8a7cc9
runtime.bgscavenge(0xc000074000)
        /usr/local/go/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc00007bfc8 sp=0xc00007bfa8 pc=0x8a8279
runtime.gcenable.gowrap2()
        /usr/local/go/src/runtime/mgc.go:204 +0x25 fp=0xc00007bfe0 sp=0xc00007bfc8 pc=0x89eae5
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00007bfe8 sp=0xc00007bfe0 pc=0x8f0ea1
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:204 +0xa5

goroutine 5 gp=0xc000066fc0 m=nil [finalizer wait]:
runtime.gopark(0xc00006de60?, 0x8a70fc?, 0xc0?, 0xe5?, 0x550011?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006de20 sp=0xc00006de00 pc=0x8c012e
runtime.runfinq()
        /usr/local/go/src/runtime/mfinal.go:194 +0x107 fp=0xc00006dfe0 sp=0xc00006de20 pc=0x89dbc7
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006dfe8 sp=0xc00006dfe0 pc=0x8f0ea1
created by runtime.createfing in goroutine 1
        /usr/local/go/src/runtime/mfinal.go:164 +0x3d

goroutine 6 gp=0xc0001b76c0 m=nil [select]:
runtime.gopark(0xc00007df78?, 0x3?, 0x98?, 0x5?, 0xc00007df72?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00007de18 sp=0xc00007ddf8 pc=0x8c012e
runtime.selectgo(0xc00007df78, 0xc00007df6c, 0xc0000e5700?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc00007df38 sp=0xc00007de18 pc=0x8d0585
go.opencensus.io/stats/view.(*worker).start(0xc0000e5700)
        /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0x9f fp=0xc00007dfc8 sp=0xc00007df38 pc=0x1294dff
go.opencensus.io/stats/view.init.0.gowrap1()
        /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x25 fp=0xc00007dfe0 sp=0xc00007dfc8 pc=0x1294165
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00007dfe8 sp=0xc00007dfe0 pc=0x8f0ea1
created by go.opencensus.io/stats/view.init.0 in goroutine 1
        /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x8d

goroutine 18 gp=0xc0004041c0 m=nil [chan receive]:
runtime.gopark(0x955c05?, 0x154a9c0?, 0x1?, 0x80?, 0xc000077f60?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000077ed0 sp=0xc000077eb0 pc=0x8c012e
runtime.chanrecv(0xc00010c120, 0xc000077fa0, 0x1)
        /usr/local/go/src/runtime/chan.go:583 +0x3cd fp=0xc000077f48 sp=0xc000077ed0 pc=0x88aa0d
runtime.chanrecv2(0xc00010c120?, 0xc000077fc0?)
        /usr/local/go/src/runtime/chan.go:447 +0x12 fp=0xc000077f70 sp=0xc000077f48 pc=0x88a632
github.com/restic/restic/internal/restic.init.0.func1.1()
        /restic/internal/restic/lock.go:404 +0x99 fp=0xc000077fe0 sp=0xc000077f70 pc=0xcd92f9
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000077fe8 sp=0xc000077fe0 pc=0x8f0ea1
created by github.com/restic/restic/internal/restic.init.0.func1 in goroutine 1
        /restic/internal/restic/lock.go:401 +0x1a

goroutine 34 gp=0xc0001041c0 m=6 mp=0xc000400008 [syscall]:
runtime.notetsleepg(0x21cd9a0, 0xffffffffffffffff)
        /usr/local/go/src/runtime/lock_sema.go:296 +0x31 fp=0xc000117fa0 sp=0xc000117f68 pc=0x890771
os/signal.signal_recv()
        /usr/local/go/src/runtime/sigqueue.go:152 +0x29 fp=0xc000117fc0 sp=0xc000117fa0 pc=0x8ec789
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:23 +0x13 fp=0xc000117fe0 sp=0xc000117fc0 pc=0x955fd3
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000117fe8 sp=0xc000117fe0 pc=0x8f0ea1
created by os/signal.Notify.func1.1 in goroutine 18
        /usr/local/go/src/os/signal/signal.go:151 +0x1f

goroutine 19 gp=0xc000405dc0 m=nil [GC worker (idle)]:
runtime.gopark(0x9a1dd2f48?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000079f50 sp=0xc000079f30 pc=0x8c012e
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc000079fe0 sp=0xc000079f50 pc=0x8a0c85
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000079fe8 sp=0xc000079fe0 pc=0x8f0ea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 50 gp=0xc000580000 m=nil [GC worker (idle)]:
runtime.gopark(0x9a1d1df94?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000113f50 sp=0xc000113f30 pc=0x8c012e
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc000113fe0 sp=0xc000113f50 pc=0x8a0c85
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000113fe8 sp=0xc000113fe0 pc=0x8f0ea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 7 gp=0xc0001b7a40 m=9 mp=0xc0003a6008 [running]:
        goroutine running on other thread; stack unavailable
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 20 gp=0xc000536000 m=nil [GC worker (idle)]:
runtime.gopark(0x9a1dd2f48?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00039ff50 sp=0xc00039ff30 pc=0x8c012e
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc00039ffe0 sp=0xc00039ff50 pc=0x8a0c85
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00039ffe8 sp=0xc00039ffe0 pc=0x8f0ea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 21 gp=0xc0005361c0 m=nil [chan receive]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc0003a1e98 sp=0xc0003a1e78 pc=0x8c012e
runtime.chanrecv(0xc00041f620, 0xc0003a1fa0, 0x1)
        /usr/local/go/src/runtime/chan.go:583 +0x3cd fp=0xc0003a1f10 sp=0xc0003a1e98 pc=0x88aa0d
runtime.chanrecv1(0x0?, 0x0?)
        /usr/local/go/src/runtime/chan.go:442 +0x12 fp=0xc0003a1f38 sp=0xc0003a1f10 pc=0x88a612
main.cleanupHandler(0x0?, 0xc000411950)
        /restic/cmd/restic/cleanup.go:24 +0x2a fp=0xc0003a1fc0 sp=0xc0003a1f38 pc=0x138f8aa
main.createGlobalContext.gowrap1()
        /restic/cmd/restic/cleanup.go:16 +0x25 fp=0xc0003a1fe0 sp=0xc0003a1fc0 pc=0x138f845
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0003a1fe8 sp=0xc0003a1fe0 pc=0x8f0ea1
created by main.createGlobalContext in goroutine 1
        /restic/cmd/restic/cleanup.go:16 +0x98
rax     0x17
rbx     0xc0001003c0
rcx     0xc000100008
rdx     0x0
rdi     0xe653814000
rsi     0xc00063fb18
rbp     0xc00063fc38
rsp     0xc00063fc18
r8      0xc00063f8e8
r9      0xc00063fac0
r10     0x0
r11     0x246
r12     0xe6547ff7e0
r13     0x0
r14     0xc000104000
r15     0x1
rip     0x8f0d01
rflags  0x212
cs      0x33
fs      0x53
gs      0x2b
PS C:\restic> .\restic_0.17.0_windows_amd64.exe
fatal error: runtime: mcall called on m->g0 stack

runtime stack:
runtime.throw({0x1722ad1?, 0x10?})
        /usr/local/go/src/runtime/panic.go:1023 +0x65 fp=0xc00035f050 sp=0xc00035f020 pc=0x8bd405
runtime.badmcall(0x18d1c720108?)
        /usr/local/go/src/runtime/proc.go:507 +0x1f fp=0xc00035f070 sp=0xc00035f050 pc=0x8c07bf
runtime.badmcall(0x8be899)
        <autogenerated>:1 +0x25 fp=0xc00035f088 sp=0xc00035f070 pc=0x8f5be5
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc00035f0a0
stack: frame={sp:0xc00035f070, fp:0xc00035f088} stack=[0x438da04000,0x438dbffca0)

goroutine 23 gp=0xc00046ea80 m=0 mp=0x2145fc0 [GC worker (active)]:
runtime.systemstack_switch()
        /usr/local/go/src/runtime/asm_amd64.s:474 +0x8 fp=0xc0004c3f50 sp=0xc0004c3f40 pc=0x8eee88
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1370 +0x1f7 fp=0xc0004c3fe0 sp=0xc0004c3f50 pc=0x8a0d97
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0004c3fe8 sp=0xc0004c3fe0 pc=0x8f0ea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 1 gp=0xc000066000 m=nil [runnable]:
text/template.goodFunc({0x18f6b50, 0x14d9100})
        /usr/local/go/src/text/template/funcs.go:112 +0xac fp=0xc00063f238 sp=0xc00063f230 pc=0xbfd1ac
reflect.unpackEface(...)
        /usr/local/go/src/reflect/value.go:160
reflect.ValueOf(...)
        /usr/local/go/src/reflect/value.go:3264
text/template.addValueFuncs(0xc000046540, 0xc00035f390?)
        /usr/local/go/src/text/template/funcs.go:92 +0xf3 fp=0xc00063f360 sp=0xc00063f238 pc=0xbfce73
text/template.createValueFuncs(...)
        /usr/local/go/src/text/template/funcs.go:82
text/template.builtinFuncs.func1()
        /usr/local/go/src/text/template/funcs.go:74 +0x5ae fp=0xc00063f3d0 sp=0xc00063f360 pc=0xc036ce
sync.(*Once).doSlow(0x75?, 0xc000090000?)
        /usr/local/go/src/sync/once.go:74 +0xc2 fp=0xc00063f430 sp=0xc00063f3d0 pc=0x8feca2
sync.(*Once).Do(...)
        /usr/local/go/src/sync/once.go:65
text/template.builtinFuncs()
        /usr/local/go/src/text/template/funcs.go:73 +0x2c fp=0xc00063f450 sp=0xc00063f430 pc=0xbfcd4c
text/template.findFunction({0x174c033, 0x2}, 0xc00035f520?)
        /usr/local/go/src/text/template/funcs.go:149 +0x145 fp=0xc00063f4b8 sp=0xc00063f450 pc=0xbfd4c5
text/template.(*state).evalFunction(0xc00063fc00, {0x16eda80?, 0x21345e0?, 0x2?}, 0x47?, {0x18e87f0, 0xc0000461b0}, {0xc00007e040, 0x3, 0x4}, ...)
        /usr/local/go/src/text/template/exec.go:619 +0xa6 fp=0xc00063f5b0 sp=0xc00063f4b8 pc=0xbf8ec6
text/template.(*state).evalCommand(0xc00035fc00, {0x16eda80?, 0x21345e0?, 0x16ef3b6?}, 0xc0000461b0, {0x1512160?, 0x21cc760?, 0x8?})
        /usr/local/go/src/text/template/exec.go:510 +0x1ce fp=0xc00063f6a8 sp=0xc00063f5b0 pc=0xbf7fce
text/template.(*state).evalPipeline(0xc00063fc00, {0x16eda80?, 0x21345e0?, 0xc000058008?}, 0xc000080240)
        /usr/local/go/src/text/template/exec.go:479 +0x125 fp=0xc00063f7a8 sp=0xc00063f6a8 pc=0xbf7945
text/template.(*state).evalCommand(0xc00035fc00, {0x16eda80?, 0x21345e0?, 0xd0?}, 0xc000046180, {0x1512160?, 0x21cc760?, 0xc0000480d0?})
        /usr/local/go/src/text/template/exec.go:514 +0x269 fp=0xc00063f8a0 sp=0xc00063f7a8 pc=0xbf8069
text/template.(*state).evalPipeline(0xc00063fc00, {0x16eda80?, 0x21345e0?, 0x15e9580?}, 0xc0000801e0)
        /usr/local/go/src/text/template/exec.go:479 +0x125 fp=0xc00063f9a0 sp=0xc00063f8a0 pc=0xbf7945
text/template.(*state).walkIfOrWith(0xc00063fc00, 0x13, {0x16eda80?, 0x21345e0?, 0x89a53d?}, 0x18d1c72ac48?, 0xc000046270, 0x0)
        /usr/local/go/src/text/template/exec.go:301 +0xc6 fp=0xc00063fa60 sp=0xc00063f9a0 pc=0xbf63a6
text/template.(*state).walk(0xc00035fc00, {0x16eda80?, 0x21345e0?, 0x18d1c72ac48?}, {0x18e8ac0?, 0xc00007e080?})
        /usr/local/go/src/text/template/exec.go:291 +0x1bb fp=0xc00063faf8 sp=0xc00063fa60 pc=0xbf60db
text/template.(*state).walk(0xc00063fc00, {0x16eda80?, 0x21345e0?, 0xc000092120?}, {0x18e8880?, 0xc000046150?})
        /usr/local/go/src/text/template/exec.go:280 +0x2e5 fp=0xc00063fb90 sp=0xc00063faf8 pc=0xbf6205
text/template.(*Template).execute(0xc000437100, {0x18d6860?, 0xc00010c058?}, {0x16eda80?, 0x21345e0?})
        /usr/local/go/src/text/template/exec.go:224 +0x278 fp=0xc00063fc80 sp=0xc00063fb90 pc=0xbf5e38
text/template.(*Template).Execute(...)
        /usr/local/go/src/text/template/exec.go:207
github.com/spf13/cobra.tmpl({0x18d6860, 0xc00010c058}, {0x174c02b, 0x7c}, {0x16eda80, 0x21345e0})
        /home/build/go/pkg/mod/github.com/spf13/cobra@v1.8.1/cobra.go:184 +0x165 fp=0xc00063fcc8 sp=0xc00063fc80 pc=0xc241a5
github.com/spf13/cobra.(*Command).HelpFunc.func1(0x21345e0, {0x2103460?, 0x18d6a60?, 0x2103460?})
        /home/build/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:465 +0x69 fp=0xc00063fd28 sp=0xc00063fcc8 pc=0xc355e9
github.com/spf13/cobra.(*Command).ExecuteC(0x21345e0)
        /home/build/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1122 +0x5d7 fp=0xc00063fe00 sp=0xc00063fd28 pc=0xc28ab7
github.com/spf13/cobra.(*Command).Execute(...)
        /home/build/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041
github.com/spf13/cobra.(*Command).ExecuteContext(...)
        /home/build/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1034
main.main()
        /restic/cmd/restic/main.go:125 +0x265 fp=0xc00063ff50 sp=0xc00063fe00 pc=0x13ca3e5
runtime.main()
        /usr/local/go/src/runtime/proc.go:271 +0x28b fp=0xc00063ffe0 sp=0xc00063ff50 pc=0x8bfd2b
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00063ffe8 sp=0xc00063ffe0 pc=0x8f0ea1

goroutine 2 gp=0xc000066700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000069fa8 sp=0xc000069f88 pc=0x8c012e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:408
runtime.forcegchelper()
        /usr/local/go/src/runtime/proc.go:326 +0xb8 fp=0xc000069fe0 sp=0xc000069fa8 pc=0x8bffb8
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000069fe8 sp=0xc000069fe0 pc=0x8f0ea1
created by runtime.init.6 in goroutine 1
        /usr/local/go/src/runtime/proc.go:314 +0x1a

goroutine 3 gp=0xc000066a80 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006bf80 sp=0xc00006bf60 pc=0x8c012e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:408
runtime.bgsweep(0xc000074000)
        /usr/local/go/src/runtime/mgcsweep.go:278 +0x94 fp=0xc00006bfc8 sp=0xc00006bf80 pc=0x8aa2d4
runtime.gcenable.gowrap1()
        /usr/local/go/src/runtime/mgc.go:203 +0x25 fp=0xc00006bfe0 sp=0xc00006bfc8 pc=0x89eb45
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0x8f0ea1
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000066c40 m=nil [GC scavenge wait]:
runtime.gopark(0xc000074000?, 0x18cdd08?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00007bf78 sp=0xc00007bf58 pc=0x8c012e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:408
runtime.(*scavengerState).park(0x21443c0)
        /usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00007bfa8 sp=0xc00007bf78 pc=0x8a7cc9
runtime.bgscavenge(0xc000074000)
        /usr/local/go/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc00007bfc8 sp=0xc00007bfa8 pc=0x8a8279
runtime.gcenable.gowrap2()
        /usr/local/go/src/runtime/mgc.go:204 +0x25 fp=0xc00007bfe0 sp=0xc00007bfc8 pc=0x89eae5
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00007bfe8 sp=0xc00007bfe0 pc=0x8f0ea1
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:204 +0xa5

goroutine 18 gp=0xc0001041c0 m=nil [finalizer wait]:
runtime.gopark(0xc00006de60?, 0x8a70fc?, 0xc0?, 0xe5?, 0x550011?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006de20 sp=0xc00006de00 pc=0x8c012e
runtime.runfinq()
        /usr/local/go/src/runtime/mfinal.go:194 +0x107 fp=0xc00006dfe0 sp=0xc00006de20 pc=0x89dbc7
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006dfe8 sp=0xc00006dfe0 pc=0x8f0ea1
created by runtime.createfing in goroutine 1
        /usr/local/go/src/runtime/mfinal.go:164 +0x3d

goroutine 19 gp=0xc0002168c0 m=nil [select]:
runtime.gopark(0xc000077f78?, 0x3?, 0x8?, 0xca?, 0xc000077f72?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000077e18 sp=0xc000077df8 pc=0x8c012e
runtime.selectgo(0xc000077f78, 0xc000077f6c, 0xc000193700?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000077f38 sp=0xc000077e18 pc=0x8d0585
go.opencensus.io/stats/view.(*worker).start(0xc000193700)
        /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0x9f fp=0xc000077fc8 sp=0xc000077f38 pc=0x1294dff
go.opencensus.io/stats/view.init.0.gowrap1()
        /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x25 fp=0xc000077fe0 sp=0xc000077fc8 pc=0x1294165
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000077fe8 sp=0xc000077fe0 pc=0x8f0ea1
created by go.opencensus.io/stats/view.init.0 in goroutine 1
        /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x8d

goroutine 20 gp=0xc000216a80 m=nil [chan receive]:
runtime.gopark(0x955be5?, 0x154a9c0?, 0x1?, 0x0?, 0xc000079f60?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000079ed0 sp=0xc000079eb0 pc=0x8c012e
runtime.chanrecv(0xc000080060, 0xc000079fa0, 0x1)
        /usr/local/go/src/runtime/chan.go:583 +0x3cd fp=0xc000079f48 sp=0xc000079ed0 pc=0x88aa0d
runtime.chanrecv2(0xc000080060?, 0xc000079fc0?)
        /usr/local/go/src/runtime/chan.go:447 +0x12 fp=0xc000079f70 sp=0xc000079f48 pc=0x88a632
github.com/restic/restic/internal/restic.init.0.func1.1()
        /restic/internal/restic/lock.go:404 +0x99 fp=0xc000079fe0 sp=0xc000079f70 pc=0xcd92f9
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000079fe8 sp=0xc000079fe0 pc=0x8f0ea1
created by github.com/restic/restic/internal/restic.init.0.func1 in goroutine 1
        /restic/internal/restic/lock.go:401 +0x1a

goroutine 21 gp=0xc00046e700 m=nil [chan receive]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc0004c7e98 sp=0xc0004c7e78 pc=0x8c012e
runtime.chanrecv(0xc00041d0e0, 0xc0004c7fa0, 0x1)
        /usr/local/go/src/runtime/chan.go:583 +0x3cd fp=0xc0004c7f10 sp=0xc0004c7e98 pc=0x88aa0d
runtime.chanrecv1(0x0?, 0x0?)
        /usr/local/go/src/runtime/chan.go:442 +0x12 fp=0xc0004c7f38 sp=0xc0004c7f10 pc=0x88a612
main.cleanupHandler(0x0?, 0xc000444ae0)
        /restic/cmd/restic/cleanup.go:24 +0x2a fp=0xc0004c7fc0 sp=0xc0004c7f38 pc=0x138f8aa
main.createGlobalContext.gowrap1()
        /restic/cmd/restic/cleanup.go:16 +0x25 fp=0xc0004c7fe0 sp=0xc0004c7fc0 pc=0x138f845
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0004c7fe8 sp=0xc0004c7fe0 pc=0x8f0ea1
created by main.createGlobalContext in goroutine 1
        /restic/cmd/restic/cleanup.go:16 +0x98

goroutine 22 gp=0xc00046e8c0 m=6 mp=0xc000084008 [syscall]:
runtime.notetsleepg(0x21cd9a0, 0xffffffffffffffff)
        /usr/local/go/src/runtime/lock_sema.go:296 +0x31 fp=0xc0004c9fa0 sp=0xc0004c9f68 pc=0x890771
os/signal.signal_recv()
        /usr/local/go/src/runtime/sigqueue.go:152 +0x29 fp=0xc0004c9fc0 sp=0xc0004c9fa0 pc=0x8ec789
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:23 +0x13 fp=0xc0004c9fe0 sp=0xc0004c9fc0 pc=0x955fd3
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0004c9fe8 sp=0xc0004c9fe0 pc=0x8f0ea1
created by os/signal.Notify.func1.1 in goroutine 1
        /usr/local/go/src/os/signal/signal.go:151 +0x1f

goroutine 5 gp=0xc000067180 m=nil [GC worker (idle)]:
runtime.gopark(0x21ce5c0?, 0x3?, 0x4?, 0xe2?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00007df50 sp=0xc00007df30 pc=0x8c012e
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc00007dfe0 sp=0xc00007df50 pc=0x8a0c85
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00007dfe8 sp=0xc00007dfe0 pc=0x8f0ea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 34 gp=0xc000500000 m=nil [GC worker (idle)]:
runtime.gopark(0x9f5f67c74?, 0x3?, 0xe0?, 0x88?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000507f50 sp=0xc000507f30 pc=0x8c012e
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc000507fe0 sp=0xc000507f50 pc=0x8a0c85
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000507fe8 sp=0xc000507fe0 pc=0x8f0ea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 50 gp=0xc000582000 m=nil [GC worker (idle)]:
runtime.gopark(0x9f5f67c74?, 0xc000580020?, 0x1a?, 0xa?, 0x0?)
        /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000503f50 sp=0xc000503f30 pc=0x8c012e
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc000503fe0 sp=0xc000503f50 pc=0x8a0c85
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000503fe8 sp=0xc000503fe0 pc=0x8f0ea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1234 +0x1c
PS C:\restic> .\restic_0.17.0_windows_amd64.exe

restic is a backup program which allows saving multiple revisions of files and
directories in an encrypted repository stored on different backends.

The full documentation can be found at https://restic.readthedocs.io/ .

Usage:
  restic [command]

Available Commands:
  backup        Create a new backup of files and/or directories
  cache         Operate on local cache directories
  cat           Print internal objects to stdout
  check         Check the repository for errors
  copy          Copy snapshots from one repository to another
  diff          Show differences between two snapshots
  dump          Print a backed-up file to stdout
  find          Find a file, a directory or restic IDs
  forget        Remove snapshots from the repository
  generate      Generate manual pages and auto-completion files (bash, fish, zsh, powershell)
  help          Help about any command
  init          Initialize a new repository
  key           Manage keys (passwords)
  list          List objects in the repository
  ls            List files in a snapshot
  migrate       Apply migrations
  prune         Remove unneeded data from the repository
  recover       Recover data from the repository not referenced by snapshots
  repair        Repair the repository
  restore       Extract the data from a snapshot
  rewrite       Rewrite snapshots to exclude unwanted files
  self-update   Update the restic binary
  snapshots     List all snapshots
  stats         Scan the repository and show basic statistics
  tag           Modify tags on snapshots
  unlock        Remove locks other processes created
  version       Print version information

Flags:
      --cacert file                file to load root certificates from (default: use system certificates or $RESTIC_CACERT)
      --cache-dir directory        set the cache directory. (default: use system default cache directory)
      --cleanup-cache              auto remove old cache directories
      --compression mode           compression mode (only available for repository format version 2), one of (auto|off|max) (default: $RESTIC_COMPRESSION) (default auto)
  -h, --help                       help for restic
      --http-user-agent string     set a http user agent for outgoing http requests
      --insecure-no-password       use an empty password for the repository, must be passed to every restic command (insecure)
      --insecure-tls               skip TLS certificate verification when connecting to the repository (insecure)
      --json                       set output mode to JSON for commands that support it
      --key-hint key               key ID of key to try decrypting first (default: $RESTIC_KEY_HINT)
      --limit-download rate        limits downloads to a maximum rate in KiB/s. (default: unlimited)
      --limit-upload rate          limits uploads to a maximum rate in KiB/s. (default: unlimited)
      --no-cache                   do not use a local cache
      --no-extra-verify            skip additional verification of data before upload (see documentation)
      --no-lock                    do not lock the repository, this allows some operations on read-only repositories
  -o, --option key=value           set extended option (key=value, can be specified multiple times)
      --pack-size size             set target pack size in MiB, created pack files may be larger (default: $RESTIC_PACK_SIZE)
      --password-command command   shell command to obtain the repository password from (default: $RESTIC_PASSWORD_COMMAND)
  -p, --password-file file         file to read the repository password from (default: $RESTIC_PASSWORD_FILE)
  -q, --quiet                      do not output comprehensive progress report
  -r, --repo repository            repository to backup to or restore from (default: $RESTIC_REPOSITORY)
      --repository-file file       file to read the repository location from (default: $RESTIC_REPOSITORY_FILE)
      --retry-lock duration        retry to lock the repository if it is already locked, takes a value like 5m or 2h (default: no retries)
      --tls-client-cert file       path to a file containing PEM encoded TLS client certificate and private key (default: $RESTIC_TLS_CLIENT_CERT)
  -v, --verbose                    be verbose (specify multiple times or a level using --verbose=n, max level/times is 2)

Use "restic [command] --help" for more information about a command.
PS C:\restic> systeminfo /fo csv | ConvertFrom-Csv | select OS*, System*, Hotfix* | Format-List

OS Name             : Microsoft Windows Server 2019 Standard
OS Version          : 10.0.17763 N/A Build 17763
OS Manufacturer     : Microsoft Corporation
OS Configuration    : Member Server
OS Build Type       : Multiprocessor Free
System Boot Time    : 8/21/2024, 12:16:08 AM
System Manufacturer : FreeBSD
System Model        : BHYVE
System Type         : x64-based PC
System Directory    : C:\Windows\system32
System Locale       : en-us;English (United States)
Hotfix(s)           : 5 Hotfix(s) Installed.,[01]: KB5041913,[02]: KB4577667,[03]: KB4589208,[04]: KB5041578,[05]:
                      KB5041577

PS C:\restic> .\restic_0.17.0_windows_amd64.exe version
restic 0.17.0 compiled with go1.22.5 on windows/amd64

[ianlancetaylor]

Thanks. CC @golang/windows

[distancesprinter]

Contributing some additional information:

I recompiled restic from source with go 1.23.0 (in docker, debian-stable as per the restic build instructions).

Here is the output following a reboot when trying to print the version string--three panics before a success:

Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

PS C:\Windows\system32> cd C:\restic\
PS C:\restic> .\restic_0.17.0_windows_amd64_g123.exe version
fatal error: runtime: mcall called on m->g0 stack

runtime stack:
runtime.throw({0x1acd5ed?, 0x0?})
        /usr/local/go/src/runtime/panic.go:1067 +0x4d fp=0xc0000517d8 sp=0xc0000517a8 pc=0xc4ea6d
runtime.badmcall(0x0?)
        /usr/local/go/src/runtime/proc.go:539 +0x1f fp=0xc0000517f8 sp=0xc0000517d8 pc=0xc1daff
runtime.badmcall(0xc1bc99)
        <autogenerated>:1 +0x25 fp=0xc000051810 sp=0xc0000517f8 pc=0xc5bf25
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc000051828
stack: frame={sp:0xc0000517f8, fp:0xc000051810} stack=[0x4a4a404000,0x4a4a5ff710)

goroutine 1 gp=0xc000068000 m=nil [runnable, locked to thread]:
sync.(*RWMutex).Lock(0x256fdb0?)
        /usr/local/go/src/sync/rwmutex.go:142 +0x7a fp=0xc0000519c8 sp=0xc0000519c0 pc=0xc676da
google.golang.org/protobuf/reflect/protoregistry.(*Types).RegisterEnum(0xc000120990, {0x1c8e450, 0xc000112d80})
        /home/build/go/pkg/mod/google.golang.org/protobuf@v1.34.2/reflect/protoregistry/registry.go:522 +0x79 fp=0xc000051a58 sp=0xc0000519c8 pc=0x1451c59
google.golang.org/protobuf/internal/filetype.Builder.Build({{{0x18264e9, 0x37}, {0x248fce0, 0x226, 0x226}, 0x2, 0x2, 0x0, 0x0, {0x0, ...}, ...}, ...})
        /home/build/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/filetype/build.go:153 +0x169e fp=0xc000051c30 sp=0xc000051a58 pc=0x14e12fe
github.com/google/s2a-go/internal/proto/common_go_proto.file_internal_proto_common_common_proto_init()
        /home/build/go/pkg/mod/github.com/google/s2a-go@v0.1.7/internal/proto/common_go_proto/common.pb.go:384 +0x278 fp=0xc000051e10 sp=0xc000051c30 pc=0x15b3e38
github.com/google/s2a-go/internal/proto/common_go_proto.init.0()
        /home/build/go/pkg/mod/github.com/google/s2a-go@v0.1.7/internal/proto/common_go_proto/common.pb.go:344 +0xf fp=0xc000051e20 sp=0xc000051e10 pc=0x15b3baf
runtime.doInit1(0x2484970)
        /usr/local/go/src/runtime/proc.go:7278 +0xed fp=0xc000051f50 sp=0xc000051e20 pc=0xc2b10d
runtime.doInit(...)
        /usr/local/go/src/runtime/proc.go:7245
runtime.main()
        /usr/local/go/src/runtime/proc.go:254 +0x325 fp=0xc000051fe0 sp=0xc000051f50 pc=0xc1d1a5
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000051fe8 sp=0xc000051fe0 pc=0xc571e1

goroutine 2 gp=0xc000068700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00006bfa8 sp=0xc00006bf88 pc=0xc4eb8e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:430
runtime.forcegchelper()
        /usr/local/go/src/runtime/proc.go:337 +0xb8 fp=0xc00006bfe0 sp=0xc00006bfa8 pc=0xc1d418
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0xc571e1
created by runtime.init.7 in goroutine 1
        /usr/local/go/src/runtime/proc.go:325 +0x1a

goroutine 18 gp=0xc0001061c0 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00010ff80 sp=0xc00010ff60 pc=0xc4eb8e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:430
runtime.bgsweep(0xc000078000)
        /usr/local/go/src/runtime/mgcsweep.go:277 +0x94 fp=0xc00010ffc8 sp=0xc00010ff80 pc=0xc06334
runtime.gcenable.gowrap1()
        /usr/local/go/src/runtime/mgc.go:203 +0x25 fp=0xc00010ffe0 sp=0xc00010ffc8 pc=0xbfa965
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00010ffe8 sp=0xc00010ffe0 pc=0xc571e1
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:203 +0x66

goroutine 19 gp=0xc000106380 m=nil [GC scavenge wait]:
runtime.gopark(0xc000078000?, 0x1c7a718?, 0x1?, 0x0?, 0xc000106380?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000111f78 sp=0xc000111f58 pc=0xc4eb8e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:430
runtime.(*scavengerState).park(0x25264c0)
        /usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000111fa8 sp=0xc000111f78 pc=0xc03d69
runtime.bgscavenge(0xc000078000)
        /usr/local/go/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc000111fc8 sp=0xc000111fa8 pc=0xc042dc
runtime.gcenable.gowrap2()
        /usr/local/go/src/runtime/mgc.go:204 +0x25 fp=0xc000111fe0 sp=0xc000111fc8 pc=0xbfa905
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000111fe8 sp=0xc000111fe0 pc=0xc571e1
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:204 +0xa5

goroutine 34 gp=0xc0000861c0 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00010be20 sp=0xc00010be00 pc=0xc4eb8e
runtime.runfinq()
        /usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc00010bfe0 sp=0xc00010be20 pc=0xbf9a27
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00010bfe8 sp=0xc00010bfe0 pc=0xc571e1
created by runtime.createfing in goroutine 1
        /usr/local/go/src/runtime/mfinal.go:163 +0x3d
PS C:\restic> .\restic_0.17.0_windows_amd64_g123.exe version
fatal error: runtime: mcall called on m->g0 stack

runtime stack:
runtime.throw({0x1acd5ed?, 0x3ff5554?})
        /usr/local/go/src/runtime/panic.go:1067 +0x4d fp=0xc000051bd8 sp=0xc000051ba8 pc=0xc4ea6d
runtime.badmcall(0xc000051c50?)
        /usr/local/go/src/runtime/proc.go:539 +0x1f fp=0xc000051bf8 sp=0xc000051bd8 pc=0xc1daff
runtime.badmcall(0xc1bc99)
        <autogenerated>:1 +0x25 fp=0xc000051c10 sp=0xc000051bf8 pc=0xc5bf25
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc000051c28
stack: frame={sp:0xc000051bf8, fp:0xc000051c10} stack=[0x7574c04000,0x7574dff900)

goroutine 1 gp=0xc000068000 m=nil [runnable, locked to thread]:
runtime.mapassign_faststr(0x18a9820?, 0xc00017dce0?, {0x1a9989c, 0x2})
        /usr/local/go/src/runtime/map_faststr.go:223 +0x3f5 fp=0xc000051dc0 sp=0xc000051db8 pc=0xc4cef5
regexp/syntax.map.init.0()
        /usr/local/go/src/regexp/syntax/perl_groups.go:26 +0x1c fp=0xc000051e10 sp=0xc000051dc0 pc=0xf910dc
regexp/syntax.init()
        /usr/local/go/src/regexp/syntax/perl_groups.go:26 +0xf fp=0xc000051e20 sp=0xc000051e10 pc=0xf910af
runtime.doInit1(0x2483140)
        /usr/local/go/src/runtime/proc.go:7278 +0xed fp=0xc000051f50 sp=0xc000051e20 pc=0xc2b10d
runtime.doInit(...)
        /usr/local/go/src/runtime/proc.go:7245
runtime.main()
        /usr/local/go/src/runtime/proc.go:254 +0x325 fp=0xc000051fe0 sp=0xc000051f50 pc=0xc1d1a5
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000051fe8 sp=0xc000051fe0 pc=0xc571e1

goroutine 2 gp=0xc000068700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00006bfa8 sp=0xc00006bf88 pc=0xc4eb8e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:430
runtime.forcegchelper()
        /usr/local/go/src/runtime/proc.go:337 +0xb8 fp=0xc00006bfe0 sp=0xc00006bfa8 pc=0xc1d418
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0xc571e1
created by runtime.init.7 in goroutine 1
        /usr/local/go/src/runtime/proc.go:325 +0x1a

goroutine 3 gp=0xc000068a80 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00006df80 sp=0xc00006df60 pc=0xc4eb8e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:430
runtime.bgsweep(0xc000078000)
        /usr/local/go/src/runtime/mgcsweep.go:277 +0x94 fp=0xc00006dfc8 sp=0xc00006df80 pc=0xc06334
runtime.gcenable.gowrap1()
        /usr/local/go/src/runtime/mgc.go:203 +0x25 fp=0xc00006dfe0 sp=0xc00006dfc8 pc=0xbfa965
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00006dfe8 sp=0xc00006dfe0 pc=0xc571e1
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000068c40 m=nil [GC scavenge wait]:
runtime.gopark(0xc000078000?, 0x1c7a718?, 0x1?, 0x0?, 0xc000068c40?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000085f78 sp=0xc000085f58 pc=0xc4eb8e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:430
runtime.(*scavengerState).park(0x25264c0)
        /usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000085fa8 sp=0xc000085f78 pc=0xc03d69
runtime.bgscavenge(0xc000078000)
        /usr/local/go/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc000085fc8 sp=0xc000085fa8 pc=0xc042dc
runtime.gcenable.gowrap2()
        /usr/local/go/src/runtime/mgc.go:204 +0x25 fp=0xc000085fe0 sp=0xc000085fc8 pc=0xbfa905
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000085fe8 sp=0xc000085fe0 pc=0xc571e1
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:204 +0xa5

goroutine 5 gp=0xc000068fc0 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000087e20 sp=0xc000087e00 pc=0xc4eb8e
runtime.runfinq()
        /usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc000087fe0 sp=0xc000087e20 pc=0xbf9a27
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000087fe8 sp=0xc000087fe0 pc=0xc571e1
created by runtime.createfing in goroutine 1
        /usr/local/go/src/runtime/mfinal.go:163 +0x3d
PS C:\restic> .\restic_0.17.0_windows_amd64_g123.exe version
fatal error: runtime: mcall called on m->g0 stack

runtime stack:
runtime.throw({0x1acd5ed?, 0x6?})
        /usr/local/go/src/runtime/panic.go:1067 +0x4d fp=0xc00048fb78 sp=0xc00048fb48 pc=0xc4ea6d
runtime.badmcall(0x8?)
        /usr/local/go/src/runtime/proc.go:539 +0x1f fp=0xc00048fb98 sp=0xc00048fb78 pc=0xc1daff
runtime.badmcall(0xc1bc99)
        <autogenerated>:1 +0x25 fp=0xc00048fbb0 sp=0xc00048fb98 pc=0xc5bf25
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc00048fbc8
stack: frame={sp:0xc00048fb98, fp:0xc00048fbb0} stack=[0x4eb6604000,0x4eb67ffc10)

goroutine 1 gp=0xc000068000 m=nil [runnable, locked to thread]:
github.com/restic/restic/internal/restic.SnapshotGroupByOptions.String({0x0?, 0x1?, 0x1?})
        /restic/internal/restic/snapshot_group.go:34 +0x17c fp=0xc00048fd68 sp=0xc00048fd60 pc=0x106a21c
github.com/restic/restic/internal/restic.(*SnapshotGroupByOptions).String(0xc0004e0100?)
        <autogenerated>:1 +0x2e fp=0xc00048fd80 sp=0xc00048fd68 pc=0x1073b6e
github.com/spf13/pflag.(*FlagSet).VarPF(0xc0004e0100, {0x1c90130, 0x25273d0}, {0x1aa01de, 0x8}, {0x1c7a740, 0x1}, {0x1af4025, 0x5b})
        /home/build/go/pkg/mod/github.com/spf13/pflag@v1.0.5/flag.go:829 +0x48 fp=0xc00048fdb8 sp=0xc00048fd80 pc=0xf5c028
github.com/spf13/pflag.(*FlagSet).VarP(...)
        /home/build/go/pkg/mod/github.com/spf13/pflag@v1.0.5/flag.go:837
main.init.0()
        /restic/cmd/restic/cmd_backup.go:106 +0x108 fp=0xc00048fe20 sp=0xc00048fdb8 pc=0x172d2a8
runtime.doInit1(0x248c800)
        /usr/local/go/src/runtime/proc.go:7278 +0xed fp=0xc00048ff50 sp=0xc00048fe20 pc=0xc2b10d
runtime.doInit(...)
        /usr/local/go/src/runtime/proc.go:7245
runtime.main()
        /usr/local/go/src/runtime/proc.go:254 +0x325 fp=0xc00048ffe0 sp=0xc00048ff50 pc=0xc1d1a5
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00048ffe8 sp=0xc00048ffe0 pc=0xc571e1

goroutine 2 gp=0xc000068700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00006bfa8 sp=0xc00006bf88 pc=0xc4eb8e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:430
runtime.forcegchelper()
        /usr/local/go/src/runtime/proc.go:337 +0xb8 fp=0xc00006bfe0 sp=0xc00006bfa8 pc=0xc1d418
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0xc571e1
created by runtime.init.7 in goroutine 1
        /usr/local/go/src/runtime/proc.go:325 +0x1a

goroutine 3 gp=0xc000068a80 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00006df80 sp=0xc00006df60 pc=0xc4eb8e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:430
runtime.bgsweep(0xc000078000)
        /usr/local/go/src/runtime/mgcsweep.go:277 +0x94 fp=0xc00006dfc8 sp=0xc00006df80 pc=0xc06334
runtime.gcenable.gowrap1()
        /usr/local/go/src/runtime/mgc.go:203 +0x25 fp=0xc00006dfe0 sp=0xc00006dfc8 pc=0xbfa965
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00006dfe8 sp=0xc00006dfe0 pc=0xc571e1
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000068c40 m=nil [GC scavenge wait]:
runtime.gopark(0x10000?, 0x1c7a718?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000085f78 sp=0xc000085f58 pc=0xc4eb8e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:430
runtime.(*scavengerState).park(0x25264c0)
        /usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000085fa8 sp=0xc000085f78 pc=0xc03d69
runtime.bgscavenge(0xc000078000)
        /usr/local/go/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc000085fc8 sp=0xc000085fa8 pc=0xc042f9
runtime.gcenable.gowrap2()
        /usr/local/go/src/runtime/mgc.go:204 +0x25 fp=0xc000085fe0 sp=0xc000085fc8 pc=0xbfa905
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000085fe8 sp=0xc000085fe0 pc=0xc571e1
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:204 +0xa5

goroutine 5 gp=0xc000068fc0 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000087e20 sp=0xc000087e00 pc=0xc4eb8e
runtime.runfinq()
        /usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc000087fe0 sp=0xc000087e20 pc=0xbf9a27
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000087fe8 sp=0xc000087fe0 pc=0xc571e1
created by runtime.createfing in goroutine 1
        /usr/local/go/src/runtime/mfinal.go:163 +0x3d

goroutine 6 gp=0xc0001ab6c0 m=nil [select]:
runtime.gopark(0xc00006ff78?, 0x3?, 0x90?, 0x0?, 0xc00006ff72?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00006fe10 sp=0xc00006fdf0 pc=0xc4eb8e
runtime.selectgo(0xc00006ff78, 0xc00006ff6c, 0xc00015b700?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:335 +0x7a5 fp=0xc00006ff38 sp=0xc00006fe10 pc=0xc2df05
go.opencensus.io/stats/view.(*worker).start(0xc00015b700)
        /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0x9f fp=0xc00006ffc8 sp=0xc00006ff38 pc=0x16332df
go.opencensus.io/stats/view.init.0.gowrap1()
        /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x25 fp=0xc00006ffe0 sp=0xc00006ffc8 pc=0x1632645
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00006ffe8 sp=0xc00006ffe0 pc=0xc571e1
created by go.opencensus.io/stats/view.init.0 in goroutine 1
        /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x8d

goroutine 18 gp=0xc00008e380 m=nil [chan receive]:
runtime.gopark(0xc000081f60?, 0xdbfd05?, 0x30?, 0x23?, 0x1ca8ac0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000081f18 sp=0xc000081ef8 pc=0xc4eb8e
runtime.chanrecv(0xc0000c0000, 0x0, 0x1)
        /usr/local/go/src/runtime/chan.go:639 +0x41e fp=0xc000081f90 sp=0xc000081f18 pc=0xbea41e
runtime.chanrecv1(0x0?, 0x0?)
        /usr/local/go/src/runtime/chan.go:489 +0x12 fp=0xc000081fb8 sp=0xc000081f90 pc=0xbe9fd2
runtime.unique_runtime_registerUniqueMapCleanup.func1(...)
        /usr/local/go/src/runtime/mgc.go:1732
runtime.unique_runtime_registerUniqueMapCleanup.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1735 +0x2f fp=0xc000081fe0 sp=0xc000081fb8 pc=0xbfda0f
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000081fe8 sp=0xc000081fe0 pc=0xc571e1
created by unique.runtime_registerUniqueMapCleanup in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1730 +0x96

goroutine 34 gp=0xc00035c000 m=nil [chan receive]:
runtime.gopark(0xcbef05?, 0x18ee5c0?, 0x1?, 0x1a?, 0xc000363f60?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000363ed0 sp=0xc000363eb0 pc=0xc4eb8e
runtime.chanrecv(0xc0003b29a0, 0xc000363fa0, 0x1)
        /usr/local/go/src/runtime/chan.go:639 +0x41e fp=0xc000363f48 sp=0xc000363ed0 pc=0xbea41e
runtime.chanrecv2(0xc0003b29a0?, 0xc000363fc0?)
        /usr/local/go/src/runtime/chan.go:494 +0x12 fp=0xc000363f70 sp=0xc000363f48 pc=0xbe9ff2
github.com/restic/restic/internal/restic.init.0.func1.1()
        /restic/internal/restic/lock.go:404 +0x94 fp=0xc000363fe0 sp=0xc000363f70 pc=0x1070834
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000363fe8 sp=0xc000363fe0 pc=0xc571e1
created by github.com/restic/restic/internal/restic.init.0.func1 in goroutine 1
        /restic/internal/restic/lock.go:401 +0x1a

goroutine 19 gp=0xc00008e700 m=3 mp=0xc000075008 [syscall]:
runtime.notetsleepg(0x25701c0, 0xffffffffffffffff)
        /usr/local/go/src/runtime/lock_sema.go:296 +0x31 fp=0xc000083fa0 sp=0xc000083f68 pc=0xbefd51
os/signal.signal_recv()
        /usr/local/go/src/runtime/sigqueue.go:152 +0x29 fp=0xc000083fc0 sp=0xc000083fa0 pc=0xc502e9
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:23 +0x13 fp=0xc000083fe0 sp=0xc000083fc0 pc=0xcbf2d3
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000083fe8 sp=0xc000083fe0 pc=0xc571e1
created by os/signal.Notify.func1.1 in goroutine 34
        /usr/local/go/src/os/signal/signal.go:151 +0x1f

goroutine 35 gp=0xc00035cc40 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000365f38 sp=0xc000365f18 pc=0xc4eb8e
runtime.gcBgMarkWorker(0xc000344150)
        /usr/local/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc000365fc8 sp=0xc000365f38 pc=0xbfcd09
runtime.gcBgMarkStartWorkers.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1279 +0x25 fp=0xc000365fe0 sp=0xc000365fc8 pc=0xbfcbe5
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000365fe8 sp=0xc000365fe0 pc=0xc571e1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1279 +0x105

goroutine 7 gp=0xc0001aba40 m=8 mp=0xc00044c008 [GC mark termination]:
runtime.systemstack_switch()
        /usr/local/go/src/runtime/asm_amd64.s:479 +0x8 fp=0xc00035fe98 sp=0xc00035fe88 pc=0xc551c8
runtime.forEachP(...)
        /usr/local/go/src/runtime/proc.go:2000
runtime.gcMarkDone()
        /usr/local/go/src/runtime/mgc.go:847 +0x152 fp=0xc00035ff38 sp=0xc00035fe98 pc=0xbfb5d2
runtime.gcBgMarkWorker(0xc000344150)
        /usr/local/go/src/runtime/mgc.go:1499 +0x345 fp=0xc00035ffc8 sp=0xc00035ff38 pc=0xbfcf65
runtime.gcBgMarkStartWorkers.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1279 +0x25 fp=0xc00035ffe0 sp=0xc00035ffc8 pc=0xbfcbe5
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00035ffe8 sp=0xc00035ffe0 pc=0xc571e1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1279 +0x105

goroutine 8 gp=0xc0001abc00 m=nil [GC worker (idle)]:
runtime.gopark(0xa4ad77be4?, 0x3?, 0x18?, 0xa7?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000361f38 sp=0xc000361f18 pc=0xc4eb8e
runtime.gcBgMarkWorker(0xc000344150)
        /usr/local/go/src/runtime/mgc.go:1363 +0xe9 fp=0xc000361fc8 sp=0xc000361f38 pc=0xbfcd09
runtime.gcBgMarkStartWorkers.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1279 +0x25 fp=0xc000361fe0 sp=0xc000361fc8 pc=0xbfcbe5
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000361fe8 sp=0xc000361fe0 pc=0xc571e1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1279 +0x105

goroutine 20 gp=0xc00008e8c0 m=nil [semacquire]:
runtime.gopark(0xc0004b8000?, 0xc55217?, 0x0?, 0xc0?, 0xbe9889?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc0003e5e30 sp=0xc0003e5e10 pc=0xc4eb8e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:430
runtime.semacquire1(0x2527efc, 0x0, 0x0, 0x0, 0x12)
        /usr/local/go/src/runtime/sema.go:178 +0x232 fp=0xc0003e5e98 sp=0xc0003e5e30 pc=0xc2efd2
runtime.semacquire(...)
        /usr/local/go/src/runtime/sema.go:129
runtime.gcMarkDone()
        /usr/local/go/src/runtime/mgc.go:827 +0x32 fp=0xc0003e5f38 sp=0xc0003e5e98 pc=0xbfb4b2
runtime.gcBgMarkWorker(0xc000344150)
        /usr/local/go/src/runtime/mgc.go:1499 +0x345 fp=0xc0003e5fc8 sp=0xc0003e5f38 pc=0xbfcf65
runtime.gcBgMarkStartWorkers.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1279 +0x25 fp=0xc0003e5fe0 sp=0xc0003e5fc8 pc=0xbfcbe5
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0003e5fe8 sp=0xc0003e5fe0 pc=0xc571e1
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1279 +0x105
PS C:\restic> .\restic_0.17.0_windows_amd64_g123.exe version
restic 0.17.0 compiled with go1.23.0 on windows/amd64
PS C:\restic>

Once the app runs successfully, it seems to be relatively stable for successive executions, but it will eventually panic again.

[distancesprinter]

A few more data points:

• I compiled restic with every point version of go between 1.20.3 and 1.23.0.
• I wrote a PowerShell script to execute restic with the version option 20 times.
• I found that restic compiled with versions of go 1.21.4 and earlier did not panic.
• The version of restic compiled with go version 1.21.5 panicked during the first 3 of 20 iterations.
• All future version of go panicked, but I am not sharing output of 1.21.6+ for brevity, except for 1.22.5 (because this is the version for the release build of restic) and 1.23.0 (to show the problem is present in the latest release).
• The version of restic compiled with go version 1.22.5 (the version of go used by restic release build) panicked during the first 4 of 20 iterations.
• The version of restic compiled with go version 1.23.0 panicked during the first 2 of 20 iterations.
• The problem did not present itself on my laptop running Windows 10 Enterprise build 19045 with the following CPU: AMD Ryzen 5 PRO 5650U with Radeon Graphics

Methodology:

• Compile restic in docker container as per restic build procedure.
• For each build: (1) Reboot system. (2) Execute iterations for specific restic build running PowerShell script.

PowerShell:

$ResticBin="C:\restic\restic_0.17.0_windows_amd64_g1214.exe"
$ResticCmd="version"

$N=0; $Limit=Read-Host "Please input loop limit"
While ($N -lt $Limit) { Write-Host "Loop $N in $Limit"; &"${ResticBin}" $ResticCmd; $N+=1 }

# Run the following at PS command prompt:
# .\iter_g1214.ps1 6>&1 5>&1 4>&1 3>&1 2>&1 | tee -filePath "C:\restic\out_g1214.txt"

out_g1214.txt:

Loop 0 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 1 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 2 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 3 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 4 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 5 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 6 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 7 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 8 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 9 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 10 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 11 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 12 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 13 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 14 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 15 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 16 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 17 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 18 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64
Loop 19 in 20
restic 0.17.0 compiled with go1.21.4 on windows/amd64

out_g1215.txt:

Loop 0 in 20
restic_0.17.0_windows_amd64_g1215.exe : fatal error: runtime: mcall called on m->g0 stack
At C:\restic\iter_g1215.ps1:5 char:57
+ ...  { Write-Host "Loop $N in $Limit"; &"${ResticBin}" $ResticCmd; $N+=1  ...
+                                        ~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (fatal error: ru... on m->g0 stack:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError

runtime stack:
runtime.throw({0x107f27f?, 0x1a4dda0?})
    /usr/local/go/src/runtime/panic.go:1077 +0x65 fp=0xc000145c20 sp=0xc000145bf0 pc=0x23b1c5
runtime.badmcall(0xc000145c78?)
    /usr/local/go/src/runtime/proc.go:503 +0x1f fp=0xc000145c40 sp=0xc000145c20 pc=0x23e4bf
runtime.badmcall(0x23c539)
    <autogenerated>:1 +0x25 fp=0xc000145c58 sp=0xc000145c40 pc=0x2706c5
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc000145c70
stack: frame={sp:0xc000145c40, fp:0xc000145c58} stack=[0x3d41804000,0x3d419ffc70)

goroutine 1 [runnable, locked to thread]:
github.com/Azure/azure-sdk-for-go/sdk/internal/log.init.0()
    /home/build/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/internal@v1.9.0/log/log.go:90 +0x16 fp=0xc000145e10 
sp=0xc000145e08 pc=0x88f716
runtime.doInit1(0x19e51e0)
    /usr/local/go/src/runtime/proc.go:6740 +0xdd fp=0xc000145f40 sp=0xc000145e10 pc=0x24a93d
runtime.doInit(...)
    /usr/local/go/src/runtime/proc.go:6707
runtime.main()
    /usr/local/go/src/runtime/proc.go:249 +0x356 fp=0xc000145fe0 sp=0xc000145f40 pc=0x23dab6
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000145fe8 sp=0xc000145fe0 pc=0x26ba61

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000061fa8 sp=0xc000061f88 pc=0x23de2e
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:404
runtime.forcegchelper()
    /usr/local/go/src/runtime/proc.go:322 +0xb8 fp=0xc000061fe0 sp=0xc000061fa8 pc=0x23dcb8
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000061fe8 sp=0xc000061fe0 pc=0x26ba61
created by runtime.init.6 in goroutine 1
    /usr/local/go/src/runtime/proc.go:310 +0x1a

goroutine 18 [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00010df78 sp=0xc00010df58 pc=0x23de2e
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:404
runtime.bgsweep(0x0?)
    /usr/local/go/src/runtime/mgcsweep.go:280 +0x94 fp=0xc00010dfc8 sp=0xc00010df78 pc=0x2283f4
runtime.gcenable.func1()
    /usr/local/go/src/runtime/mgc.go:200 +0x25 fp=0xc00010dfe0 sp=0xc00010dfc8 pc=0x21d4e5
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00010dfe8 sp=0xc00010dfe0 pc=0x26ba61
created by runtime.gcenable in goroutine 1
    /usr/local/go/src/runtime/mgc.go:200 +0x66

goroutine 19 [GC scavenge wait]:
runtime.gopark(0xc00006c000?, 0x1223630?, 0x1?, 0x0?, 0xc000104340?)
    /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc00010ff70 sp=0xc00010ff50 pc=0x23de2e
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:404
runtime.(*scavengerState).park(0x1a7c5a0)
    /usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00010ffa0 sp=0xc00010ff70 pc=0x225cc9
runtime.bgscavenge(0x0?)
    /usr/local/go/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc00010ffc8 sp=0xc00010ffa0 pc=0x22625c
runtime.gcenable.func2()
    /usr/local/go/src/runtime/mgc.go:201 +0x25 fp=0xc00010ffe0 sp=0xc00010ffc8 pc=0x21d485
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc00010ffe8 sp=0xc00010ffe0 pc=0x26ba61
created by runtime.gcenable in goroutine 1
    /usr/local/go/src/runtime/mgc.go:201 +0xa5

goroutine 20 [finalizer wait]:
runtime.gopark(0x1876b1b0b30?, 0x400000?, 0x70?, 0x5e?, 0x0?)
    /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000065e20 sp=0xc000065e00 pc=0x23de2e
runtime.runfinq()
    /usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc000065fe0 sp=0xc000065e20 pc=0x21c547
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000065fe8 sp=0xc000065fe0 pc=0x26ba61
created by runtime.createfing in goroutine 1
    /usr/local/go/src/runtime/mfinal.go:163 +0x3d
Loop 1 in 20
restic_0.17.0_windows_amd64_g1215.exe : fatal: morestack on g0
At C:\restic\iter_g1215.ps1:5 char:57
+ ...  { Write-Host "Loop $N in $Limit"; &"${ResticBin}" $ResticCmd; $N+=1  ...
+                                        ~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (fatal: morestack on g0:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError

Exception 0x80000003 0x0 0x0 0x26b8c1
PC=0x26b8c1

runtime.abort()
    /usr/local/go/src/runtime/asm_amd64.s:1113 +0x1 fp=0xc0001197d0 sp=0xc0001197c8 pc=0x26b8c1
traceback: unexpected SPWRITE function runtime.morestack
runtime.morestack()
    /usr/local/go/src/runtime/asm_amd64.s:560 +0x24 fp=0xc0001197d8 sp=0xc0001197d0 pc=0x269b24

goroutine 1 [runnable, locked to thread]:
google.golang.org/protobuf/reflect/protoregistry.(*Files).RegisterFile(0xc000008510, {0x1244428?, 0xc00015a000?})
    /home/build/go/pkg/mod/google.golang.org/protobuf@v1.34.2/reflect/protoregistry/registry.go:114 +0xb86 
fp=0xc0001199b0 sp=0xc0001199a8 pc=0xa0c466
google.golang.org/protobuf/internal/filetype.(*resolverByIndex).RegisterFile(0xddf5c1?, {0x1244428?, 0xc00015a000?})
    <autogenerated>:1 +0x2f fp=0xc0001199d8 sp=0xc0001199b0 pc=0xa9d18f
google.golang.org/protobuf/internal/filedesc.Builder.Build({{0xddf5c1, 0x37}, {0x19f18c0, 0x226, 0x226}, 0x2, 0x2, 
0x0, 0x0, {0x12353d0, ...}, ...})
    /home/build/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/filedesc/build.go:112 +0x1d6 fp=0xc000119a40 
sp=0xc0001199d8 pc=0xa28e56
google.golang.org/protobuf/internal/filetype.Builder.Build({{{0xddf5c1, 0x37}, {0x19f18c0, 0x226, 0x226}, 0x2, 0x2, 
0x0, 0x0, {0x0, ...}, ...}, ...})
    /home/build/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/filetype/build.go:138 +0x199 fp=0xc000119c20 
sp=0xc000119a40 pc=0xa9ae79
github.com/google/s2a-go/internal/proto/common_go_proto.file_internal_proto_common_common_proto_init()
    /home/build/go/pkg/mod/github.com/google/s2a-go@v0.1.7/internal/proto/common_go_proto/common.pb.go:384 +0x278 
fp=0xc000119e00 sp=0xc000119c20 pc=0xb6ff38
github.com/google/s2a-go/internal/proto/common_go_proto.init.0()
    /home/build/go/pkg/mod/github.com/google/s2a-go@v0.1.7/internal/proto/common_go_proto/common.pb.go:344 +0xf 
fp=0xc000119e10 sp=0xc000119e00 pc=0xb6fcaf
runtime.doInit1(0x19e6860)
    /usr/local/go/src/runtime/proc.go:6740 +0xdd fp=0xc000119f40 sp=0xc000119e10 pc=0x24a93d
runtime.doInit(...)
    /usr/local/go/src/runtime/proc.go:6707
runtime.main()
    /usr/local/go/src/runtime/proc.go:249 +0x356 fp=0xc000119fe0 sp=0xc000119f40 pc=0x23dab6
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000119fe8 sp=0xc000119fe0 pc=0x26ba61

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000061fa8 sp=0xc000061f88 pc=0x23de2e
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:404
runtime.forcegchelper()
    /usr/local/go/src/runtime/proc.go:322 +0xb8 fp=0xc000061fe0 sp=0xc000061fa8 pc=0x23dcb8
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000061fe8 sp=0xc000061fe0 pc=0x26ba61
created by runtime.init.6 in goroutine 1
    /usr/local/go/src/runtime/proc.go:310 +0x1a

goroutine 3 [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000063f78 sp=0xc000063f58 pc=0x23de2e
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:404
runtime.bgsweep(0x0?)
    /usr/local/go/src/runtime/mgcsweep.go:280 +0x94 fp=0xc000063fc8 sp=0xc000063f78 pc=0x2283f4
runtime.gcenable.func1()
    /usr/local/go/src/runtime/mgc.go:200 +0x25 fp=0xc000063fe0 sp=0xc000063fc8 pc=0x21d4e5
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000063fe8 sp=0xc000063fe0 pc=0x26ba61
created by runtime.gcenable in goroutine 1
    /usr/local/go/src/runtime/mgc.go:200 +0x66

goroutine 4 [GC scavenge wait]:
runtime.gopark(0xc00006c000?, 0x1223630?, 0x1?, 0x0?, 0xc00005eb60?)
    /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000073f70 sp=0xc000073f50 pc=0x23de2e
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:404
runtime.(*scavengerState).park(0x1a7c5a0)
    /usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000073fa0 sp=0xc000073f70 pc=0x225cc9
runtime.bgscavenge(0x0?)
    /usr/local/go/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc000073fc8 sp=0xc000073fa0 pc=0x22625c
runtime.gcenable.func2()
    /usr/local/go/src/runtime/mgc.go:201 +0x25 fp=0xc000073fe0 sp=0xc000073fc8 pc=0x21d485
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000073fe8 sp=0xc000073fe0 pc=0x26ba61
created by runtime.gcenable in goroutine 1
    /usr/local/go/src/runtime/mgc.go:201 +0xa5

goroutine 5 [finalizer wait]:
runtime.gopark(0x2100be?, 0x400000?, 0x70?, 0x5e?, 0x0?)
    /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000065e20 sp=0xc000065e00 pc=0x23de2e
runtime.runfinq()
    /usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc000065fe0 sp=0xc000065e20 pc=0x21c547
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000065fe8 sp=0xc000065fe0 pc=0x26ba61
created by runtime.createfing in goroutine 1
    /usr/local/go/src/runtime/mfinal.go:163 +0x3d
rax     0x17
rbx     0x1a7dc40
rcx     0x1a7d9c0
rdi     0xc8c1a7f000
rsi     0xc0001196c8
rbp     0xc0001197e8
rsp     0xc0001197c8
r8      0xc000119498
r9      0xc000119670
r10     0x0
r11     0x246
r12     0xc8c1dff6a0
r13     0x1
r14     0x1a7d3a0
r15     0x19f18ef
rip     0x26b8c1
rflags  0x216
cs      0x33
fs      0x53
gs      0x2b
Loop 2 in 20
restic_0.17.0_windows_amd64_g1215.exe : fatal error: runtime: mcall called on m->g0 stack
At C:\restic\iter_g1215.ps1:5 char:57
+ ...  { Write-Host "Loop $N in $Limit"; &"${ResticBin}" $ResticCmd; $N+=1  ...
+                                        ~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (fatal error: ru... on m->g0 stack:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError

runtime stack:
runtime.throw({0x107f27f?, 0x218948?})
    /usr/local/go/src/runtime/panic.go:1077 +0x65 fp=0xc000099c20 sp=0xc000099bf0 pc=0x23b1c5
runtime.badmcall(0x1ac8a503d50?)
    /usr/local/go/src/runtime/proc.go:503 +0x1f fp=0xc000099c40 sp=0xc000099c20 pc=0x23e4bf
runtime.badmcall(0x23c539)
    <autogenerated>:1 +0x25 fp=0xc000099c58 sp=0xc000099c40 pc=0x2706c5
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc000099c70
stack: frame={sp:0xc000099c40, fp:0xc000099c58} stack=[0x60d5204000,0x60d53ff900)

goroutine 1 [runnable, locked to thread]:
github.com/restic/restic/internal/filter.init()
    /restic/internal/filter/filter.go:12 +0x5b fp=0xc000099e10 sp=0xc000099e08 pc=0x81361b
runtime.doInit1(0x19e4c40)
    /usr/local/go/src/runtime/proc.go:6740 +0xdd fp=0xc000099f40 sp=0xc000099e10 pc=0x24a93d
runtime.doInit(...)
    /usr/local/go/src/runtime/proc.go:6707
runtime.main()
    /usr/local/go/src/runtime/proc.go:249 +0x356 fp=0xc000099fe0 sp=0xc000099f40 pc=0x23dab6
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000099fe8 sp=0xc000099fe0 pc=0x26ba61

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000061fa8 sp=0xc000061f88 pc=0x23de2e
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:404
runtime.forcegchelper()
    /usr/local/go/src/runtime/proc.go:322 +0xb8 fp=0xc000061fe0 sp=0xc000061fa8 pc=0x23dcb8
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000061fe8 sp=0xc000061fe0 pc=0x26ba61
created by runtime.init.6 in goroutine 1
    /usr/local/go/src/runtime/proc.go:310 +0x1a

goroutine 3 [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000063f78 sp=0xc000063f58 pc=0x23de2e
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:404
runtime.bgsweep(0x0?)
    /usr/local/go/src/runtime/mgcsweep.go:280 +0x94 fp=0xc000063fc8 sp=0xc000063f78 pc=0x2283f4
runtime.gcenable.func1()
    /usr/local/go/src/runtime/mgc.go:200 +0x25 fp=0xc000063fe0 sp=0xc000063fc8 pc=0x21d4e5
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000063fe8 sp=0xc000063fe0 pc=0x26ba61
created by runtime.gcenable in goroutine 1
    /usr/local/go/src/runtime/mgc.go:200 +0x66

goroutine 4 [GC scavenge wait]:
runtime.gopark(0xc00006c000?, 0x1223630?, 0x1?, 0x0?, 0xc00005eb60?)
    /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000073f70 sp=0xc000073f50 pc=0x23de2e
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:404
runtime.(*scavengerState).park(0x1a7c5a0)
    /usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000073fa0 sp=0xc000073f70 pc=0x225cc9
runtime.bgscavenge(0x0?)
    /usr/local/go/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc000073fc8 sp=0xc000073fa0 pc=0x22625c
runtime.gcenable.func2()
    /usr/local/go/src/runtime/mgc.go:201 +0x25 fp=0xc000073fe0 sp=0xc000073fc8 pc=0x21d485
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000073fe8 sp=0xc000073fe0 pc=0x26ba61
created by runtime.gcenable in goroutine 1
    /usr/local/go/src/runtime/mgc.go:201 +0xa5

goroutine 5 [finalizer wait]:
runtime.gopark(0x2100be?, 0x400000?, 0x70?, 0x5e?, 0x0?)
    /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000065e20 sp=0xc000065e00 pc=0x23de2e
runtime.runfinq()
    /usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc000065fe0 sp=0xc000065e20 pc=0x21c547
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000065fe8 sp=0xc000065fe0 pc=0x26ba61
created by runtime.createfing in goroutine 1
    /usr/local/go/src/runtime/mfinal.go:163 +0x3d
Loop 3 in 20
restic 0.17.0 compiled with go1.21.5 on windows/amd64
Loop 4 in 20
restic 0.17.0 compiled with go1.21.5 on windows/amd64
Loop 5 in 20
restic 0.17.0 compiled with go1.21.5 on windows/amd64
Loop 6 in 20
restic 0.17.0 compiled with go1.21.5 on windows/amd64
Loop 7 in 20
restic 0.17.0 compiled with go1.21.5 on windows/amd64
Loop 8 in 20
restic 0.17.0 compiled with go1.21.5 on windows/amd64
Loop 9 in 20
restic 0.17.0 compiled with go1.21.5 on windows/amd64
Loop 10 in 20
restic 0.17.0 compiled with go1.21.5 on windows/amd64
Loop 11 in 20
restic 0.17.0 compiled with go1.21.5 on windows/amd64
Loop 12 in 20
restic 0.17.0 compiled with go1.21.5 on windows/amd64
Loop 13 in 20
restic 0.17.0 compiled with go1.21.5 on windows/amd64
Loop 14 in 20
restic 0.17.0 compiled with go1.21.5 on windows/amd64
Loop 15 in 20
restic 0.17.0 compiled with go1.21.5 on windows/amd64
Loop 16 in 20
restic 0.17.0 compiled with go1.21.5 on windows/amd64
Loop 17 in 20
restic 0.17.0 compiled with go1.21.5 on windows/amd64
Loop 18 in 20
restic 0.17.0 compiled with go1.21.5 on windows/amd64
Loop 19 in 20
restic 0.17.0 compiled with go1.21.5 on windows/amd64

out_g1225.txt:

Loop 0 in 20
restic_0.17.0_windows_amd64.exe : fatal error: runtime: mcall called on m->g0 stack
At C:\restic\iter_g1225.ps1:5 char:57
+ ...  { Write-Host "Loop $N in $Limit"; &"${ResticBin}" $ResticCmd; $N+=1  ...
+                                        ~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (fatal error: ru... on m->g0 stack:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError

runtime stack:
runtime.throw({0x1c72ad1?, 0x25?})
    /usr/local/go/src/runtime/panic.go:1023 +0x65 fp=0xc0000a3c30 sp=0xc0000a3c00 pc=0xe0d405
runtime.badmcall(0xc0000a3ca8?)
    /usr/local/go/src/runtime/proc.go:507 +0x1f fp=0xc0000a3c50 sp=0xc0000a3c30 pc=0xe107bf
runtime.badmcall(0xe0e899)
    <autogenerated>:1 +0x25 fp=0xc0000a3c68 sp=0xc0000a3c50 pc=0xe45be5
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc0000a3c80
stack: frame={sp:0xc0000a3c50, fp:0xc0000a3c68} stack=[0xac86604000,0xac867ff710)

goroutine 1 gp=0xc000066000 m=nil [runnable, locked to thread]:
compress/flate.init()
    <autogenerated>:1 +0xf8 fp=0xc0000a3e20 sp=0xc0000a3e18 pc=0x106fd58
runtime.doInit1(0x25f33b0)
    /usr/local/go/src/runtime/proc.go:7176 +0xed fp=0xc0000a3f50 sp=0xc0000a3e20 pc=0xe1e06d
runtime.doInit(...)
    /usr/local/go/src/runtime/proc.go:7143
runtime.main()
    /usr/local/go/src/runtime/proc.go:253 +0x327 fp=0xc0000a3fe0 sp=0xc0000a3f50 pc=0xe0fdc7
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000a3fe8 sp=0xc0000a3fe0 pc=0xe40ea1

goroutine 2 gp=0xc000066700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000069fa8 sp=0xc000069f88 pc=0xe1012e
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:408
runtime.forcegchelper()
    /usr/local/go/src/runtime/proc.go:326 +0xb8 fp=0xc000069fe0 sp=0xc000069fa8 pc=0xe0ffb8
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000069fe8 sp=0xc000069fe0 pc=0xe40ea1
created by runtime.init.6 in goroutine 1
    /usr/local/go/src/runtime/proc.go:314 +0x1a

goroutine 3 gp=0xc000066a80 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006bf80 sp=0xc00006bf60 pc=0xe1012e
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:408
runtime.bgsweep(0xc000074000)
    /usr/local/go/src/runtime/mgcsweep.go:278 +0x94 fp=0xc00006bfc8 sp=0xc00006bf80 pc=0xdfa2d4
runtime.gcenable.gowrap1()
    /usr/local/go/src/runtime/mgc.go:203 +0x25 fp=0xc00006bfe0 sp=0xc00006bfc8 pc=0xdeeb45
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0xe40ea1
created by runtime.gcenable in goroutine 1
    /usr/local/go/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000066c40 m=nil [GC scavenge wait]:
runtime.gopark(0xc000074000?, 0x1e1dd08?, 0x1?, 0x0?, 0xc000066c40?)
    /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00007bf78 sp=0xc00007bf58 pc=0xe1012e
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:408
runtime.(*scavengerState).park(0x26943c0)
    /usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00007bfa8 sp=0xc00007bf78 pc=0xdf7cc9
runtime.bgscavenge(0xc000074000)
    /usr/local/go/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc00007bfc8 sp=0xc00007bfa8 pc=0xdf825c
runtime.gcenable.gowrap2()
    /usr/local/go/src/runtime/mgc.go:204 +0x25 fp=0xc00007bfe0 sp=0xc00007bfc8 pc=0xdeeae5
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00007bfe8 sp=0xc00007bfe0 pc=0xe40ea1
created by runtime.gcenable in goroutine 1
    /usr/local/go/src/runtime/mgc.go:204 +0xa5

goroutine 5 gp=0xc000066fc0 m=nil [finalizer wait]:
runtime.gopark(0xc00006de60?, 0xdf70fc?, 0xc0?, 0xe5?, 0x550011?)
    /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006de20 sp=0xc00006de00 pc=0xe1012e
runtime.runfinq()
    /usr/local/go/src/runtime/mfinal.go:194 +0x107 fp=0xc00006dfe0 sp=0xc00006de20 pc=0xdedbc7
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006dfe8 sp=0xc00006dfe0 pc=0xe40ea1
created by runtime.createfing in goroutine 1
    /usr/local/go/src/runtime/mfinal.go:164 +0x3d
Loop 1 in 20
restic_0.17.0_windows_amd64.exe : fatal: morestack on g0
At C:\restic\iter_g1225.ps1:5 char:57
+ ...  { Write-Host "Loop $N in $Limit"; &"${ResticBin}" $ResticCmd; $N+=1  ...
+                                        ~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (fatal: morestack on g0:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError

Exception 0x80000003 0x0 0x0 0xe40d01
PC=0xe40d01

runtime.abort()
    /usr/local/go/src/runtime/asm_amd64.s:1157 +0x1 fp=0xc00035fb08 sp=0xc00035fb00 pc=0xe40d01
runtime.morestack()
    /usr/local/go/src/runtime/asm_amd64.s:593 +0x3d fp=0xc00035fb10 sp=0xc00035fb08 pc=0xe3ef7d

goroutine 1 gp=0xc000066000 m=nil [runnable, locked to thread]:
net/http.(*Transport).RegisterProtocol(0xc0003f3180, {0x1c40a0f, 0x5}, {0x1e286e0, 0xc000426b40})
    /usr/local/go/src/net/http/transport.go:764 +0x312 fp=0xc00035fce8 sp=0xc00035fce0 pc=0x10f1d32
golang.org/x/net/http2.registerHTTPSProtocol(0xc000382000?, {0xc0002066c0?})
    /home/build/go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:3219 +0x6a fp=0xc00035fd50 sp=0xc00035fce8 
pc=0x1207baa
golang.org/x/net/http2.configureTransports(0xc0003f3180)
    /home/build/go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:287 +0xa5 fp=0xc00035fde8 sp=0xc00035fd50 
pc=0x11f7fe5
golang.org/x/net/http2.ConfigureTransports(...)
    /home/build/go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:277
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.init.0()
    /home/build/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.12.0/runtime/transport_default_http_client.go:
39 +0x15c fp=0xc00035fe20 sp=0xc00035fde8 pc=0x14904dc
runtime.doInit1(0x25f3410)
    /usr/local/go/src/runtime/proc.go:7176 +0xed fp=0xc00035ff50 sp=0xc00035fe20 pc=0xe1e06d
runtime.doInit(...)
    /usr/local/go/src/runtime/proc.go:7143
runtime.main()
    /usr/local/go/src/runtime/proc.go:253 +0x327 fp=0xc00035ffe0 sp=0xc00035ff50 pc=0xe0fdc7
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00035ffe8 sp=0xc00035ffe0 pc=0xe40ea1

goroutine 2 gp=0xc000066700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000069fa8 sp=0xc000069f88 pc=0xe1012e
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:408
runtime.forcegchelper()
    /usr/local/go/src/runtime/proc.go:326 +0xb8 fp=0xc000069fe0 sp=0xc000069fa8 pc=0xe0ffb8
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000069fe8 sp=0xc000069fe0 pc=0xe40ea1
created by runtime.init.6 in goroutine 1
    /usr/local/go/src/runtime/proc.go:314 +0x1a

goroutine 3 gp=0xc000066a80 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006bf80 sp=0xc00006bf60 pc=0xe1012e
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:408
runtime.bgsweep(0xc000074000)
    /usr/local/go/src/runtime/mgcsweep.go:278 +0x94 fp=0xc00006bfc8 sp=0xc00006bf80 pc=0xdfa2d4
runtime.gcenable.gowrap1()
    /usr/local/go/src/runtime/mgc.go:203 +0x25 fp=0xc00006bfe0 sp=0xc00006bfc8 pc=0xdeeb45
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0xe40ea1
created by runtime.gcenable in goroutine 1
    /usr/local/go/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000066c40 m=nil [GC scavenge wait]:
runtime.gopark(0xc000074000?, 0x1e1dd08?, 0x1?, 0x0?, 0xc000066c40?)
    /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00007bf78 sp=0xc00007bf58 pc=0xe1012e
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:408
runtime.(*scavengerState).park(0x26943c0)
    /usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00007bfa8 sp=0xc00007bf78 pc=0xdf7cc9
runtime.bgscavenge(0xc000074000)
    /usr/local/go/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc00007bfc8 sp=0xc00007bfa8 pc=0xdf825c
runtime.gcenable.gowrap2()
    /usr/local/go/src/runtime/mgc.go:204 +0x25 fp=0xc00007bfe0 sp=0xc00007bfc8 pc=0xdeeae5
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00007bfe8 sp=0xc00007bfe0 pc=0xe40ea1
created by runtime.gcenable in goroutine 1
    /usr/local/go/src/runtime/mgc.go:204 +0xa5

goroutine 5 gp=0xc000066fc0 m=nil [finalizer wait]:
runtime.gopark(0xc00006de60?, 0xdf70fc?, 0xc0?, 0xe5?, 0x550011?)
    /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006de20 sp=0xc00006de00 pc=0xe1012e
runtime.runfinq()
    /usr/local/go/src/runtime/mfinal.go:194 +0x107 fp=0xc00006dfe0 sp=0xc00006de20 pc=0xdedbc7
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006dfe8 sp=0xc00006dfe0 pc=0xe40ea1
created by runtime.createfing in goroutine 1
    /usr/local/go/src/runtime/mfinal.go:164 +0x3d

goroutine 18 gp=0xc00038a1c0 m=nil [select]:
runtime.gopark(0xc000077f78?, 0x3?, 0x98?, 0x5?, 0xc000077f72?)
    /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000077e18 sp=0xc000077df8 pc=0xe1012e
runtime.selectgo(0xc000077f78, 0xc000077f6c, 0xc00009ca00?, 0x0, 0x0?, 0x1)
    /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc000077f38 sp=0xc000077e18 pc=0xe20585
go.opencensus.io/stats/view.(*worker).start(0xc00009ca00)
    /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0x9f fp=0xc000077fc8 sp=0xc000077f38 
pc=0x17e4dff
go.opencensus.io/stats/view.init.0.gowrap1()
    /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x25 fp=0xc000077fe0 sp=0xc000077fc8 
pc=0x17e4165
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000077fe8 sp=0xc000077fe0 pc=0xe40ea1
created by go.opencensus.io/stats/view.init.0 in goroutine 1
    /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x8d
rax     0x17
rbx     0x2696378
rcx     0x2695fc0
rdx     0x0
rdi     0xf850475000
rsi     0xc00035fa00
rbp     0xc00035fb20
rsp     0xc00035fb00
r8      0xc00035f7d8
r9      0xc00035f9b0
r10     0x0
r11     0x246
r12     0xf8507ffd30
r13     0x0
r14     0x26953c0
r15     0x1fffffffffff
rip     0xe40d01
rflags  0x202
cs      0x33
fs      0x53
gs      0x2b
Loop 2 in 20
restic_0.17.0_windows_amd64.exe : fatal error: runtime: mcall called on m->g0 stack
At C:\restic\iter_g1225.ps1:5 char:57
+ ...  { Write-Host "Loop $N in $Limit"; &"${ResticBin}" $ResticCmd; $N+=1  ...
+                                        ~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (fatal error: ru... on m->g0 stack:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError

runtime stack:
runtime.throw({0x1c72ad1?, 0xcb?})
    /usr/local/go/src/runtime/panic.go:1023 +0x65 fp=0xc00035fc30 sp=0xc00035fc00 pc=0xe0d405
runtime.badmcall(0xc00035fc98?)
    /usr/local/go/src/runtime/proc.go:507 +0x1f fp=0xc00035fc50 sp=0xc00035fc30 pc=0xe107bf
runtime.badmcall(0xe0e899)
    <autogenerated>:1 +0x25 fp=0xc00035fc68 sp=0xc00035fc50 pc=0xe45be5
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc00035fc80
stack: frame={sp:0xc00035fc50, fp:0xc00035fc68} stack=[0x7d9c604000,0x7d9c7ffcd0)

goroutine 1 gp=0xc000066000 m=nil [runnable, locked to thread]:
main.init.36()
    /restic/cmd/restic/main.go:23 +0x26 fp=0xc00053fe20 sp=0xc00053fe18 pc=0x191a126
runtime.doInit1(0x25fb3e0)
    /usr/local/go/src/runtime/proc.go:7176 +0xed fp=0xc00053ff50 sp=0xc00053fe20 pc=0xe1e06d
runtime.doInit(...)
    /usr/local/go/src/runtime/proc.go:7143
runtime.main()
    /usr/local/go/src/runtime/proc.go:253 +0x327 fp=0xc00053ffe0 sp=0xc00053ff50 pc=0xe0fdc7
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00053ffe8 sp=0xc00053ffe0 pc=0xe40ea1

goroutine 2 gp=0xc000066700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000069fa8 sp=0xc000069f88 pc=0xe1012e
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:408
runtime.forcegchelper()
    /usr/local/go/src/runtime/proc.go:326 +0xb8 fp=0xc000069fe0 sp=0xc000069fa8 pc=0xe0ffb8
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000069fe8 sp=0xc000069fe0 pc=0xe40ea1
created by runtime.init.6 in goroutine 1
    /usr/local/go/src/runtime/proc.go:314 +0x1a

goroutine 3 gp=0xc000066a80 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006bf80 sp=0xc00006bf60 pc=0xe1012e
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:408
runtime.bgsweep(0xc000074000)
    /usr/local/go/src/runtime/mgcsweep.go:278 +0x94 fp=0xc00006bfc8 sp=0xc00006bf80 pc=0xdfa2d4
runtime.gcenable.gowrap1()
    /usr/local/go/src/runtime/mgc.go:203 +0x25 fp=0xc00006bfe0 sp=0xc00006bfc8 pc=0xdeeb45
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0xe40ea1
created by runtime.gcenable in goroutine 1
    /usr/local/go/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000066c40 m=nil [GC scavenge wait]:
runtime.gopark(0x10000?, 0x1e1dd08?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00007bf78 sp=0xc00007bf58 pc=0xe1012e
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:408
runtime.(*scavengerState).park(0x26943c0)
    /usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00007bfa8 sp=0xc00007bf78 pc=0xdf7cc9
runtime.bgscavenge(0xc000074000)
    /usr/local/go/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc00007bfc8 sp=0xc00007bfa8 pc=0xdf8279
runtime.gcenable.gowrap2()
    /usr/local/go/src/runtime/mgc.go:204 +0x25 fp=0xc00007bfe0 sp=0xc00007bfc8 pc=0xdeeae5
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00007bfe8 sp=0xc00007bfe0 pc=0xe40ea1
created by runtime.gcenable in goroutine 1
    /usr/local/go/src/runtime/mgc.go:204 +0xa5

goroutine 5 gp=0xc000066fc0 m=nil [finalizer wait]:
runtime.gopark(0xc00006de60?, 0xdf70fc?, 0xc0?, 0xe5?, 0x550011?)
    /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006de20 sp=0xc00006de00 pc=0xe1012e
runtime.runfinq()
    /usr/local/go/src/runtime/mfinal.go:194 +0x107 fp=0xc00006dfe0 sp=0xc00006de20 pc=0xdedbc7
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006dfe8 sp=0xc00006dfe0 pc=0xe40ea1
created by runtime.createfing in goroutine 1
    /usr/local/go/src/runtime/mfinal.go:164 +0x3d

goroutine 6 gp=0xc0001b76c0 m=nil [select]:
runtime.gopark(0xc00007df78?, 0x3?, 0x98?, 0x5?, 0xc00007df72?)
    /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00007de18 sp=0xc00007ddf8 pc=0xe1012e
runtime.selectgo(0xc00007df78, 0xc00007df6c, 0xc0000e5700?, 0x0, 0x0?, 0x1)
    /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc00007df38 sp=0xc00007de18 pc=0xe20585
go.opencensus.io/stats/view.(*worker).start(0xc0000e5700)
    /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0x9f fp=0xc00007dfc8 sp=0xc00007df38 
pc=0x17e4dff
go.opencensus.io/stats/view.init.0.gowrap1()
    /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x25 fp=0xc00007dfe0 sp=0xc00007dfc8 
pc=0x17e4165
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00007dfe8 sp=0xc00007dfe0 pc=0xe40ea1
created by go.opencensus.io/stats/view.init.0 in goroutine 1
    /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x8d

goroutine 7 gp=0xc0001b7880 m=nil [chan receive]:
runtime.gopark(0xea5c05?, 0x1a9a9c0?, 0x1?, 0x80?, 0xc000077f60?)
    /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000077ed0 sp=0xc000077eb0 pc=0xe1012e
runtime.chanrecv(0xc00010c120, 0xc000077fa0, 0x1)
    /usr/local/go/src/runtime/chan.go:583 +0x3cd fp=0xc000077f48 sp=0xc000077ed0 pc=0xddaa0d
runtime.chanrecv2(0xc00010c120?, 0xc000077fc0?)
    /usr/local/go/src/runtime/chan.go:447 +0x12 fp=0xc000077f70 sp=0xc000077f48 pc=0xdda632
github.com/restic/restic/internal/restic.init.0.func1.1()
    /restic/internal/restic/lock.go:404 +0x99 fp=0xc000077fe0 sp=0xc000077f70 pc=0x12292f9
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000077fe8 sp=0xc000077fe0 pc=0xe40ea1
created by github.com/restic/restic/internal/restic.init.0.func1 in goroutine 1
    /restic/internal/restic/lock.go:401 +0x1a

goroutine 18 gp=0xc0001041c0 m=4 mp=0xc000073808 [syscall]:
runtime.notetsleepg(0x271d9a0, 0xffffffffffffffff)
    /usr/local/go/src/runtime/lock_sema.go:296 +0x31 fp=0xc000117fa0 sp=0xc000117f68 pc=0xde0771
os/signal.signal_recv()
    /usr/local/go/src/runtime/sigqueue.go:152 +0x29 fp=0xc000117fc0 sp=0xc000117fa0 pc=0xe3c789
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:23 +0x13 fp=0xc000117fe0 sp=0xc000117fc0 pc=0xea5fd3
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000117fe8 sp=0xc000117fe0 pc=0xe40ea1
created by os/signal.Notify.func1.1 in goroutine 7
    /usr/local/go/src/os/signal/signal.go:151 +0x1f

goroutine 34 gp=0xc000405c00 m=nil [GC worker (idle)]:
runtime.gopark(0x271e5c0?, 0x3?, 0xd4?, 0xf4?, 0x0?)
    /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000113f50 sp=0xc000113f30 pc=0xe1012e
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc000113fe0 sp=0xc000113f50 pc=0xdf0c85
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000113fe8 sp=0xc000113fe0 pc=0xe40ea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
    /usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 19 gp=0xc000104380 m=nil [GC worker (idle)]:
runtime.gopark(0x271e5c0?, 0x3?, 0xb8?, 0x78?, 0x0?)
    /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000119f50 sp=0xc000119f30 pc=0xe1012e
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc000119fe0 sp=0xc000119f50 pc=0xdf0c85
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000119fe8 sp=0xc000119fe0 pc=0xe40ea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
    /usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 8 gp=0xc0001b7c00 m=nil [GC worker (idle)]:
runtime.gopark(0x271e5c0?, 0x3?, 0xb8?, 0x78?, 0x0?)
    /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000079f50 sp=0xc000079f30 pc=0xe1012e
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc000079fe0 sp=0xc000079f50 pc=0xdf0c85
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000079fe8 sp=0xc000079fe0 pc=0xe40ea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
    /usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 35 gp=0xc000405dc0 m=nil [GC worker (idle)]:
runtime.gopark(0x271e5c0?, 0x1?, 0x14?, 0x23?, 0x0?)
    /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000115f50 sp=0xc000115f30 pc=0xe1012e
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc000115fe0 sp=0xc000115f50 pc=0xdf0c85
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000115fe8 sp=0xc000115fe0 pc=0xe40ea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
    /usr/local/go/src/runtime/mgc.go:1234 +0x1c
Loop 3 in 20
restic_0.17.0_windows_amd64.exe : fatal error: runtime: mcall called on m->g0 stack
At C:\restic\iter_g1225.ps1:5 char:57
+ ...  { Write-Host "Loop $N in $Limit"; &"${ResticBin}" $ResticCmd; $N+=1  ...
+                                        ~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (fatal error: ru... on m->g0 stack:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError

runtime stack:
runtime.throw({0x1c72ad1?, 0x374000?})
    /usr/local/go/src/runtime/panic.go:1023 +0x65 fp=0xc00053fc10 sp=0xc00053fbe0 pc=0xe0d405
runtime.badmcall(0x10?)
    /usr/local/go/src/runtime/proc.go:507 +0x1f fp=0xc00053fc30 sp=0xc00053fc10 pc=0xe107bf
runtime.badmcall(0xe0e899)
    <autogenerated>:1 +0x25 fp=0xc00053fc48 sp=0xc00053fc30 pc=0xe45be5
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc00053fc60
stack: frame={sp:0xc00053fc30, fp:0xc00053fc48} stack=[0xbb31204000,0xbb313ffbe0)

goroutine 1 gp=0xc000066000 m=nil [runnable]:
github.com/spf13/cobra.(*Command).ExecuteC(0x26845e0)
    /home/build/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1054 +0x625 fp=0xc00053fe00 sp=0xc00053fdf8 
pc=0x1178b05
github.com/spf13/cobra.(*Command).Execute(...)
    /home/build/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041
github.com/spf13/cobra.(*Command).ExecuteContext(...)
    /home/build/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1034
main.main()
    /restic/cmd/restic/main.go:125 +0x265 fp=0xc00053ff50 sp=0xc00053fe00 pc=0x191a3e5
runtime.main()
    /usr/local/go/src/runtime/proc.go:271 +0x28b fp=0xc00053ffe0 sp=0xc00053ff50 pc=0xe0fd2b
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00053ffe8 sp=0xc00053ffe0 pc=0xe40ea1

goroutine 2 gp=0xc000066700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000069fa8 sp=0xc000069f88 pc=0xe1012e
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:408
runtime.forcegchelper()
    /usr/local/go/src/runtime/proc.go:326 +0xb8 fp=0xc000069fe0 sp=0xc000069fa8 pc=0xe0ffb8
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000069fe8 sp=0xc000069fe0 pc=0xe40ea1
created by runtime.init.6 in goroutine 1
    /usr/local/go/src/runtime/proc.go:314 +0x1a

goroutine 3 gp=0xc000066a80 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006bf80 sp=0xc00006bf60 pc=0xe1012e
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:408
runtime.bgsweep(0xc000074000)
    /usr/local/go/src/runtime/mgcsweep.go:278 +0x94 fp=0xc00006bfc8 sp=0xc00006bf80 pc=0xdfa2d4
runtime.gcenable.gowrap1()
    /usr/local/go/src/runtime/mgc.go:203 +0x25 fp=0xc00006bfe0 sp=0xc00006bfc8 pc=0xdeeb45
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0xe40ea1
created by runtime.gcenable in goroutine 1
    /usr/local/go/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000066c40 m=nil [GC scavenge wait]:
runtime.gopark(0xc000074000?, 0x1e1dd08?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00007bf78 sp=0xc00007bf58 pc=0xe1012e
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:408
runtime.(*scavengerState).park(0x26943c0)
    /usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00007bfa8 sp=0xc00007bf78 pc=0xdf7cc9
runtime.bgscavenge(0xc000074000)
    /usr/local/go/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc00007bfc8 sp=0xc00007bfa8 pc=0xdf8279
runtime.gcenable.gowrap2()
    /usr/local/go/src/runtime/mgc.go:204 +0x25 fp=0xc00007bfe0 sp=0xc00007bfc8 pc=0xdeeae5
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00007bfe8 sp=0xc00007bfe0 pc=0xe40ea1
created by runtime.gcenable in goroutine 1
    /usr/local/go/src/runtime/mgc.go:204 +0xa5

goroutine 5 gp=0xc000066fc0 m=nil [finalizer wait]:
runtime.gopark(0xc00006de60?, 0xdf70fc?, 0xc0?, 0xe5?, 0x550011?)
    /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00006de20 sp=0xc00006de00 pc=0xe1012e
runtime.runfinq()
    /usr/local/go/src/runtime/mfinal.go:194 +0x107 fp=0xc00006dfe0 sp=0xc00006de20 pc=0xdedbc7
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00006dfe8 sp=0xc00006dfe0 pc=0xe40ea1
created by runtime.createfing in goroutine 1
    /usr/local/go/src/runtime/mfinal.go:164 +0x3d

goroutine 6 gp=0xc0001376c0 m=nil [select]:
runtime.gopark(0xc00007df78?, 0x3?, 0x28?, 0xa?, 0xc00007df72?)
    /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc00007de18 sp=0xc00007ddf8 pc=0xe1012e
runtime.selectgo(0xc00007df78, 0xc00007df6c, 0xc0000e5700?, 0x0, 0x0?, 0x1)
    /usr/local/go/src/runtime/select.go:327 +0x725 fp=0xc00007df38 sp=0xc00007de18 pc=0xe20585
go.opencensus.io/stats/view.(*worker).start(0xc0000e5700)
    /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0x9f fp=0xc00007dfc8 sp=0xc00007df38 
pc=0x17e4dff
go.opencensus.io/stats/view.init.0.gowrap1()
    /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x25 fp=0xc00007dfe0 sp=0xc00007dfc8 
pc=0x17e4165
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00007dfe8 sp=0xc00007dfe0 pc=0xe40ea1
created by go.opencensus.io/stats/view.init.0 in goroutine 1
    /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x8d

goroutine 7 gp=0xc000137880 m=nil [chan receive]:
runtime.gopark(0xea5c05?, 0x1a9a9c0?, 0x1?, 0x20?, 0xc000077f60?)
    /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000077ed0 sp=0xc000077eb0 pc=0xe1012e
runtime.chanrecv(0xc0002e6120, 0xc000077fa0, 0x1)
    /usr/local/go/src/runtime/chan.go:583 +0x3cd fp=0xc000077f48 sp=0xc000077ed0 pc=0xddaa0d
runtime.chanrecv2(0xc0002e6120?, 0xc000077fc0?)
    /usr/local/go/src/runtime/chan.go:447 +0x12 fp=0xc000077f70 sp=0xc000077f48 pc=0xdda632
github.com/restic/restic/internal/restic.init.0.func1.1()
    /restic/internal/restic/lock.go:404 +0x99 fp=0xc000077fe0 sp=0xc000077f70 pc=0x12292f9
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000077fe8 sp=0xc000077fe0 pc=0xe40ea1
created by github.com/restic/restic/internal/restic.init.0.func1 in goroutine 1
    /restic/internal/restic/lock.go:401 +0x1a

goroutine 18 gp=0xc0002ec1c0 m=4 mp=0xc000073808 [syscall]:
runtime.notetsleepg(0x271d9a0, 0xffffffffffffffff)
    /usr/local/go/src/runtime/lock_sema.go:296 +0x31 fp=0xc0002f7fa0 sp=0xc0002f7f68 pc=0xde0771
os/signal.signal_recv()
    /usr/local/go/src/runtime/sigqueue.go:152 +0x29 fp=0xc0002f7fc0 sp=0xc0002f7fa0 pc=0xe3c789
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:23 +0x13 fp=0xc0002f7fe0 sp=0xc0002f7fc0 pc=0xea5fd3
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0002f7fe8 sp=0xc0002f7fe0 pc=0xe40ea1
created by os/signal.Notify.func1.1 in goroutine 7
    /usr/local/go/src/os/signal/signal.go:151 +0x1f

goroutine 8 gp=0xc000391500 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc000079f50 sp=0xc000079f30 pc=0xe1012e
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc000079fe0 sp=0xc000079f50 pc=0xdf0c85
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000079fe8 sp=0xc000079fe0 pc=0xe40ea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
    /usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 34 gp=0xc0004841c0 m=nil [GC worker (idle)]:
runtime.gopark(0x11b72e1120?, 0x3?, 0xbc?, 0x5c?, 0x0?)
    /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc0002f3f50 sp=0xc0002f3f30 pc=0xe1012e
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc0002f3fe0 sp=0xc0002f3f50 pc=0xdf0c85
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0002f3fe8 sp=0xc0002f3fe0 pc=0xe40ea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
    /usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 9 gp=0xc000391880 m=0 mp=0x2695fc0 [GC mark termination]:
runtime.systemstack_switch()
    /usr/local/go/src/runtime/asm_amd64.s:474 +0x8 fp=0xc00040fec0 sp=0xc00040feb0 pc=0xe3ee88
runtime.forEachP(...)
    /usr/local/go/src/runtime/proc.go:1895
runtime.gcMarkDone()
    /usr/local/go/src/runtime/mgc.go:827 +0x152 fp=0xc00040ff50 sp=0xc00040fec0 pc=0xdef752
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1446 +0x347 fp=0xc00040ffe0 sp=0xc00040ff50 pc=0xdf0ee7
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00040ffe8 sp=0xc00040ffe0 pc=0xe40ea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
    /usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 19 gp=0xc0002ec380 m=nil [GC worker (idle)]:
runtime.gopark(0x11b72e1120?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc0002f9f50 sp=0xc0002f9f30 pc=0xe1012e
runtime.gcBgMarkWorker()
    /usr/local/go/src/runtime/mgc.go:1310 +0xe5 fp=0xc0002f9fe0 sp=0xc0002f9f50 pc=0xdf0c85
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0002f9fe8 sp=0xc0002f9fe0 pc=0xe40ea1
created by runtime.gcBgMarkStartWorkers in goroutine 1
    /usr/local/go/src/runtime/mgc.go:1234 +0x1c

goroutine 35 gp=0xc000484380 m=nil [chan receive]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:402 +0xce fp=0xc0002f5e98 sp=0xc0002f5e78 pc=0xe1012e
runtime.chanrecv(0xc000492000, 0xc0002f5fa0, 0x1)
    /usr/local/go/src/runtime/chan.go:583 +0x3cd fp=0xc0002f5f10 sp=0xc0002f5e98 pc=0xddaa0d
runtime.chanrecv1(0x0?, 0x0?)
    /usr/local/go/src/runtime/chan.go:442 +0x12 fp=0xc0002f5f38 sp=0xc0002f5f10 pc=0xdda612
main.cleanupHandler(0x0?, 0xc000490000)
    /restic/cmd/restic/cleanup.go:24 +0x2a fp=0xc0002f5fc0 sp=0xc0002f5f38 pc=0x18df8aa
main.createGlobalContext.gowrap1()
    /restic/cmd/restic/cleanup.go:16 +0x25 fp=0xc0002f5fe0 sp=0xc0002f5fc0 pc=0x18df845
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0002f5fe8 sp=0xc0002f5fe0 pc=0xe40ea1
created by main.createGlobalContext in goroutine 1
    /restic/cmd/restic/cleanup.go:16 +0x98
Loop 4 in 20
restic 0.17.0 compiled with go1.22.5 on windows/amd64
Loop 5 in 20
restic 0.17.0 compiled with go1.22.5 on windows/amd64
Loop 6 in 20
restic 0.17.0 compiled with go1.22.5 on windows/amd64
Loop 7 in 20
restic 0.17.0 compiled with go1.22.5 on windows/amd64
Loop 8 in 20
restic 0.17.0 compiled with go1.22.5 on windows/amd64
Loop 9 in 20
restic 0.17.0 compiled with go1.22.5 on windows/amd64
Loop 10 in 20
restic 0.17.0 compiled with go1.22.5 on windows/amd64
Loop 11 in 20
restic 0.17.0 compiled with go1.22.5 on windows/amd64
Loop 12 in 20
restic 0.17.0 compiled with go1.22.5 on windows/amd64
Loop 13 in 20
restic 0.17.0 compiled with go1.22.5 on windows/amd64
Loop 14 in 20
restic 0.17.0 compiled with go1.22.5 on windows/amd64
Loop 15 in 20
restic 0.17.0 compiled with go1.22.5 on windows/amd64
Loop 16 in 20
restic 0.17.0 compiled with go1.22.5 on windows/amd64
Loop 17 in 20
restic 0.17.0 compiled with go1.22.5 on windows/amd64
Loop 18 in 20
restic 0.17.0 compiled with go1.22.5 on windows/amd64
Loop 19 in 20
restic 0.17.0 compiled with go1.22.5 on windows/amd64

out_g123:

Loop 0 in 20
restic_0.17.0_windows_amd64_g123.exe : fatal: morestack on g0
At C:\restic\iter_g123.ps1:5 char:57
+ ...  { Write-Host "Loop $N in $Limit"; &"${ResticBin}" $ResticCmd; $N+=1  ...
+                                        ~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (fatal: morestack on g0:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError

Exception 0x80000003 0x0 0x0 0xbc7041
PC=0xbc7041

runtime.abort()
    /usr/local/go/src/runtime/asm_amd64.s:1162 +0x1 fp=0xc000051408 sp=0xc000051400 pc=0xbc7041
runtime.morestack()
    /usr/local/go/src/runtime/asm_amd64.s:598 +0x3d fp=0xc000051410 sp=0xc000051408 pc=0xbc52bd

goroutine 1 gp=0xc000068000 m=nil [runnable, locked to thread]:
google.golang.org/protobuf/internal/impl.getExtensionFieldInfo({0x1c085b0?, 0x247c400?})
    /home/build/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/codec_extension.go:24 +0xe5 fp=0xc0000515e0 
sp=0xc0000515d8 pc=0x13ff4c5
google.golang.org/protobuf/internal/impl.(*MessageInfo).unmarshalExtension(0xc000189860, {0x23f83a9, 0x1, 0x54}, 
0x3ea, 0x2, 0xc0001e8480, {0x0, {0x1bfe478, 0xc00009a720}, ...})
    /home/build/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/decode.go:215 +0x285 fp=0xc000051720 
sp=0xc0000515e0 pc=0x1430085
google.golang.org/protobuf/internal/impl.(*MessageInfo).unmarshalPointer(0xc000189860, {0x23f83a7?, 0x3, 0x56?}, 
{0x1409c91?}, 0x0, {0x0, {0x1bfe478, 0xc00009a720}, 0x270e})
    /home/build/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/decode.go:169 +0x5da fp=0xc000051858 
sp=0xc000051720 pc=0x142f81a
google.golang.org/protobuf/internal/impl.consumeMessageInfo({0x23f83a6?, 0x23fff20?, 0xc0000519b0?}, {0xbb995d?}, 
0x0?, 0xc0000a48a8, {0x0, {0x1bfe478, 0xc00009a720}, 0x270e})
    /home/build/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/codec_field.go:258 +0x11b fp=0xc0000518e0 
sp=0xc000051858 pc=0x140277b
google.golang.org/protobuf/internal/impl.(*MessageInfo).unmarshalPointer(0xc00018a678, {0x23f83a2?, 0x1b, 0x5b?}, 
{0x0?}, 0x0, {0x0, {0x1bfe478, 0xc00009a720}, 0x270f})
    /home/build/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/decode.go:148 +0x3a9 fp=0xc000051a18 
sp=0xc0000518e0 pc=0x142f5e9
google.golang.org/protobuf/internal/impl.consumeMessageSliceInfo({0x23f83a1?, 0x0?, 0xc0000a4680?}, {0x0?}, 0x7?, 
0xc0000a4680, {0x0, {0x1bfe478, 0xc00009a720}, 0x270f})
    /home/build/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/codec_field.go:520 +0x105 fp=0xc000051ac0 
sp=0xc000051a18 pc=0x14048c5
google.golang.org/protobuf/internal/impl.(*MessageInfo).unmarshalPointer(0xc0001899a8, {0x23f83a0?, 0x5d, 
0xc00009b1a0?}, {0xc00009b1d0?}, 0x0, {0x0, {0x1bfe478, 0xc00009a720}, 0x2710})
    /home/build/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/decode.go:148 +0x3a9 fp=0xc000051bf8 
sp=0xc000051ac0 pc=0x142f5e9
google.golang.org/protobuf/internal/impl.(*MessageInfo).unmarshal(0xc0001899a8?, {{}, {0x1c0e8a0, 0x2495e00}, 
{0x23f83a0, 0x5d, 0x5d}, 0x0, {0x1bfe478, 0xc00009a720}, ...})
    /home/build/go/pkg/mod/google.golang.org/protobuf@v1.34.2/internal/impl/decode.go:66 +0x97 fp=0xc000051c58 
sp=0xc000051bf8 pc=0x142f1f7
google.golang.org/protobuf/internal/impl.(*MessageInfo).unmarshal-fm({{}, {0x1c0e8a0, 0x2495e00}, {0x23f83a0, 0x5d, 
0x5d}, 0x0, {0x1bfe478, 0xc00009a720}, 0x2710})
    <autogenerated>:1 +0x9b fp=0xc000051cb8 sp=0xc000051c58 pc=0x144f09b
google.golang.org/protobuf/proto.UnmarshalOptions.unmarshal({{}, 0x1, 0x1, 0x0, {0x1bfe478, 0xc00009a720}, 0x2710}, 
{0x23f83a0, 0x5d, 0x5d}, ...)
    /home/build/go/pkg/mod/google.golang.org/protobuf@v1.34.2/proto/decode.go:107 +0x1fb fp=0xc000051d70 sp=0xc000051cb8 
pc=0x13d227b
google.golang.org/protobuf/proto.Unmarshal({0x23f83a0, 0x5d, 0x5d}, {0x1bf9d40?, 0x2495e00?})
    /home/build/go/pkg/mod/google.golang.org/protobuf@v1.34.2/proto/decode.go:57 +0x5d fp=0xc000051dc8 sp=0xc000051d70 
pc=0x13d1ddd
google.golang.org/protobuf/reflect/protodesc.init.0()
    /home/build/go/pkg/mod/google.golang.org/protobuf@v1.34.2/reflect/protodesc/editions.go:25 +0x3a fp=0xc000051e20 
sp=0xc000051dc8 pc=0x1522d5a
runtime.doInit1(0x23f4a30)
    /usr/local/go/src/runtime/proc.go:7278 +0xed fp=0xc000051f50 sp=0xc000051e20 pc=0xb9b10d
runtime.doInit(...)
    /usr/local/go/src/runtime/proc.go:7245
runtime.main()
    /usr/local/go/src/runtime/proc.go:254 +0x325 fp=0xc000051fe0 sp=0xc000051f50 pc=0xb8d1a5
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000051fe8 sp=0xc000051fe0 pc=0xbc71e1

goroutine 2 gp=0xc000068700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00006bfa8 sp=0xc00006bf88 pc=0xbbeb8e
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:430
runtime.forcegchelper()
    /usr/local/go/src/runtime/proc.go:337 +0xb8 fp=0xc00006bfe0 sp=0xc00006bfa8 pc=0xb8d418
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0xbc71e1
created by runtime.init.7 in goroutine 1
    /usr/local/go/src/runtime/proc.go:325 +0x1a

goroutine 3 gp=0xc000068a80 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00006df80 sp=0xc00006df60 pc=0xbbeb8e
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:430
runtime.bgsweep(0xc000078000)
    /usr/local/go/src/runtime/mgcsweep.go:277 +0x94 fp=0xc00006dfc8 sp=0xc00006df80 pc=0xb76334
runtime.gcenable.gowrap1()
    /usr/local/go/src/runtime/mgc.go:203 +0x25 fp=0xc00006dfe0 sp=0xc00006dfc8 pc=0xb6a965
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00006dfe8 sp=0xc00006dfe0 pc=0xbc71e1
created by runtime.gcenable in goroutine 1
    /usr/local/go/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000068c40 m=nil [GC scavenge wait]:
runtime.gopark(0xc000078000?, 0x1bea718?, 0x1?, 0x0?, 0xc000068c40?)
    /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000085f78 sp=0xc000085f58 pc=0xbbeb8e
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:430
runtime.(*scavengerState).park(0x24964c0)
    /usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000085fa8 sp=0xc000085f78 pc=0xb73d69
runtime.bgscavenge(0xc000078000)
    /usr/local/go/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc000085fc8 sp=0xc000085fa8 pc=0xb742dc
runtime.gcenable.gowrap2()
    /usr/local/go/src/runtime/mgc.go:204 +0x25 fp=0xc000085fe0 sp=0xc000085fc8 pc=0xb6a905
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000085fe8 sp=0xc000085fe0 pc=0xbc71e1
created by runtime.gcenable in goroutine 1
    /usr/local/go/src/runtime/mgc.go:204 +0xa5

goroutine 5 gp=0xc000068fc0 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000087e20 sp=0xc000087e00 pc=0xbbeb8e
runtime.runfinq()
    /usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc000087fe0 sp=0xc000087e20 pc=0xb69a27
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000087fe8 sp=0xc000087fe0 pc=0xbc71e1
created by runtime.createfing in goroutine 1
    /usr/local/go/src/runtime/mfinal.go:163 +0x3d
rax     0x17
rbx     0x2498d80
rcx     0x2498aa0
rdx     0x0
rdi     0xd9aab8c000
rsi     0x2498aa0
rbp     0xc000051420
rsp     0xc000051400
r8      0xc0000510b8
r9      0xc000051290
r10     0x0
r11     0x246
r12     0xd9aadff790
r13     0xc00009a720
r14     0x2497640
r15     0x23f83a9
rip     0xbc7041
rflags  0x202
cs      0x33
fs      0x53
gs      0x2b
Loop 1 in 20
restic_0.17.0_windows_amd64_g123.exe : fatal error: runtime: mcall called on m->g0 stack
At C:\restic\iter_g123.ps1:5 char:57
+ ...  { Write-Host "Loop $N in $Limit"; &"${ResticBin}" $ResticCmd; $N+=1  ...
+                                        ~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (fatal error: ru... on m->g0 stack:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError

runtime stack:
runtime.throw({0x1a3d5ed?, 0x0?})
    /usr/local/go/src/runtime/panic.go:1067 +0x4d fp=0xc0001b5a18 sp=0xc0001b59e8 pc=0xbbea6d
runtime.badmcall(0xf03b7f?)
    /usr/local/go/src/runtime/proc.go:539 +0x1f fp=0xc0001b5a38 sp=0xc0001b5a18 pc=0xb8daff
runtime.badmcall(0xb8bc99)
    <autogenerated>:1 +0x25 fp=0xc0001b5a50 sp=0xc0001b5a38 pc=0xbcbf25
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc0001b5a68
stack: frame={sp:0xc0001b5a38, fp:0xc0001b5a50} stack=[0x7fbe604000,0x7fbe7ff7f0)

goroutine 1 gp=0xc000068000 m=nil [runnable, locked to thread]:
regexp/syntax.(*parser).op(0xc00015b080, 0x9)
    /usr/local/go/src/regexp/syntax/parse.go:393 +0xe5 fp=0xc0001b5c00 sp=0xc0001b5bf8 pc=0xf05945
regexp/syntax.parse({0x1a53de5, 0x35}, 0xd4)
    /usr/local/go/src/regexp/syntax/parse.go:987 +0x2cf fp=0xc0001b5d00 sp=0xc0001b5c00 pc=0xf0870f
regexp/syntax.Parse(...)
    /usr/local/go/src/regexp/syntax/parse.go:887
regexp.compile({0x1a53de5, 0x35}, 0x25?, 0x0)
    /usr/local/go/src/regexp/regexp.go:168 +0x30 fp=0xc0001b5d88 sp=0xc0001b5d00 pc=0xf1a5f0
regexp.Compile(...)
    /usr/local/go/src/regexp/regexp.go:131
regexp.MustCompile({0x1a53de5, 0x35})
    /usr/local/go/src/regexp/regexp.go:311 +0x2c fp=0xc0001b5e00 sp=0xc0001b5d88 pc=0xf1b14c
go.opencensus.io/resource.init()
    /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/resource/resource.go:60 +0x1f fp=0xc0001b5e20 sp=0xc0001b5e00 
pc=0x159cd9f
runtime.doInit1(0x23f3910)
    /usr/local/go/src/runtime/proc.go:7278 +0xed fp=0xc0001b5f50 sp=0xc0001b5e20 pc=0xb9b10d
runtime.doInit(...)
    /usr/local/go/src/runtime/proc.go:7245
runtime.main()
    /usr/local/go/src/runtime/proc.go:254 +0x325 fp=0xc0001b5fe0 sp=0xc0001b5f50 pc=0xb8d1a5
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc0001b5fe8 sp=0xc0001b5fe0 pc=0xbc71e1

goroutine 2 gp=0xc000068700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00006bfa8 sp=0xc00006bf88 pc=0xbbeb8e
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:430
runtime.forcegchelper()
    /usr/local/go/src/runtime/proc.go:337 +0xb8 fp=0xc00006bfe0 sp=0xc00006bfa8 pc=0xb8d418
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00006bfe8 sp=0xc00006bfe0 pc=0xbc71e1
created by runtime.init.7 in goroutine 1
    /usr/local/go/src/runtime/proc.go:325 +0x1a

goroutine 3 gp=0xc000068a80 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00006df80 sp=0xc00006df60 pc=0xbbeb8e
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:430
runtime.bgsweep(0xc000078000)
    /usr/local/go/src/runtime/mgcsweep.go:277 +0x94 fp=0xc00006dfc8 sp=0xc00006df80 pc=0xb76334
runtime.gcenable.gowrap1()
    /usr/local/go/src/runtime/mgc.go:203 +0x25 fp=0xc00006dfe0 sp=0xc00006dfc8 pc=0xb6a965
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00006dfe8 sp=0xc00006dfe0 pc=0xbc71e1
created by runtime.gcenable in goroutine 1
    /usr/local/go/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000068c40 m=nil [GC scavenge wait]:
runtime.gopark(0xc000078000?, 0x1bea718?, 0x1?, 0x0?, 0xc000068c40?)
    /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000085f78 sp=0xc000085f58 pc=0xbbeb8e
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:430
runtime.(*scavengerState).park(0x24964c0)
    /usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000085fa8 sp=0xc000085f78 pc=0xb73d69
runtime.bgscavenge(0xc000078000)
    /usr/local/go/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc000085fc8 sp=0xc000085fa8 pc=0xb742dc
runtime.gcenable.gowrap2()
    /usr/local/go/src/runtime/mgc.go:204 +0x25 fp=0xc000085fe0 sp=0xc000085fc8 pc=0xb6a905
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000085fe8 sp=0xc000085fe0 pc=0xbc71e1
created by runtime.gcenable in goroutine 1
    /usr/local/go/src/runtime/mgc.go:204 +0xa5

goroutine 5 gp=0xc000068fc0 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000087e20 sp=0xc000087e00 pc=0xbbeb8e
runtime.runfinq()
    /usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc000087fe0 sp=0xc000087e20 pc=0xb69a27
runtime.goexit({})
    /usr/local/go/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc000087fe8 sp=0xc000087fe0 pc=0xbc71e1
created by runtime.createfing in goroutine 1
    /usr/local/go/src/runtime/mfinal.go:163 +0x3d
Loop 2 in 20
restic 0.17.0 compiled with go1.23.0 on windows/amd64
Loop 3 in 20
restic 0.17.0 compiled with go1.23.0 on windows/amd64
Loop 4 in 20
restic 0.17.0 compiled with go1.23.0 on windows/amd64
Loop 5 in 20
restic 0.17.0 compiled with go1.23.0 on windows/amd64
Loop 6 in 20
restic 0.17.0 compiled with go1.23.0 on windows/amd64
Loop 7 in 20
restic 0.17.0 compiled with go1.23.0 on windows/amd64
Loop 8 in 20
restic 0.17.0 compiled with go1.23.0 on windows/amd64
Loop 9 in 20
restic 0.17.0 compiled with go1.23.0 on windows/amd64
Loop 10 in 20
restic 0.17.0 compiled with go1.23.0 on windows/amd64
Loop 11 in 20
restic 0.17.0 compiled with go1.23.0 on windows/amd64
Loop 12 in 20
restic 0.17.0 compiled with go1.23.0 on windows/amd64
Loop 13 in 20
restic 0.17.0 compiled with go1.23.0 on windows/amd64
Loop 14 in 20
restic 0.17.0 compiled with go1.23.0 on windows/amd64
Loop 15 in 20
restic 0.17.0 compiled with go1.23.0 on windows/amd64
Loop 16 in 20
restic 0.17.0 compiled with go1.23.0 on windows/amd64
Loop 17 in 20
restic 0.17.0 compiled with go1.23.0 on windows/amd64
Loop 18 in 20
restic 0.17.0 compiled with go1.23.0 on windows/amd64
Loop 19 in 20
restic 0.17.0 compiled with go1.23.0 on windows/amd64

[distancesprinter]

Returning here to state that, while I cannot get restic 0.17.0 built with go1.21.4 to panic with my iter script, I started using that version believing it to be stable. I can now state that I have encountered a handful of panics and I do not believe 1.21.4 to be immune from whatever bug I am experiencing. Happy to share output if & when someone gets assigned. I wanted to clarify that point in case somebody starts looking at changes between 1.21.4 to 1.21.5 to try to isolate the cause.

[RichieSams]

Just wanted to update here. We're still getting this issue with our custom golang cli tool. (Unfortunately, I can't share it). We're seeing it at a rate of about 1 to 2 crashes a day on our CI servers. Our temporary workaround has been to wrap the call of our tool with a retry. It usually works on the second attempt.

Thank you @distancesprinter for finding other repros. :)

[prattmic]

@distancesprinter

I am running Windows Server 2019 Standard (Desktop Experience) in a bhyve VM.

"In a bhyve VM" stands out to me here as the most non-standard component. Have you managed to reproduce in an environment other than a bhyve VM? (Apologies if I missed it) I wonder if this could be a register save/restore bug in bhyve.

[prattmic]

@RichieSams Do I understand correctly that your crashes are occurring on bare-metal AMD machines?

[distancesprinter]

@distancesprinter

I am running Windows Server 2019 Standard (Desktop Experience) in a bhyve VM.

"In a bhyve VM" stands out to me here as the most non-standard component. Have you managed to reproduce in an environment other than a bhyve VM? (Apologies if I missed it) I wonder if this could be a register save/restore bug in bhyve.

@prattmic I was going to point the finger at bhyve too... but without good justification. It was only a hunch since some SmartOS folks were encountering this--but I stopped myself from jumping to that conclusion because @RichieSams was seeing it on the metal.

You may be interested in reviewing/engaging in the discussion there:

I will note, the issue seemed to involve AMD processors, and mostly seemed to be presenting in Windows VMs where the issue would not occur if the VM was moved to Intel-based hardware.

SmartOS community (which uses bhyve as its hypervisor): https://smartos.topicbox.com/groups/smartos-discuss/T6938eca11dfeb1e3/smartos-discuss-golang-executables-crash-in-windows-vms-on-amd

Which I revived: https://smartos.topicbox.com/groups/smartos-discuss/Tf8f450fd133908fb/smartos-discuss-golang-executables-crash-in-windows-vms-on-amd

I would think that the bhyve folks would be interested in fixing any such bug on AMD hardware, especially as Epyc becomes more common. I think Oxide is using an AMD Epyc processor and bhyve as their hypervisor. I'm sure they'd want to track this down.

All of my AMD EPYC servers are hypervisors in production. Sadly, I cannot blow them away to test on the metal. I'd like to know if others are seeing this on the more affordable Threadripper desktop chips, though. Maybe @RichieSams could provide hardware specs where this is being seen on metal...

I'd also want to know if this is seen consistently on all Epyc hardware or just some Epyc hardware. If AMD shipped bad CPUs, that would terrify me. Mine were all bought at the same time.

[prattmic]

Indeed, the mismatch is suspicious, but there could be different issues.

e.g., it looks like you have only gotten "mcall called on m->g0 stack" and "morestack on g0", while @RichieSams has gotten a slightly wider variety of other corruption-looking crashes.

[prattmic]

In summary, we have:

AMD Ryzen Threadripper 3990X -> Zen 2 -> bare metal crashes AMD EPYC 7543 -> Zen 3 -> Bhyve crashes AMD EPYC 7542 -> Zen 2 -> Bhyve crashes (less frequent) (from https://smartos.topicbox.com/groups/smartos-discuss/T6938eca11dfeb1e3)

[distancesprinter]

@prattmic The frequency of my specific crash might be related to the fact that I was trying to invoke it (it tended to happen reliably after a fresh boot) and probably also related to the fact that all I was doing in my tests was printing the version string for restic over and over again.

I got the following today, which occurred on a host with 10 day uptime:

PS C:\restic> .\backup_general.cmd
open repository
repository 0d6a1559 opened (version 2, compression level auto)
using parent snapshot 1caa731e
load index files
[0:00] 100.00%  34 / 34 index files loaded
start scan on [M:\]
start backup on [M:\]
creating VSS snapshot for [m:\]
successfully created snapshot for [m:\]
                                      0x1 0x1
PC=0x1

runtime: g 0: unknown pc 0x1
0x0} stack=[0x0,0x86a79ff6f0)

runtime: g 0: unknown pc 0x1
stack: frame={sp:0xc000936d30, fp:0x0} stack=[0x0,0x86a79ff6f0)

goroutine 1 [semacquire]:
                )
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000140d30 sp=0xc000140d10 pc=0x4a6f6
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:387
0x0, 0xb1?)  , 0x18?, 0x1[0:01] 1796 files 2.201 GiB, total 3073 files 2.664 GiB, 0 errors,
                                                        0xc000140d98 sp=0xc000140d30 pc=0x59ddc
?)nc.runtime_Semacquire([0:01] 1839 files 2.206 GiB, total 3073 files 2.664 GiB, 0 errors0xc0005ae0f0
                                                                                   pc=0x72a87
(0xe59e80?) files 2.211 GiB, total 3073 files 2.664 GiB, 0 errorssync.(*WaitGroup).Wait

golang.org/x/sync/errgroup.(*Group).Wait(0xc0003c2280)
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:[0:01] 1926 files 2.216 GiB, total 3073 fil +0x27 fp=0xc000140e18 sp=0xc000140df8 pc=0x3c31e7
                                                                                                           0xc00043e120, 0x1, 0x1}, {{0x1934b98, 0x0, 0x0}, {0xc0003b2020, ...}, ...})
 pc=0x47bf92fp=0xc000141030 sp=[0:01] 2009 files 2.225 GiB, total 3073 files 2.664 GiB, 0 errors0xc000140e18
                     , {{{0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}, {0xc00058c040, 0x1, ...}, ...}, ...}, ...)
        /restic/cmd/restic/cmd_backup.go:658 +0x1a28 fp=0xc000141890 sp=0xc000141030 pc=0xb7c468
)ain.glob..func2(0x18c45a0, {0xc00059a0b0, 0x0, 0xb[0:01] 2061 files 2.227 GiB, total 3073 files 2.664 GiB, 0 errors}

               , 0xb, 0xb})
        /home/build/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:985 +0xaaa fp=0xc000141d58 sp=0xc000141bb8 pc=0x3b4aea
          )
        [0:01] 2186 files 2.263 GiB, total 3073 files 2.664 GiB, 0 errors/home/build/go/pkg/mod/github.com/spf13/cobra@v
github.com/spf13/cobra.(*Command).Execute(...)
        /home/build/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:[0:01] 2197 files 2.276 GiB, total 3073 files 2.664 GiB, 0 errors1041
github.com/spf13/cobra.(*Command).ExecuteContext(...)
        /home/build/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1034
main.main()
                                                 0xc000141f80 sp=0xc000141e30 pc=0xbb5dd8
runtime.main()
        /usr/local/go/src/runtime/proc.go:250 +0x1f7 fp=0xc000141fe0 sp=0xc000141f80 pc=0x4a2d7
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000141fe8 sp=0xc000141fe0 pc=0x77a81

goroutine 2 [force gc (idle)]:
0x0?)me.gopark(0x0?, 0x0?[0:01] 2203 files 2.280 GiB, total 3073 files 2.664 GiB, 0 errors, 0x0?, 0x0?,
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000061fb0 sp=0xc000061f90 pc=0x4a6f6[0:01] 2229 files 2.297 Gi
        /usr/local/go/src/runtime/proc.go:387
runtime.forcegchelper()
        /usr/local/go/src/runtime/proc.go:305 +0xb2 fp=0xc000061fe0 sp=0xc000061fb0 pc=0x4a512
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000061fe8 sp=0xc000061fe0 pc=0x77a81
created by runtime.init.6
        /usr/local/go/src/runtime/proc.go:293 +0x25

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000063f80 sp=0xc000063f60 pc=0x4a6f6[0:01] 2247 files 2.317 Gi                    (...)
        /usr/local/go/src/runtime/proc.go:387
(0x0?) 2247 files 2.317 GiB, total 3073 files 2.664 GiB, 0 errorsruntime.bgsweep
                                                  +0xde[0:02] 2248 files 2.317 GiB, total 3073 files 2.664 GiB, 0 errors             pc=0x3489e
runtime.gcenable.func1()
        /usr/local/go/src/runtime/mgc.go:178 +0x26 fp=0xc000063fe0 sp=0xc000063fc8 pc=0x299a6
runtime.goexit([0:02] 2248 files 2.317 GiB, total 3073 files 2.664 GiB, 0 errors)

created by runtime.gcenable
        /usr/local/go/src/runtime/mgc.go:178 +0x6b

goroutine 4 [sleep]:
                                       0x0?, 0xf85f00?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000073f30 sp=0xc000073f10 pc=0x4a6f6

        /usr/local/go/src/runtime/proc.go:387
runtime.(*scavengerState).sleep(0x18de640, 0x4131170000000000)
        /usr/local/go/src/runtime/mgcscavenge.go:479 +0x12a fp=0xc000073fa0 sp=0xc000073f30 pc=0x3296a
runtime.bgscavenge(0x0?)
        /usr/local/go/src/runtime/mgcscavenge.go:637 +0x7f fp=0xc000073fc8 sp=0xc000073fa0 pc=0x32d7f
runtime.gcenable.func2()

        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000073fe8 sp=0xc000073fe0 pc=0x77a81
created by runtime.gcenable
        /usr/local/go/src/runtime/mgc.go:179 +0xaa

   ?, 0xc000065f70?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000065e28 sp=0xc000065e08 pc=0x4a6f6
runtime.runfinq()
        /usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc000065fe0 sp=0xc000065e28 pc=0x28a07
runtime.goexit()
                      sp=0xc000065fe0 pc=0x77a81
created by runtime.createfing
        /usr/local/go/src/runtime/mfinal.go:163 +0x45

goroutine 6 [chan receive]:
               0xc000075e90?, 0xc000075ee8?, 0x91?, 0xa5?, 0xc000075ee8?)
     +0xd6 fp=0xc000075eb8 sp=0xc000075e98 pc=0x4a6f6
               0xc000075fa0, 0x1)
        /usr/local/go/src/runtime/chan.go:583 +0x49d fp=[0:02] 2322 files 2.361 GiB, total 3073 files 2.664 GiB, 0 error, 0xc000075fc0?)    [0:02] 2349 files 2.361 GiB, total 3073 files 2.664 GiB, 0 errors0x1729d

github.com/restic/restic/internal/restic.init.0.func1.1()
        /restic/internal/restic/lock.go:404 +0x9d fp=0xc000075fe0 sp=0xc000075f70 pc=0x471e3d
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000075fe8 sp=0xc000075fe0 pc=[0:02] 2366 files 2.373 GiB, total 3073 files 2.664 GiB, 0 errors0x77a81

        /restic/internal/restic/lock.go:401 +0x2564 GiB, 0 errors

                     ]:
runtime.notetsleepg(0x0?, 0x0?)
        /usr/local/go/src/runtime/lock_sema.go:294 +0x3c fp=0xc00006ffa0 sp=0xc00006ff58 pc=0x1c8dc
os/signal.signal_recv()
        /usr/local/go/src/runtime/sigqueue.go:152 +0x2f fp=0xc00006ffc0 sp=0xc00006ffa0 pc=0x7326f
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:23 +0x19 fp=0xc00006ffe0 sp=0xc00006ffc0 pc=0xda979
runtime.goexit()
          fp=0xc00006ffe8 sp=0xc00006ffe0 pc=0x77a81
created by os/signal.Notify.func1.1
        /usr/local/go/src/os/signal/signal.go:151 +0x2a

goroutine 7 [select]:
runtime.gopark(0xc000423f88?, 0x3?, 0x90?, 0x0?, 0xc000423f72?)

               0xc000423f6c, 0xc000237580?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000423f38 sp=0xc000423df8 pc=0x58cde

        /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0xad fp=0xc000423fc8[0:02] 2460 files                 0xa54b6d
go.opencensus.io/stats/view.init.0.func1()
        /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x26 fp=0xc000423fe0 sp=0xc000423fc8 pc=0xa53da6
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000423fe8 sp=0xc000423fe0 pc=0x77a81
created by go.opencensus.io/stats/view.init.0
        /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x96

goroutine 8 [chan receive]:

        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000425e80 sp=0xc000425e60 pc=0x4a6f6
runtime.chanrecv(0xc0003b1140, 0xc000425f80, 0x1)
        /usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc000425f10 sp=0xc000425e80[0:02] 2498 files 2.447 GiB, total 3
runtime.chanrecv1(0x0?, 0x0?)
        /usr/local/go/src/runtime/chan.go:442 +0x18 fp=0xc000425f38 sp=0xc000425f10 pc=0x16d98
main.cleanupHandler(0x0?, 0xc00043e4e0)
                          0xc000425f38 pc=0xb77d30
main.createGlobalContext.func1()
         fp=0xc000425fe0 sp=0xc000425fc0 pc=[0:02] 2501 files 2.448 GiB, total 3073 files 2.664 GiB, 0 errors0xb77cca

        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000425fe8 sp=0xc000425fe0 pc=0x77a81
created by main.createGlobalContext
                                 +0xa5

goroutine 9 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00041ff50 sp=0xc00041ff30 pc=0x4a6f6
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc00041ffe0 sp=0xc00041ff50 pc=0x2b751
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00041ffe8 sp=0xc00041ffe0 pc=0x77a81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 19 [GC worker (idle)]:

        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000071f50 sp=0xc000071f30 pc=0x4a6f6
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000071fe0 sp=0xc000071f50 pc=0x2b751
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000071fe8 sp=0xc000071fe0 pc=0x77a81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 34 [GC worker (idle)]:
runtime.gopark(0x31faea02dd2e8?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000495f50 sp=0xc000495f30 pc=0x4a6f6
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000495fe0 sp=0xc000495f50 pc=0x2b751
)untime.goexit[0:02] 2568 files 2.498 GiB, total 3073 files 2.664 GiB, 0 errors(
                                             :1598 +0x1 fp=0xc000495fe8 sp=0xc000495fe0 pc=0x77a81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 10 [GC worker (idle)]:
                   ?, 0x14?, 0x0?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000421f50 sp=0xc000421f30 pc=0x4a6f6
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000421fe0 sp=0xc000421f50 pc=0x2b751
runtime.goexit()
                    sp=0xc000421fe0 pc=0x77a81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 11 [syscall, locked to thread]:

       fp=0xc000493608 sp=0xc0004935d0 pc=0x14d4a
syscall.SyscallN(0xc0004936e8?, {0xc0004936a0?, 0x1e3a5b50eb8?, 0x17?})
0x743c9x109 fp=0xc000493680 sp=0xc000493608[0:02] 2625 files 2.527 GiB, total 3073 files 2.664 GiB, 0 errors pc=
                                             ?, 0x82?, 0xc8a1c0?[0:02] 2640 files 2.542 GiB, total 3073 files 2.664 GiB,?, 0xc000493748?, 0x869db?)
                                                         +0x50 fp=0xc0004936e0 sp=0xc000493680 pc=0x74050
syscall.WriteConsole(0xc0004937f0?, 0x0?, 0x41, 0x0?, 0x0?)
        /usr/local/go/src/syscall/zsyscall_windows.go:1154 +0x8a fp=0xc000493758 sp=0xc0004936e0 pc=0x9376a
       0x79420?, 0xc0003a52c0, 0x4c?})
        /usr/local/go/src/internal/poll/fd_windows.go:735 +0x515[0:02] 2728 files 2.566 GiB, total 3073 files 2.664 GiB,                    0xc6bb5
internal/poll.(*FD).Write(0xc000036500, {0xc000514000, 0x41, 0x1000})
  0x245 fp=0xc000493ce8 sp=0xc000493c28 pc=0xc6345
[0:02] 2742 files 2.570 GiB, total 3073 files 2.664 GiB, 0 errorsos.(*File).write(...)

os.(*File).Write(0xc00000a018, {0xc000514000?, 0x41, 0x48007804af0077?})
        /usr/local/go/src/os/file.go:175 +0x65 fp=0xc000493d40 sp=0xc000493ce8 pc=0xd1445
                                  )
        /usr/local/go/src/bufio/bufio.go:628 +0x62 fp=0xc000493d88 sp=0xc000493d40 pc=0x29c802
               {0xc00006b740?, 0x1, 0xc0037ee220?})
        /restic/internal/ui/termstatus/status.go:180 +0x2ec fp=0xc000493e38 sp=0xc000493d88 pc=0x63de8c
github.com/restic/restic/internal/ui/termstatus.(*Terminal).run([0:02] 2775 files 2.579 GiB, total 3073 files 2.664 GiB,              0xc00042c000})
        /restic/internal/ui/termstatus/status.go:153 +0x1c6 fp=0xc000493f50 sp=0xc000493e38 pc=0x63d906
github.com/restic/restic/internal/ui/termstatus.(*Terminal).Run(0x0?, {0x10d5f88?, 0xc00042c000?}[0:02] 2791 files 2.582                                                :94 +0x68 fp=0xc000493f98 sp=0xc000493f50 pc=0x63d668
main.setupTermstatus.func1()
         fp=0xc000493fe0 sp=0xc000493f98 pc=0xbb8265
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000493fe8 sp=0xc000493fe0 pc=[0:02] 2912 files 2.610 GiB, total 3073 files 2.664 GiB, 0 errors0x77a81

        /restic/cmd/restic/termstatus.go:26 +0x138

goroutine 14 [select]:
runtime.gopark(0xc00053ff08?, 0x3?, 0xaa?, 0xde?, 0xc00053fe8a?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00053fcc8 sp=0xc00053fca8 pc=0x4a6f6
runtime.selectgo(0xc00053ff08, 0xc00053fe84, 0x0?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc00053fe08 sp=0xc00053fcc8 pc=0x58cde
[0:02] 3004 files 2.637 GiB, total 3073 files 2.664 GiB, 0 errorsgithub.com/restic/restic/internal/repository.(*locker).}, 0xc000486040, 0xc000536000, 0xc000536060, 0xf859a8)les 2.664 GiB, 0 errors0xc000432db0
                                                          0xc00053ff88 sp=0xc00053fe08 pc=0x6240ea
github.com/restic/restic/internal/repository.(*locker).Lock.func1()
        /restic/internal/repository/lock.go:105 +0x46 fp=0xc00053ffe0 sp=0xc00053ff88 pc=0x623e86
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00053ffe8 sp=0xc00053ffe0 pc=0x77a81
created by github.com/restic/restic/internal/repository.(*locker).Lock

goroutine 43 [IO wait]:
                                               0xd6 fp=0xc0004a35a8 sp=0xc0004a3588 pc=0x4a6f6
                              , 0x0?)
        /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc0004a35e0 sp=0xc0004a35a8 pc=0x41157
internal/poll.runtime_pollWait(0x1e3cb7a0f68, 0x72)
        /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc0004a3600 sp=0xc0004a35e0[0:02] 3215 files 2.769 GiB, total0x713c9iles 2.664 GiB, 0 errors pc=
                               0x5a82289861331c74?, 0xc0004a3710?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc0004a3628 sp=0xc0004a3600 pc=0xc2692
                                                     :175 +0xf7 fp=0xc0004a3698 sp=0xc0004a3628 pc=0xc3d17[0:03] 3254 fi, 0xc00})        0xc00[0:03] 3269 files 2.784 GiB, total 3073 files 2.664 GiB, 0 errors
        /usr/local/go/src/internal/poll/fd_windows.go:436 +0x2b8 fp=0xc0004a3740 sp=0xc0004a3698 pc=0xc4a18
                                                0xc0004bc005?, 0xbcb?})
        /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc0004a3788 sp=0xc0004a3740 pc=0x1faf29
   0xc0004bc000?, 0xc0004a37f0?, 0xc0000de530?})
        /usr/local/go/src/net/net.go:183 +0x45 fp=0xc0004a37d0 sp=0xc0004a3788 pc=0x210905
              , {0xc0004bc000?, 0xc0005ae498?, 0x18?})
        <autogenerated>:1 +0x29 fp=0xc0004a3800 sp=0xc0004a37d0[0:03] 3332 files 2.847 GiB, total 3073 files 2.664 GiB, 
(0xc0005ae498, {0xc0004bc000?, 0xc0005ae498?, 0x0?})GiB, 0 errorscrypto/tls.(*atLeastReader).Read
        /usr/local/go/src/crypto/tls/conn.go:788 +0x3d fp=0xc0004a3848 sp=0xc0004a3800 pc=0x26287d
bytes.(*Buffer).ReadFrom(0xc0000de610, {0x10c9140, 0xc0005ae498})
        /usr/local/go/src/bytes/buffer.go:202 +0x98 fp=0xc0004a38a0 sp=0xc0004a3848 pc=0x132358
                                                             ?, 0xc00058a060}, 0xc00?)
        /usr/local/go/src/crypto/tls/conn.go:810 +0xe5 fp=0xc0004a38e0 sp=0xc0004a38a0 pc=0x262a65
                                         +0x116 fp=0xc0004a3c40 sp=0xc0004a38e0 pc=0x25ff56
(...)] 3401 files 2.943 GiB, total 3073 files 2.664 GiB, 0 errorscrypto/tls.(*Conn).readRecord
                                             583
crypto/tls.(*Conn).Read(0xc0000de380, {0xc000011000, 0x1000, 0x0?})
        /usr/local/go/src/crypto/tls/conn.go:1316 +0x16f fp=0xc0004a3cb0 sp=0xc0004a3c40 pc=0x265e4f
bufio.(*Reader).Read(0xc00048c960, {0xc0001c72a0, 0x9, 0xc0005fc088?})
        /usr/local/go/src/bufio/bufio.go:237 +0x1bb fp=0xc0004a3ce8 sp=0xc0004a3cb0 pc=0x29b15b
io.ReadAtLeast({0x10c7f80, 0xc00048c960}, {0xc0001c72a0, 0x9, 0x9}, 0x9)

io.ReadFull(...)
        /usr/local/go/src/io/io.go:351
})c00048c960[0:03] 3418 files 2.946 GiB, total 3073 files 2.664 GiB, 0 errors?
                                                                       237 +0x6e fp=0xc0004a3d80 sp=0xc0004a3d30 pc=0x43468e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0001c7260)
        /home/build/go/pkg/mod/golang.org/x/net@v0.26.0/http2/frame.go:501 +0x95 fp=0xc0004a3e30 sp=0xc0004a3d80 pc=0x434ed5
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc0004a3f98)
            fp=0xc0004a3f60 sp=0xc0004a3e30 pc=0x4481b8
)olang.org/x/net/http2.(*ClientConn).readLoop([0:03] 3480 files 2.977 GiB, total 3073 files 2.664 GiB, 0 errors0xc000002
        /home/build/go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:2254 +0x99 fp=0xc0004a3fc8 sp=0xc0004a3f60 pc=0x447759
                                                        )
        /home/build/go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:869 +0x26 fp=0xc0004a3fe0 sp=0xc0004a3fc8 pc=0x4400e6
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0004a3fe8 sp=0xc0004a3fe0 pc=[0:03] 3500 files 2.998 GiB, total 3073 files 2.664 GiB, 0 errors0x77a81

        /home/build/go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:869 +0xc9f

goroutine 15 [select]:

        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000017ce8 sp=0xc000017cc8 pc=0x4a6f6
          ?, 0x0, 0x82ab7?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000017e28 sp=0xc000017ce8 pc=0x58cde
github.com/restic/restic/internal/repository.(*locker).monitorLockRefresh(0x1841cc0, {0x10d5f88, 0xc0005260f0}[0:03] 356              , 0xc000536060, 0xf859a8)
        /restic/internal/repository/lock.go:208 +0x23e fp=0xc000017f98 sp=0xc000017e28 pc=0x62489e

        /restic/internal/repository/lock.go:106 +0x3e fp=0xc000017fe0 sp=0xc000017f98 pc=0x623dfe
runtime.goexit()
                                           pc=0x77a81
created by github.com/restic/restic/internal/repository.(*locker).Lock
        /restic/internal/repository/lock.go:106 +0x7ec

goroutine 16 [select]:
                  ?)
0x4a6f6                   [0:03] 3623 files 3.142 GiB, total 3073 files 2.664 GiB, 0 errors0xc00013d9f0 pc=
runtime.selectgo(0xc00013dba0, 0xc00013db80, 0x1?, 0x1, 0x1e3a5b50eb8?, 0x1)
                                                          0xc00013db50 sp=0xc00013da10 pc=0x58cde
github.com/restic/restic/internal/ui/termstatus.(*Terminal).SetStatus(0xc0003b0000, {0xc005855be0, 0x1, 0x1})
    +0x10e fp=0xc00013dbd0 sp=0xc00013db50 pc=0x63e96e
unexpected signal during runtime executionles 2.664 GiB, 0 errorsfatal error:
[signal 0xc0000005 code=0x0 addr=0xe8 pc=0x68f5c]

runtime stack:
runtime.throw({0xf1dea5?, 0x170c018?})
        /usr/local/go/src/runtime/panic.go:1047 +0x65 fp=0xc000935b20 sp=0xc000935af0 pc=0x47b65
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_windows.go:251 +0x265 fp=0xc000935b68 sp=0xc000935b20 pc=0x5b465
runtime.gentraceback(0x7ffc00000000?, 0x58?, 0xc00010c000?, 0xc000455520, 0x0, 0x0, 0x64, 0x0, 0x7ffc9d0a2a70?, 0x0)
        /usr/local/go/src/runtime/traceback.go:258 +0x8bc fp=0xc000935ec8 sp=0xc000935b68 pc=0x68f5c
runtime.traceback1(0xc000455520?, 0x49600?, 0xc00093614d?, 0xc000455520, 0xc0009360e0?)
        /usr/local/go/src/runtime/traceback.go:776 +0x1b6 fp=0xc000936088 sp=0xc000935ec8 pc=0x6b056
runtime.traceback(...)
        /usr/local/go/src/runtime/traceback.go:723
runtime.tracebackothers.func1(0xc000455520)
        /usr/local/go/src/runtime/traceback.go:992 +0xe5 fp=0xc0009360c8 sp=0xc000936088 pc=0x6bf85
runtime.forEachGRace(0xc000936130)
        /usr/local/go/src/runtime/proc.go:604 +0x4d fp=0xc0009360f8 sp=0xc0009360c8 pc=0x4b22d
runtime.tracebackothers(0xedfb71?)
        /usr/local/go/src/runtime/traceback.go:978 +0xe5 fp=0xc000936160 sp=0xc0009360f8 pc=0x6be65
runtime.winthrow(0xbcc5a395a4960d4f?, 0xc000936600, 0xc00010a1a0)
        /usr/local/go/src/runtime/signal_windows.go:237 +0x22f fp=0xc0009361c0 sp=0xc000936160 pc=0x5b1cf
runtime.lastcontinuehandler(0x379f0f0643d30d34?, 0xf12441279228315b?, 0x56378454879cd1d2?)
        /usr/local/go/src/runtime/signal_windows.go:198 +0x48 fp=0xc0009361e8 sp=0xc0009361c0 pc=0x5af88
runtime.lastcontinuehandler(0xc000936af0, 0xc000936600, 0xc00010a1a0)
        <autogenerated>:1 +0x34 fp=0xc000936210 sp=0xc0009361e8 pc=0x7a4d4
sigtramp()
        /usr/local/go/src/runtime/sys_windows_amd64.s:166 +0x102 fp=0xc000936330 sp=0xc000936210 pc=0x79302

goroutine 1 [semacquire]:
runtime.gopark(0xc000140d38?, 0x529e0?, 0xc0?, 0x60?, 0xc00005e000?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000140d30 sp=0xc000140d10 pc=0x4a6f6
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:387
runtime.semacquire1(0xc0003c2290, 0x18?, 0x1, 0x0, 0xb1?)
        /usr/local/go/src/runtime/sema.go:160 +0x21c fp=0xc000140d98 sp=0xc000140d30 pc=0x59ddc
sync.runtime_Semacquire(0xc0005ae0f0?)
        /usr/local/go/src/runtime/sema.go:62 +0x27 fp=0xc000140dd0 sp=0xc000140d98 pc=0x72a87
sync.(*WaitGroup).Wait(0xe59e80?)
        /usr/local/go/src/sync/waitgroup.go:116 +0x4b fp=0xc000140df8 sp=0xc000140dd0 pc=0x862cb
golang.org/x/sync/errgroup.(*Group).Wait(0xc0003c2280)
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:56 +0x27 fp=0xc000140e18 sp=0xc000140df8 pc=0x3c31e7
github.com/restic/restic/internal/archiver.(*Archiver).Snapshot(0xc00007e000, {0x10d5f88?, 0xc0005260f0}, {0xc00043e120, 0x1, 0x1}, {{0x1934b98, 0x0, 0x0}, {0xc0003b2020, ...}, ...})
        /restic/internal/archiver/archiver.go:859 +0x372 fp=0xc000141030 sp=0xc000140e18 pc=0x47bf92
main.runBackup({_, _}, {{{0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}, {0xc00058c040, 0x1, ...}, ...}, ...}, ...)
        /restic/cmd/restic/cmd_backup.go:658 +0x1a28 fp=0xc000141890 sp=0xc000141030 pc=0xb7c468
main.glob..func2(0x18c45a0, {0xc00059a0b0, 0x0, 0xb})
        /restic/cmd/restic/cmd_backup.go:61 +0x116 fp=0xc000141bb8 sp=0xc000141890 pc=0xb78156
github.com/spf13/cobra.(*Command).execute(0x18c45a0, {0xc00059a000, 0xb, 0xb})
        /home/build/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:985 +0xaaa fp=0xc000141d58 sp=0xc000141bb8 pc=0x3b4aea
github.com/spf13/cobra.(*Command).ExecuteC(0x18c3a20)
        /home/build/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1117 +0x425 fp=0xc000141e30 sp=0xc000141d58 pc=0x3b5405
github.com/spf13/cobra.(*Command).Execute(...)
        /home/build/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041
github.com/spf13/cobra.(*Command).ExecuteContext(...)
        /home/build/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1034
main.main()
        /restic/cmd/restic/main.go:125 +0x258 fp=0xc000141f80 sp=0xc000141e30 pc=0xbb5dd8
runtime.main()
        /usr/local/go/src/runtime/proc.go:250 +0x1f7 fp=0xc000141fe0 sp=0xc000141f80 pc=0x4a2d7
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000141fe8 sp=0xc000141fe0 pc=0x77a81

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000061fb0 sp=0xc000061f90 pc=0x4a6f6
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:387
runtime.forcegchelper()
        /usr/local/go/src/runtime/proc.go:305 +0xb2 fp=0xc000061fe0 sp=0xc000061fb0 pc=0x4a512
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000061fe8 sp=0xc000061fe0 pc=0x77a81
created by runtime.init.6
        /usr/local/go/src/runtime/proc.go:293 +0x25

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000063f80 sp=0xc000063f60 pc=0x4a6f6
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:387
runtime.bgsweep(0x0?)
        /usr/local/go/src/runtime/mgcsweep.go:319 +0xde fp=0xc000063fc8 sp=0xc000063f80 pc=0x3489e
runtime.gcenable.func1()
        /usr/local/go/src/runtime/mgc.go:178 +0x26 fp=0xc000063fe0 sp=0xc000063fc8 pc=0x299a6
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000063fe8 sp=0xc000063fe0 pc=0x77a81
created by runtime.gcenable
        /usr/local/go/src/runtime/mgc.go:178 +0x6b

goroutine 4 [GC scavenge wait]:
runtime.gopark(0x31faec9ac3178?, 0x1c026f4?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000073f70 sp=0xc000073f50 pc=0x4a6f6
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:387
runtime.(*scavengerState).park(0x18de640)
        /usr/local/go/src/runtime/mgcscavenge.go:400 +0x53 fp=0xc000073fa0 sp=0xc000073f70 pc=0x32753
runtime.bgscavenge(0x0?)
        /usr/local/go/src/runtime/mgcscavenge.go:633 +0x65 fp=0xc000073fc8 sp=0xc000073fa0 pc=0x32d65
runtime.gcenable.func2()
        /usr/local/go/src/runtime/mgc.go:179 +0x26 fp=0xc000073fe0 sp=0xc000073fc8 pc=0x29946
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000073fe8 sp=0xc000073fe0 pc=0x77a81
created by runtime.gcenable
        /usr/local/go/src/runtime/mgc.go:179 +0xaa

goroutine 5 [finalizer wait]:
runtime.gopark(0x4aa72?, 0x1e3a5b5c0f8?, 0x0?, 0x0?, 0xc000065f70?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000065e28 sp=0xc000065e08 pc=0x4a6f6
runtime.runfinq()
        /usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc000065fe0 sp=0xc000065e28 pc=0x28a07
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000065fe8 sp=0xc000065fe0 pc=0x77a81
created by runtime.createfing
        /usr/local/go/src/runtime/mfinal.go:163 +0x45

goroutine 6 [chan receive]:
runtime.gopark(0xc000075e90?, 0xc000075ee8?, 0x91?, 0xa5?, 0xc000075ee8?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000075eb8 sp=0xc000075e98 pc=0x4a6f6
runtime.chanrecv(0xc000100000, 0xc000075fa0, 0x1)
        /usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc000075f48 sp=0xc000075eb8 pc=0x1729d
runtime.chanrecv2(0xc000100000?, 0xc000075fc0?)
        /usr/local/go/src/runtime/chan.go:447 +0x18 fp=0xc000075f70 sp=0xc000075f48 pc=0x16dd8
github.com/restic/restic/internal/restic.init.0.func1.1()
        /restic/internal/restic/lock.go:404 +0x9d fp=0xc000075fe0 sp=0xc000075f70 pc=0x471e3d
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000075fe8 sp=0xc000075fe0 pc=0x77a81
created by github.com/restic/restic/internal/restic.init.0.func1
        /restic/internal/restic/lock.go:401 +0x25

goroutine 18 [syscall]:
runtime.notetsleepg(0x0?, 0x0?)
        /usr/local/go/src/runtime/lock_sema.go:294 +0x3c fp=0xc00006ffa0 sp=0xc00006ff58 pc=0x1c8dc
os/signal.signal_recv()
        /usr/local/go/src/runtime/sigqueue.go:152 +0x2f fp=0xc00006ffc0 sp=0xc00006ffa0 pc=0x7326f
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:23 +0x19 fp=0xc00006ffe0 sp=0xc00006ffc0 pc=0xda979
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00006ffe8 sp=0xc00006ffe0 pc=0x77a81
created by os/signal.Notify.func1.1
        /usr/local/go/src/os/signal/signal.go:151 +0x2a

goroutine 7 [select]:
runtime.gopark(0xc000423f88?, 0x3?, 0x90?, 0x0?, 0xc000423f72?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000423df8 sp=0xc000423dd8 pc=0x4a6f6
runtime.selectgo(0xc000423f88, 0xc000423f6c, 0xc000237580?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000423f38 sp=0xc000423df8 pc=0x58cde
go.opencensus.io/stats/view.(*worker).start(0xc000237580)
        /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0xad fp=0xc000423fc8 sp=0xc000423f38 pc=0xa54b6d
go.opencensus.io/stats/view.init.0.func1()
        /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x26 fp=0xc000423fe0 sp=0xc000423fc8 pc=0xa53da6
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000423fe8 sp=0xc000423fe0 pc=0x77a81
created by go.opencensus.io/stats/view.init.0
        /home/build/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x96

goroutine 8 [chan receive]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000425e80 sp=0xc000425e60 pc=0x4a6f6
runtime.chanrecv(0xc0003b1140, 0xc000425f80, 0x1)
        /usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc000425f10 sp=0xc000425e80 pc=0x1729d
runtime.chanrecv1(0x0?, 0x0?)
        /usr/local/go/src/runtime/chan.go:442 +0x18 fp=0xc000425f38 sp=0xc000425f10 pc=0x16d98
main.cleanupHandler(0x0?, 0xc00043e4e0)
        /restic/cmd/restic/cleanup.go:24 +0x30 fp=0xc000425fc0 sp=0xc000425f38 pc=0xb77d30
main.createGlobalContext.func1()
        /restic/cmd/restic/cleanup.go:16 +0x2a fp=0xc000425fe0 sp=0xc000425fc0 pc=0xb77cca
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000425fe8 sp=0xc000425fe0 pc=0x77a81
created by main.createGlobalContext
        /restic/cmd/restic/cleanup.go:16 +0xa5

goroutine 9 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00041ff50 sp=0xc00041ff30 pc=0x4a6f6
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc00041ffe0 sp=0xc00041ff50 pc=0x2b751
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00041ffe8 sp=0xc00041ffe0 pc=0x77a81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 19 [GC worker (idle)]:
runtime.gopark(0x31faeaa04d104?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000071f50 sp=0xc000071f30 pc=0x4a6f6
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000071fe0 sp=0xc000071f50 pc=0x2b751
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000071fe8 sp=0xc000071fe0 pc=0x77a81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 34 [GC worker (idle)]:
runtime.gopark(0x31faea02dd2e8?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000495f50 sp=0xc000495f30 pc=0x4a6f6
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000495fe0 sp=0xc000495f50 pc=0x2b751
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000495fe8 sp=0xc000495fe0 pc=0x77a81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 10 [GC worker (idle)]:
runtime.gopark(0x31faeaa04d104?, 0xc00048e040?, 0x1a?, 0x14?, 0x0?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000421f50 sp=0xc000421f30 pc=0x4a6f6
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000421fe0 sp=0xc000421f50 pc=0x2b751
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000421fe8 sp=0xc000421fe0 pc=0x77a81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1199 +0x25

goroutine 11 [select]:
runtime.gopark(0xc000493f10?, 0x3?, 0x36?, 0xd0?, 0xc000493eaa?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000493cf8 sp=0xc000493cd8 pc=0x4a6f6
runtime.selectgo(0xc000493f10, 0xc000493ea4, 0x1?, 0x0, 0xc8a000?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000493e38 sp=0xc000493cf8 pc=0x58cde
github.com/restic/restic/internal/ui/termstatus.(*Terminal).run(0xc0003b0000, {0x10d5f88, 0xc00042c000})
        /restic/internal/ui/termstatus/status.go:105 +0x125 fp=0xc000493f50 sp=0xc000493e38 pc=0x63d865
github.com/restic/restic/internal/ui/termstatus.(*Terminal).Run(0x0?, {0x10d5f88?, 0xc00042c000?})
        /restic/internal/ui/termstatus/status.go:94 +0x68 fp=0xc000493f98 sp=0xc000493f50 pc=0x63d668
main.setupTermstatus.func1()
        /restic/cmd/restic/termstatus.go:28 +0x65 fp=0xc000493fe0 sp=0xc000493f98 pc=0xbb8265
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000493fe8 sp=0xc000493fe0 pc=0x77a81
created by main.setupTermstatus
        /restic/cmd/restic/termstatus.go:26 +0x138

goroutine 14 [select]:
runtime.gopark(0xc00053ff08?, 0x3?, 0xaa?, 0xde?, 0xc00053fe8a?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00053fcc8 sp=0xc00053fca8 pc=0x4a6f6
runtime.selectgo(0xc00053ff08, 0xc00053fe84, 0x0?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc00053fe08 sp=0xc00053fcc8 pc=0x58cde
github.com/restic/restic/internal/repository.(*locker).refreshLocks(0x1841cc0, {0x10d5f88, 0xc0005260f0}, {0x10dcce8, 0xc000432db0}, 0xc000486040, 0xc000536000, 0xc000536060, 0xf859a8)
        /restic/internal/repository/lock.go:144 +0x22a fp=0xc00053ff88 sp=0xc00053fe08 pc=0x6240ea
github.com/restic/restic/internal/repository.(*locker).Lock.func1()
        /restic/internal/repository/lock.go:105 +0x46 fp=0xc00053ffe0 sp=0xc00053ff88 pc=0x623e86
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00053ffe8 sp=0xc00053ffe0 pc=0x77a81
created by github.com/restic/restic/internal/repository.(*locker).Lock
        /restic/internal/repository/lock.go:105 +0x705

goroutine 43 [IO wait]:
runtime.gopark(0x0?, 0xc000037698?, 0x48?, 0x77?, 0xc0000376c8?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0004a35a8 sp=0xc0004a3588 pc=0x4a6f6
runtime.netpollblock(0xfbcb680f0d32a771?, 0x1480f?, 0x0?)
        /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc0004a35e0 sp=0xc0004a35a8 pc=0x41157
internal/poll.runtime_pollWait(0x1e3cb7a0f68, 0x72)
        /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc0004a3600 sp=0xc0004a35e0 pc=0x713c9
internal/poll.(*pollDesc).wait(0x5a82289861331c74?, 0xc0004a3710?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc0004a3628 sp=0xc0004a3600 pc=0xc2692
internal/poll.execIO(0xc000037698, 0xf858c8)
        /usr/local/go/src/internal/poll/fd_windows.go:175 +0xf7 fp=0xc0004a3698 sp=0xc0004a3628 pc=0xc3d17
internal/poll.(*FD).Read(0xc000037680, {0xc0004bc000, 0xc00, 0xc00})
        /usr/local/go/src/internal/poll/fd_windows.go:436 +0x2b8 fp=0xc0004a3740 sp=0xc0004a3698 pc=0xc4a18
net.(*netFD).Read(0xc000037680, {0xc0004bc000?, 0xc0004bc005?, 0xbcb?})
        /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc0004a3788 sp=0xc0004a3740 pc=0x1faf29
net.(*conn).Read(0xc00058a060, {0xc0004bc000?, 0xc0004a37f0?, 0xc0000de530?})
        /usr/local/go/src/net/net.go:183 +0x45 fp=0xc0004a37d0 sp=0xc0004a3788 pc=0x210905
net.(*TCPConn).Read(0xc0004a3868?, {0xc0004bc000?, 0xc0005ae498?, 0x18?})
        <autogenerated>:1 +0x29 fp=0xc0004a3800 sp=0xc0004a37d0 pc=0x2224e9
crypto/tls.(*atLeastReader).Read(0xc0005ae498, {0xc0004bc000?, 0xc0005ae498?, 0x0?})
        /usr/local/go/src/crypto/tls/conn.go:788 +0x3d fp=0xc0004a3848 sp=0xc0004a3800 pc=0x26287d
bytes.(*Buffer).ReadFrom(0xc0000de610, {0x10c9140, 0xc0005ae498})
        /usr/local/go/src/bytes/buffer.go:202 +0x98 fp=0xc0004a38a0 sp=0xc0004a3848 pc=0x132358
crypto/tls.(*Conn).readFromUntil(0xc0000de380, {0x1e3cb7a1058?, 0xc00058a060}, 0xc00?)
        /usr/local/go/src/crypto/tls/conn.go:810 +0xe5 fp=0xc0004a38e0 sp=0xc0004a38a0 pc=0x262a65
crypto/tls.(*Conn).readRecordOrCCS(0xc0000de380, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:617 +0x116 fp=0xc0004a3c40 sp=0xc0004a38e0 pc=0x25ff56
crypto/tls.(*Conn).readRecord(...)
        /usr/local/go/src/crypto/tls/conn.go:583
crypto/tls.(*Conn).Read(0xc0000de380, {0xc000011000, 0x1000, 0x0?})
        /usr/local/go/src/crypto/tls/conn.go:1316 +0x16f fp=0xc0004a3cb0 sp=0xc0004a3c40 pc=0x265e4f
bufio.(*Reader).Read(0xc00048c960, {0xc0001c72a0, 0x9, 0xc0005fc088?})
        /usr/local/go/src/bufio/bufio.go:237 +0x1bb fp=0xc0004a3ce8 sp=0xc0004a3cb0 pc=0x29b15b
io.ReadAtLeast({0x10c7f80, 0xc00048c960}, {0xc0001c72a0, 0x9, 0x9}, 0x9)
        /usr/local/go/src/io/io.go:332 +0x9a fp=0xc0004a3d30 sp=0xc0004a3ce8 pc=0xbb37a
io.ReadFull(...)
        /usr/local/go/src/io/io.go:351
golang.org/x/net/http2.readFrameHeader({0xc0001c72a0?, 0x9?, 0x0?}, {0x10c7f80?, 0xc00048c960?})
        /home/build/go/pkg/mod/golang.org/x/net@v0.26.0/http2/frame.go:237 +0x6e fp=0xc0004a3d80 sp=0xc0004a3d30 pc=0x43468e
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0001c7260)
        /home/build/go/pkg/mod/golang.org/x/net@v0.26.0/http2/frame.go:501 +0x95 fp=0xc0004a3e30 sp=0xc0004a3d80 pc=0x434ed5
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc0004a3f98)
        /home/build/go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:2358 +0xd8 fp=0xc0004a3f60 sp=0xc0004a3e30 pc=0x4481b8
golang.org/x/net/http2.(*ClientConn).readLoop(0xc000002f00)
        /home/build/go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:2254 +0x99 fp=0xc0004a3fc8 sp=0xc0004a3f60 pc=0x447759
golang.org/x/net/http2.(*Transport).newClientConn.func1()
        /home/build/go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:869 +0x26 fp=0xc0004a3fe0 sp=0xc0004a3fc8 pc=0x4400e6
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0004a3fe8 sp=0xc0004a3fe0 pc=0x77a81
created by golang.org/x/net/http2.(*Transport).newClientConn
        /home/build/go/pkg/mod/golang.org/x/net@v0.26.0/http2/transport.go:869 +0xc9f

goroutine 15 [select]:
runtime.gopark(0xc000017f40?, 0x4?, 0x0?, 0x0?, 0xc000017e80?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000017ce8 sp=0xc000017cc8 pc=0x4a6f6
runtime.selectgo(0xc000017f40, 0xc000017e78, 0x66db0a03?, 0x0, 0x82ab7?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000017e28 sp=0xc000017ce8 pc=0x58cde
github.com/restic/restic/internal/repository.(*locker).monitorLockRefresh(0x1841cc0, {0x10d5f88, 0xc0005260f0}, 0xc000486040, 0xc000536000, 0xc000536060, 0xf859a8)
        /restic/internal/repository/lock.go:208 +0x23e fp=0xc000017f98 sp=0xc000017e28 pc=0x62489e
github.com/restic/restic/internal/repository.(*locker).Lock.func2()
        /restic/internal/repository/lock.go:106 +0x3e fp=0xc000017fe0 sp=0xc000017f98 pc=0x623dfe
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000017fe8 sp=0xc000017fe0 pc=0x77a81
created by github.com/restic/restic/internal/repository.(*locker).Lock
        /restic/internal/repository/lock.go:106 +0x7ec

goroutine 16 [select]:
runtime.gopark(0xc00013df78?, 0x3?, 0x0?, 0xa0?, 0xc00013df0a?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00013dd90 sp=0xc00013dd70 pc=0x4a6f6
runtime.selectgo(0xc00013df78, 0xc00013df04, 0x18de1e0?, 0x0, 0x356cdb01?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc00013ded0 sp=0xc00013dd90 pc=0x58cde
github.com/restic/restic/internal/ui/progress.(*Updater).run(0xc000076300)
        /restic/internal/ui/progress/updater.go:73 +0x186 fp=0xc00013dfc8 sp=0xc00013ded0 pc=0x41cca6
github.com/restic/restic/internal/ui/progress.NewUpdater.func1()
        /restic/internal/ui/progress/updater.go:40 +0x26 fp=0xc00013dfe0 sp=0xc00013dfc8 pc=0x41ca26
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00013dfe8 sp=0xc00013dfe0 pc=0x77a81
created by github.com/restic/restic/internal/ui/progress.NewUpdater
        /restic/internal/ui/progress/updater.go:40 +0x16d

goroutine 93 [semacquire]:
runtime.gopark(0x0?, 0x0?, 0x60?, 0xc3?, 0x2?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00054fe78 sp=0xc00054fe58 pc=0x4a6f6
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:387
runtime.semacquire1(0xc0003c22d0, 0x39?, 0x1, 0x0, 0xa5?)
        /usr/local/go/src/runtime/sema.go:160 +0x21c fp=0xc00054fee0 sp=0xc00054fe78 pc=0x59ddc
sync.runtime_Semacquire(0xc00054ff30?)
        /usr/local/go/src/runtime/sema.go:62 +0x27 fp=0xc00054ff18 sp=0xc00054fee0 pc=0x72a87
sync.(*WaitGroup).Wait(0xc00054ffa8?)
        /usr/local/go/src/sync/waitgroup.go:116 +0x4b fp=0xc00054ff40 sp=0xc00054ff18 pc=0x862cb
golang.org/x/sync/errgroup.(*Group).Wait(0xc0003c22c0)
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:56 +0x27 fp=0xc00054ff60 sp=0xc00054ff40 pc=0x3c31e7
github.com/restic/restic/internal/repository.(*Repository).StartPackUploader.func1()
        /restic/internal/repository/repository.go:536 +0x1d fp=0xc00054ff78 sp=0xc00054ff60 pc=0x6317fd
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc00054ffe0 sp=0xc00054ff78 pc=0x3c3384
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00054ffe8 sp=0xc00054ffe0 pc=0x77a81
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 88 [select]:
runtime.gopark(0xc0004a5f48?, 0x2?, 0xaa?, 0xde?, 0xc0004a5f14?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0004a5d90 sp=0xc0004a5d70 pc=0x4a6f6
runtime.selectgo(0xc0004a5f48, 0xc0004a5f10, 0xf3c217925d291cf6?, 0x0, 0x2d4c6546ef6ce042?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0004a5ed0 sp=0xc0004a5d90 pc=0x58cde
github.com/restic/restic/internal/repository.newPackerUploader.func1()
        /restic/internal/repository/packer_uploader.go:32 +0xce fp=0xc0004a5f78 sp=0xc0004a5ed0 pc=0x6264ee
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc0004a5fe0 sp=0xc0004a5f78 pc=0x3c3384
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0004a5fe8 sp=0xc0004a5fe0 pc=0x77a81
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 55 [running]:
        goroutine running on other thread; stack unavailable
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 91 [select]:
runtime.gopark(0xc000539f48?, 0x2?, 0x0?, 0x0?, 0xc000539f14?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000539d90 sp=0xc000539d70 pc=0x4a6f6
runtime.selectgo(0xc000539f48, 0xc000539f10, 0x0?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000539ed0 sp=0xc000539d90 pc=0x58cde
github.com/restic/restic/internal/repository.newPackerUploader.func1()
        /restic/internal/repository/packer_uploader.go:32 +0xce fp=0xc000539f78 sp=0xc000539ed0 pc=0x6264ee
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc000539fe0 sp=0xc000539f78 pc=0x3c3384
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000539fe8 sp=0xc000539fe0 pc=0x77a81
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 89 [select]:
runtime.gopark(0xc00054df48?, 0x2?, 0xf0?, 0xff?, 0xc00054df14?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00054dd90 sp=0xc00054dd70 pc=0x4a6f6
runtime.selectgo(0xc00054df48, 0xc00054df10, 0xd31810a3b508dfda?, 0x0, 0x3d6b6b16fa01f7ce?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc00054ded0 sp=0xc00054dd90 pc=0x58cde
github.com/restic/restic/internal/repository.newPackerUploader.func1()
        /restic/internal/repository/packer_uploader.go:32 +0xce fp=0xc00054df78 sp=0xc00054ded0 pc=0x6264ee
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc00054dfe0 sp=0xc00054df78 pc=0x3c3384
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00054dfe8 sp=0xc00054dfe0 pc=0x77a81
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 90 [select]:
runtime.gopark(0xc000019f48?, 0x2?, 0xe0?, 0xc6?, 0xc000019f14?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000019d90 sp=0xc000019d70 pc=0x4a6f6
runtime.selectgo(0xc000019f48, 0xc000019f10, 0x46754b?, 0x0, 0x10d5f88?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000019ed0 sp=0xc000019d90 pc=0x58cde
github.com/restic/restic/internal/repository.newPackerUploader.func1()
        /restic/internal/repository/packer_uploader.go:32 +0xce fp=0xc000019f78 sp=0xc000019ed0 pc=0x6264ee
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc000019fe0 sp=0xc000019f78 pc=0x3c3384
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000019fe8 sp=0xc000019fe0 pc=0x77a81
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 92 [select]:
runtime.gopark(0xc0004a1f48?, 0x2?, 0x0?, 0x0?, 0xc0004a1f14?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0004a1d90 sp=0xc0004a1d70 pc=0x4a6f6
runtime.selectgo(0xc0004a1f48, 0xc0004a1f10, 0x11fca61c5b87c40d?, 0x0, 0xab6044141f294594?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0004a1ed0 sp=0xc0004a1d90 pc=0x58cde
github.com/restic/restic/internal/repository.newPackerUploader.func1()
        /restic/internal/repository/packer_uploader.go:32 +0xce fp=0xc0004a1f78 sp=0xc0004a1ed0 pc=0x6264ee
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc0004a1fe0 sp=0xc0004a1f78 pc=0x3c3384
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0004a1fe8 sp=0xc0004a1fe0 pc=0x77a81
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 94 [semacquire]:
runtime.gopark(0x1936440?, 0x1d71e?, 0xe0?, 0x1?, 0x1deaa?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00053bde0 sp=0xc00053bdc0 pc=0x4a6f6
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:387
runtime.semacquire1(0xc0003c24d0, 0x18?, 0x1, 0x0, 0xb1?)
        /usr/local/go/src/runtime/sema.go:160 +0x21c fp=0xc00053be48 sp=0xc00053bde0 pc=0x59ddc
sync.runtime_Semacquire(0xc0005ae108?)
        /usr/local/go/src/runtime/sema.go:62 +0x27 fp=0xc00053be80 sp=0xc00053be48 pc=0x72a87
sync.(*WaitGroup).Wait(0xe59d60?)
        /usr/local/go/src/sync/waitgroup.go:116 +0x4b fp=0xc00053bea8 sp=0xc00053be80 pc=0x862cb
golang.org/x/sync/errgroup.(*Group).Wait(0xc0003c24c0)
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:56 +0x27 fp=0xc00053bec8 sp=0xc00053bea8 pc=0x3c31e7
github.com/restic/restic/internal/archiver.(*Archiver).Snapshot.func1()
        /restic/internal/archiver/archiver.go:849 +0x1a9 fp=0xc00053bf78 sp=0xc00053bec8 pc=0x47c7a9
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc00053bfe0 sp=0xc00053bf78 pc=0x3c3384
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00053bfe8 sp=0xc00053bfe0 pc=0x77a81
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 95 [runnable, locked to thread]:
runtime.cgocall(0x79420, 0xc000068ac0)
        /usr/local/go/src/runtime/cgocall.go:157 +0x4a fp=0xc000577d90 sp=0xc000577d58 pc=0x14d4a
syscall.SyscallN(0x1deaa?, {0xc000577e28?, 0x0?, 0xc005b1a720?})
        /usr/local/go/src/runtime/syscall_windows.go:557 +0x109 fp=0xc000577e08 sp=0xc000577d90 pc=0x743c9
syscall.Syscall9(0xc000577f10?, 0x8b058?, 0xc005b36c00?, 0x7d?, 0x38843600?, 0xffffffffffffffff?, 0x18de1e0?, 0x3884361c?, 0xed6157509?, 0xc000577ff0, ...)
        /usr/local/go/src/runtime/syscall_windows.go:507 +0x78 fp=0xc000577e80 sp=0xc000577e08 pc=0x740d8
golang.org/x/sys/windows._getNamedSecurityInfo(0xc005b36c00?, 0x1, 0xf001007f, 0x0?, 0xc000578070?, 0x12f415?, 0xc005b36c00?, 0x7d?)
        /home/build/go/pkg/mod/golang.org/x/sys@v0.22.0/windows/zsyscall_windows.go:808 +0xcd fp=0xc000577f20 sp=0xc000577e80 pc=0x3c6acd
golang.org/x/sys/windows.getNamedSecurityInfo({0xc005b36c00?, 0x0?}, 0x0?, 0x0?, 0x7b?, 0x1841086420fa66c?, 0xa3281bb240ca2816?, 0x398f5a1eb1a6b9d7?, 0x502499702604eb31?)
        /home/build/go/pkg/mod/golang.org/x/sys@v0.22.0/windows/zsyscall_windows.go:804 +0x92 fp=0xc000577f98 sp=0xc000577f20 pc=0x3c6972
golang.org/x/sys/windows.GetNamedSecurityInfo({0xc005b36c00?, 0x7b?}, 0x0?, 0x0?)
        /home/build/go/pkg/mod/golang.org/x/sys@v0.22.0/windows/security_windows.go:1405 +0x6c fp=0xc000578028 sp=0xc000577f98 pc=0x3c644c
github.com/restic/restic/internal/fs.getNamedSecurityInfoHigh(...)
        /restic/internal/fs/sd_windows.go:132
github.com/restic/restic/internal/fs.GetSecurityDescriptor({0xc005b36c00, 0x7d})
        /restic/internal/fs/sd_windows.go:53 +0x78 fp=0xc000578080 sp=0xc000578028 pc=0x3f3ab8
github.com/restic/restic/internal/restic.(*Node).fillGenericAttributes(0xc005950dc0, {0xc005b36c00, 0x7d}, {0x10d8ab8, 0xc00580b5e0}, 0xc005921dd0)
        /restic/internal/restic/node_windows.go:367 +0xdb fp=0xc0005780b8 sp=0xc000578080 pc=0x466cdb
github.com/restic/restic/internal/restic.(*Node).fillExtra(0xc005950dc0, {0xc005b36c00, 0x7d}, {0x10d8ab8, 0xc00580b5e0}, 0x54?)
        /restic/internal/restic/node.go:706 +0x2b6 fp=0xc000578168 sp=0xc0005780b8 pc=0x463316
github.com/restic/restic/internal/restic.NodeFromFileInfo({0xc005b36c00, 0x7d}, {0x10d8ab8, 0xc00580b5e0}, 0x0?)
        /restic/internal/restic/node.go:153 +0x1e5 fp=0xc0005781e0 sp=0xc000578168 pc=0x460485
github.com/restic/restic/internal/archiver.(*Archiver).nodeFromFileInfo(0xc00007e000, {0xc005b36d00, 0x7d}, {0xc005b36c00, 0x7d}, {0x10d8ab8?, 0xc00580b5e0?}, 0x0?)
        /restic/internal/archiver/archiver.go:251 +0x65 fp=0xc000578260 sp=0xc0005781e0 pc=0x4773e5
github.com/restic/restic/internal/archiver.(*Archiver).save(0xc00007e000, {0x10d5f88, 0xc00042c230}, {0xc005b36d00, 0x7d}, {0xc005b36c00, 0x7d}, 0xc005b3a6e0)
        /restic/internal/archiver/archiver.go:458 +0x517 fp=0xc0005784f8 sp=0xc000578260 pc=0x478ab7
github.com/restic/restic/internal/archiver.(*Archiver).saveDir(0xc00007e000, {0x10d5f88, 0xc00042c230}, {0xc005b30120, 0x2b}, {0xc005b1c9c0, 0x2b}, {0x10d8ab8, 0xc005b18b60}, 0xc005aab308, ...)
        /restic/internal/archiver/archiver.go:325 +0x3ff fp=0xc000578620 sp=0xc0005784f8 pc=0x477f3f
github.com/restic/restic/internal/archiver.(*Archiver).save(0xc00007e000, {0x10d5f88, 0xc00042c230}, {0xc005b30120, 0x2b}, {0xc005b1c9c0, 0x2b}, 0xc005a9f1e0)
        /restic/internal/archiver/archiver.go:538 +0x1057 fp=0xc0005788b8 sp=0xc000578620 pc=0x4795f7
github.com/restic/restic/internal/archiver.(*Archiver).saveDir(0xc00007e000, {0x10d5f88, 0xc00042c230}, {0xc005a30940, 0x20}, {0xc005a30900, 0x20}, {0x10d8ab8, 0xc005a7a380}, 0xc005a46930, ...)
        /restic/internal/archiver/archiver.go:325 +0x3ff fp=0xc0005789e0 sp=0xc0005788b8 pc=0x477f3f
github.com/restic/restic/internal/archiver.(*Archiver).save(0xc00007e000, {0x10d5f88, 0xc00042c230}, {0xc005a30940, 0x20}, {0xc005a30900, 0x20}, 0xc00528adc0)
        /restic/internal/archiver/archiver.go:538 +0x1057 fp=0xc000578c78 sp=0xc0005789e0 pc=0x4795f7
github.com/restic/restic/internal/archiver.(*Archiver).saveDir(0xc00007e000, {0x10d5f88, 0xc00042c230}, {0xc0050c99a0, 0x1b}, {0xc0051e26c0, 0x1b}, {0x10d8ab8, 0xc0051e4700}, 0xc00518f8f0, ...)
        /restic/internal/archiver/archiver.go:325 +0x3ff fp=0xc000578da0 sp=0xc000578c78 pc=0x477f3f
github.com/restic/restic/internal/archiver.(*Archiver).save(0xc00007e000, {0x10d5f88, 0xc00042c230}, {0xc0050c99a0, 0x1b}, {0xc0051e26c0, 0x1b}, 0xc0031cac60)
        /restic/internal/archiver/archiver.go:538 +0x1057 fp=0xc000579038 sp=0xc000578da0 pc=0x4795f7
github.com/restic/restic/internal/archiver.(*Archiver).saveDir(0xc00007e000, {0x10d5f88, 0xc00042c230}, {0xc0005e7410, 0x12}, {0xc003abbee0, 0x12}, {0x10d8ab8, 0xc0031aa150}, 0xc00315c468, ...)
        /restic/internal/archiver/archiver.go:325 +0x3ff fp=0xc000579160 sp=0xc000579038 pc=0x477f3f
github.com/restic/restic/internal/archiver.(*Archiver).save(0xc00007e000, {0x10d5f88, 0xc00042c230}, {0xc0005e7410, 0x12}, {0xc003abbee0, 0x12}, 0xc0003bac60)
        /restic/internal/archiver/archiver.go:538 +0x1057 fp=0xc0005793f8 sp=0xc000579160 pc=0x4795f7
github.com/restic/restic/internal/archiver.(*Archiver).saveDir(0xc00007e000, {0x10d5f88, 0xc00042c230}, {0xc000106510, 0xc}, {0xc0001064f0, 0xc}, {0x10d8ab8, 0xc0000ce230}, 0xc0001b4240, ...)
        /restic/internal/archiver/archiver.go:325 +0x3ff fp=0xc000579520 sp=0xc0005793f8 pc=0x477f3f
github.com/restic/restic/internal/archiver.(*Archiver).save(0xc00007e000, {0x10d5f88, 0xc00042c230}, {0xc000106510, 0xc}, {0xc0001064f0, 0xc}, 0xc000224160)
        /restic/internal/archiver/archiver.go:538 +0x1057 fp=0xc0005797b8 sp=0xc000579520 pc=0x4795f7
github.com/restic/restic/internal/archiver.(*Archiver).saveDir(0xc00007e000, {0x10d5f88, 0xc00042c230}, {0xc0003b2d26, 0x2}, {0xc0003b2070, 0x3}, {0x10d8ab8, 0xc0000ce070}, 0xc0001b4078, ...)
        /restic/internal/archiver/archiver.go:325 +0x3ff fp=0xc0005798e0 sp=0xc0005797b8 pc=0x477f3f
github.com/restic/restic/internal/archiver.(*Archiver).save(0xc00007e000, {0x10d5f88, 0xc00042c230}, {0xc0003b2d26, 0x2}, {0xc0003b2070, 0x3}, 0xc0005a0160)
        /restic/internal/archiver/archiver.go:538 +0x1057 fp=0xc000579b78 sp=0xc0005798e0 pc=0x4795f7
github.com/restic/restic/internal/archiver.(*Archiver).saveTree(0xc00007e000, {0x10d5f88, 0xc00042c230}, {0x10c0898, 0x1}, 0xc0003c2240, 0xc000588078, 0xc00058c1b0?)
        /restic/internal/archiver/archiver.go:663 +0x634 fp=0xc000579e18 sp=0xc000579b78 pc=0x47a914
github.com/restic/restic/internal/archiver.(*Archiver).Snapshot.func1.1()
        /restic/internal/archiver/archiver.go:824 +0x1a5 fp=0xc000579f78 sp=0xc000579e18 pc=0x47ca85
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc000579fe0 sp=0xc000579f78 pc=0x3c3384
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000579fe8 sp=0xc000579fe0 pc=0x77a81
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 96 [select]:
runtime.gopark(0xc000299ea8?, 0x2?, 0xe0?, 0xa7?, 0xc000299e44?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000299c88 sp=0xc000299c68 pc=0x4a6f6
runtime.selectgo(0xc000299ea8, 0xc000299e40, 0x946974599a00e480?, 0x0, 0x353?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000299dc8 sp=0xc000299c88 pc=0x58cde
github.com/restic/restic/internal/archiver.(*BlobSaver).worker(0x45b4aba1bf28f700?, {0x10d5f88, 0xc00042c230}, 0xc00049a2a0)
        /restic/internal/archiver/blob_saver.go:88 +0xf5 fp=0xc000299f48 sp=0xc000299dc8 pc=0x47d3f5
github.com/restic/restic/internal/archiver.NewBlobSaver.func1()
        /restic/internal/archiver/blob_saver.go:34 +0x29 fp=0xc000299f78 sp=0xc000299f48 pc=0x47cfa9
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc000299fe0 sp=0xc000299f78 pc=0x3c3384
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000299fe8 sp=0xc000299fe0 pc=0x77a81
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 97 [select]:
runtime.gopark(0xc003699ea8?, 0x2?, 0xe0?, 0xa7?, 0xc003699e44?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc003699c88 sp=0xc003699c68 pc=0x4a6f6
runtime.selectgo(0xc003699ea8, 0xc003699e40, 0xaf537e044a417fe3?, 0x0, 0x2f7d?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc003699dc8 sp=0xc003699c88 pc=0x58cde
github.com/restic/restic/internal/archiver.(*BlobSaver).worker(0xe55fae2e9c3e0cd2?, {0x10d5f88, 0xc00042c230}, 0xc00049a2a0)
        /restic/internal/archiver/blob_saver.go:88 +0xf5 fp=0xc003699f48 sp=0xc003699dc8 pc=0x47d3f5
github.com/restic/restic/internal/archiver.NewBlobSaver.func1()
        /restic/internal/archiver/blob_saver.go:34 +0x29 fp=0xc003699f78 sp=0xc003699f48 pc=0x47cfa9
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc003699fe0 sp=0xc003699f78 pc=0x3c3384
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc003699fe8 sp=0xc003699fe0 pc=0x77a81
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 114 [select]:
runtime.gopark(0xc0036cbea8?, 0x2?, 0xe0?, 0xa7?, 0xc0036cbe44?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0036cbc88 sp=0xc0036cbc68 pc=0x4a6f6
runtime.selectgo(0xc0036cbea8, 0xc0036cbe40, 0xd1bc9c6db5419ce2?, 0x0, 0x344c?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0036cbdc8 sp=0xc0036cbc88 pc=0x58cde
github.com/restic/restic/internal/archiver.(*BlobSaver).worker(0x75318577395fa8eb?, {0x10d5f88, 0xc00042c230}, 0xc00049a2a0)
        /restic/internal/archiver/blob_saver.go:88 +0xf5 fp=0xc0036cbf48 sp=0xc0036cbdc8 pc=0x47d3f5
github.com/restic/restic/internal/archiver.NewBlobSaver.func1()
        /restic/internal/archiver/blob_saver.go:34 +0x29 fp=0xc0036cbf78 sp=0xc0036cbf48 pc=0x47cfa9
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc0036cbfe0 sp=0xc0036cbf78 pc=0x3c3384
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0036cbfe8 sp=0xc0036cbfe0 pc=0x77a81
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 115 [select]:
runtime.gopark(0xc0036cfea8?, 0x2?, 0xe0?, 0xa7?, 0xc0036cfe44?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0036cfc88 sp=0xc0036cfc68 pc=0x4a6f6
runtime.selectgo(0xc0036cfea8, 0xc0036cfe40, 0x152c95f26ea1751?, 0x0, 0x2c86?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0036cfdc8 sp=0xc0036cfc88 pc=0x58cde
github.com/restic/restic/internal/archiver.(*BlobSaver).worker(0xc3c585fcb9825cf7?, {0x10d5f88, 0xc00042c230}, 0xc00049a2a0)
        /restic/internal/archiver/blob_saver.go:88 +0xf5 fp=0xc0036cff48 sp=0xc0036cfdc8 pc=0x47d3f5
github.com/restic/restic/internal/archiver.NewBlobSaver.func1()
        /restic/internal/archiver/blob_saver.go:34 +0x29 fp=0xc0036cff78 sp=0xc0036cff48 pc=0x47cfa9
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc0036cffe0 sp=0xc0036cff78 pc=0x3c3384
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0036cffe8 sp=0xc0036cffe0 pc=0x77a81
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 116 [select]:
runtime.gopark(0xc00078fdf0?, 0x2?, 0x70?, 0xe3?, 0xc00078fddc?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00078fc20 sp=0xc00078fc00 pc=0x4a6f6
runtime.selectgo(0xc00078fdf0, 0xc00078fdd8, 0x10d8ab8?, 0x0, 0xc00078fed0?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc00078fd60 sp=0xc00078fc20 pc=0x58cde
github.com/restic/restic/internal/archiver.(*FileSaver).worker(0xc0004882d0, {0x10d5f88, 0xc00042c230}, 0xc00049a300)
        /restic/internal/archiver/file_saver.go:256 +0x1ce fp=0xc00078ff48 sp=0xc00078fd60 pc=0x47f14e
github.com/restic/restic/internal/archiver.NewFileSaver.func2()
        /restic/internal/archiver/file_saver.go:55 +0x29 fp=0xc00078ff78 sp=0xc00078ff48 pc=0x47d9a9
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc00078ffe0 sp=0xc00078ff78 pc=0x3c3384
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00078ffe8 sp=0xc00078ffe0 pc=0x77a81
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 117 [select]:
runtime.gopark(0xc003695df0?, 0x2?, 0x70?, 0xe3?, 0xc003695ddc?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc003695c20 sp=0xc003695c00 pc=0x4a6f6
runtime.selectgo(0xc003695df0, 0xc003695dd8, 0x10d8ab8?, 0x0, 0xc003695ed0?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc003695d60 sp=0xc003695c20 pc=0x58cde
github.com/restic/restic/internal/archiver.(*FileSaver).worker(0xc0004882d0, {0x10d5f88, 0xc00042c230}, 0xc00049a300)
        /restic/internal/archiver/file_saver.go:256 +0x1ce fp=0xc003695f48 sp=0xc003695d60 pc=0x47f14e
github.com/restic/restic/internal/archiver.NewFileSaver.func2()
        /restic/internal/archiver/file_saver.go:55 +0x29 fp=0xc003695f78 sp=0xc003695f48 pc=0x47d9a9
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc003695fe0 sp=0xc003695f78 pc=0x3c3384
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc003695fe8 sp=0xc003695fe0 pc=0x77a81
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 118 [select]:
runtime.gopark(0xc0006a7e10?, 0x2?, 0x0?, 0x0?, 0xc0006a7d84?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0006a7c00 sp=0xc0006a7be0 pc=0x4a6f6
runtime.selectgo(0xc0006a7e10, 0xc0006a7d80, 0x0?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0006a7d40 sp=0xc0006a7c00 pc=0x58cde
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xdca97111b88294c5?, {0x10d5f88, 0xc00042c230}, 0xc00049a360)
        /restic/internal/archiver/tree_saver.go:156 +0x1b9 fp=0xc0006a7f48 sp=0xc0006a7d40 pc=0x483ab9
github.com/restic/restic/internal/archiver.NewTreeSaver.func1()
        /restic/internal/archiver/tree_saver.go:33 +0x29 fp=0xc0006a7f78 sp=0xc0006a7f48 pc=0x4829c9
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc0006a7fe0 sp=0xc0006a7f78 pc=0x3c3384
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0006a7fe8 sp=0xc0006a7fe0 pc=0x77a81
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 119 [select]:
runtime.gopark(0xc00093be10?, 0x2?, 0x0?, 0x0?, 0xc00093bd84?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00093bc00 sp=0xc00093bbe0 pc=0x4a6f6
runtime.selectgo(0xc00093be10, 0xc00093bd80, 0x0?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc00093bd40 sp=0xc00093bc00 pc=0x58cde
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0x332eee52ae8d820e?, {0x10d5f88, 0xc00042c230}, 0xc00049a360)
        /restic/internal/archiver/tree_saver.go:156 +0x1b9 fp=0xc00093bf48 sp=0xc00093bd40 pc=0x483ab9
github.com/restic/restic/internal/archiver.NewTreeSaver.func1()
        /restic/internal/archiver/tree_saver.go:33 +0x29 fp=0xc00093bf78 sp=0xc00093bf48 pc=0x4829c9
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc00093bfe0 sp=0xc00093bf78 pc=0x3c3384
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00093bfe8 sp=0xc00093bfe0 pc=0x77a81
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 120 [select]:
runtime.gopark(0xc000793e10?, 0x2?, 0x0?, 0x0?, 0xc000793d84?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000793c00 sp=0xc000793be0 pc=0x4a6f6
runtime.selectgo(0xc000793e10, 0xc000793d80, 0x0?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000793d40 sp=0xc000793c00 pc=0x58cde
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0x9161ea416ce978fb?, {0x10d5f88, 0xc00042c230}, 0xc00049a360)
        /restic/internal/archiver/tree_saver.go:156 +0x1b9 fp=0xc000793f48 sp=0xc000793d40 pc=0x483ab9
github.com/restic/restic/internal/archiver.NewTreeSaver.func1()
        /restic/internal/archiver/tree_saver.go:33 +0x29 fp=0xc000793f78 sp=0xc000793f48 pc=0x4829c9
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc000793fe0 sp=0xc000793f78 pc=0x3c3384
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000793fe8 sp=0xc000793fe0 pc=0x77a81
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 121 [select]:
runtime.gopark(0xc00069de10?, 0x2?, 0x0?, 0x0?, 0xc00069dd84?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00069dc00 sp=0xc00069dbe0 pc=0x4a6f6
runtime.selectgo(0xc00069de10, 0xc00069dd80, 0x0?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc00069dd40 sp=0xc00069dc00 pc=0x58cde
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0x147486d65282aa80?, {0x10d5f88, 0xc00042c230}, 0xc00049a360)
        /restic/internal/archiver/tree_saver.go:156 +0x1b9 fp=0xc00069df48 sp=0xc00069dd40 pc=0x483ab9
github.com/restic/restic/internal/archiver.NewTreeSaver.func1()
        /restic/internal/archiver/tree_saver.go:33 +0x29 fp=0xc00069df78 sp=0xc00069df48 pc=0x4829c9
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc00069dfe0 sp=0xc00069df78 pc=0x3c3384
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00069dfe8 sp=0xc00069dfe0 pc=0x77a81
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 122 [select]:
runtime.gopark(0xc0006abe10?, 0x2?, 0x0?, 0x0?, 0xc0006abd84?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0006abc00 sp=0xc0006abbe0 pc=0x4a6f6
runtime.selectgo(0xc0006abe10, 0xc0006abd80, 0x0?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0006abd40 sp=0xc0006abc00 pc=0x58cde
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0x6e57f8d81178f284?, {0x10d5f88, 0xc00042c230}, 0xc00049a360)
        /restic/internal/archiver/tree_saver.go:156 +0x1b9 fp=0xc0006abf48 sp=0xc0006abd40 pc=0x483ab9
github.com/restic/restic/internal/archiver.NewTreeSaver.func1()
        /restic/internal/archiver/tree_saver.go:33 +0x29 fp=0xc0006abf78 sp=0xc0006abf48 pc=0x4829c9
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc0006abfe0 sp=0xc0006abf78 pc=0x3c3384
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0006abfe8 sp=0xc0006abfe0 pc=0x77a81
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

goroutine 123 [select]:
runtime.gopark(0xc000699e10?, 0x2?, 0x0?, 0x0?, 0xc000699d84?)
        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000699c00 sp=0xc000699be0 pc=0x4a6f6
runtime.selectgo(0xc000699e10, 0xc000699d80, 0x0?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000699d40 sp=0xc000699c00 pc=0x58cde
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xdd8c9d7e4003676c?, {0x10d5f88, 0xc00042c230}, 0xc00049a360)
        /restic/internal/archiver/tree_saver.go:156 +0x1b9 fp=0xc000699f48 sp=0xc000699d40 pc=0x483ab9
github.com/restic/restic/internal/archiver.NewTreeSaver.func1()
        /restic/internal/archiver/tree_saver.go:33 +0x29 fp=0xc000699f78 sp=0xc000699f48 pc=0x4829c9
golang.org/x/sync/errgroup.(*Group).Go.func1()
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78 +0x64 fp=0xc000699fe0 sp=0xc000699f78 pc=0x3c3384
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000699fe8 sp=0xc000699fe0 pc=0x77a81
created by golang.org/x/sync/errgroup.(*Group).Go
        /home/build/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:75 +0xa5

I'm not sure where to go next since I'm not a Go developer. I was thinking of trying to compile on Windows instead of Linux docker container, just as an experiment, but I'm not sure whether the restic folks have a good recipe for me, given cross-compiling in docker seems to be the preferred methodology for obvious reasons. I might also try to spin a Server 2022 Evaluation VM to see if I encounter the same behavior. I think Server 2022 supports DTrace, which could be helpful to the right person.

[RichieSams]

Do I understand correctly that your crashes are occurring on bare-metal AMD machines?

@prattmic Correct. They're bare metal servers running Windows 10. Below is the dxdiag report of one of the machines:

      Time of this report: 4/23/2024, 09:36:37
             Machine name: XXXXXXXX
               Machine Id: {F429D65A-427C-4F90-A587-8AB68B33FB57}
         Operating System: Windows 10 Enterprise 64-bit (10.0, Build 19045) (19041.vb_release.191206-1406)
                 Language: English (Regional Setting: English)
      System Manufacturer: Equus Computer Systems
             System Model: Nobilis
                     BIOS: F5c (type: UEFI)
                Processor: AMD Ryzen Threadripper 3990X 64-Core Processor  (64 CPUs), ~2.9GHz
                   Memory: 262144MB RAM
      Available OS Memory: 262032MB RAM
                Page File: 38200MB used, 668832MB available
              Windows Dir: C:\Windows
          DirectX Version: DirectX 12
      DX Setup Parameters: Not found
         User DPI Setting: 96 DPI (100 percent)
       System DPI Setting: 96 DPI (100 percent)
          DWM DPI Scaling: Disabled
                 Miracast: Available, no HDCP
Microsoft Graphics Hybrid: Not Supported
 DirectX Database Version: 1.0.8
           DxDiag Version: 10.00.19041.3636 64bit Unicode

Google says 3990X is Zen 2

[RichieSams]

Yesterday, I also got a crash on a user's desktop. (I'll ask them for their CPU model number and post it here when they respond). Stack trace:

unexpected fault address 0xbebc1fc2
fatal error: fault
[signal 0xc0000005 code=0x0 addr=0xbebc1fc2 pc=0x93ed74]

goroutine 1 gp=0xc000106000 m=0 mp=0x2ab3880 [running]:
runtime.throw({0x1a9af4e?, 0xc000399e90?})
        runtime/panic.go:1023 +0x65 fp=0xc0006c5868 sp=0xc0006c5838 pc=0x89ebc5
runtime.sigpanic()
        runtime/signal_windows.go:414 +0xd0 fp=0xc0006c58b0 sp=0xc0006c5868 pc=0x8b4670
strconv.mult64bitPow10(0x6c5958?, 0x8721a5?, 0x87a07c?)
        strconv/ftoaryu.go:502 +0x34 fp=0xc0006c58d0 sp=0xc0006c58b0 pc=0x93ed74
strconv.ryuFtoaShortest(0xc0006c5a58, 0x800000?, 0xffffffffffffffe8?, 0x2774ff0?)
        strconv/ftoaryu.go:254 +0x165 fp=0xc0006c5968 sp=0xc0006c58d0 pc=0x93e485
strconv.genericFtoa({0xc0000475d8, 0x0, 0x6a362939eb6e0553?}, 0x2389aa90598?, 0x67, 0xffffffffffffffff, 0x1850c80?)
        strconv/ftoa.go:119 +0x505 fp=0xc0006c5a98 sp=0xc0006c5968 pc=0x93be85
strconv.FormatFloat(...)
        strconv/ftoa.go:48
github.com/spf13/pflag.(*float32Value).String(0xc0006c5b10?)
        github.com/spf13/pflag@v1.0.5/float32.go:23 +0x53 fp=0xc0006c5ae8 sp=0xc0006c5a98 pc=0xa3d173
github.com/spf13/pflag.(*FlagSet).VarPF(0xc0001ea800, {0x1d95500, 0xc000393c08}, {0x1aae91c, 0x13}, {0x0, 0x0}, {0x1ac6d7a, 0x20})
        github.com/spf13/pflag@v1.0.5/flag.go:829 +0x48 fp=0xc0006c5b20 sp=0xc0006c5ae8 pc=0xa3af28
github.com/spf13/pflag.(*FlagSet).VarP(...)
        github.com/spf13/pflag@v1.0.5/flag.go:837
github.com/spf13/pflag.(*FlagSet).Float32VarP(...)
        github.com/spf13/pflag@v1.0.5/float32.go:50
github.com/spf13/pflag.(*FlagSet).Float32(0xc0001ea800, {0x1aae91c, 0x13}, 0x3f000000, {0x1ac6d7a, 0x20})
        github.com/spf13/pflag@v1.0.5/float32.go:68 +0x8b fp=0xc0006c5b80 sp=0xc0006c5b20 pc=0xa3d4ab
my-project.com/cmd.CreateRootCommand(0x0, 0x2a9cdc0)
        my-project.com/cmd/root.go:411 +0x1a0a fp=0xc0006c5e58 sp=0xc0006c5b80 pc=0x16c072a
my-project.com/cmd.Execute({0x1d98348, 0xc0002de990}, 0x2a9cdc0)
        my-project.com/cmd/root.go:519 +0x2b fp=0xc0006c5ec0 sp=0xc0006c5e58 pc=0x16c5d8b
main.main.func1()
        my-project.com/main.go:38 +0xf2 fp=0xc0006c5f40 sp=0xc0006c5ec0 pc=0x16e7732
main.main()
        my-project.com/main.go:39 +0xf fp=0xc0006c5f50 sp=0xc0006c5f40 pc=0x16e762f
runtime.main()
        runtime/proc.go:271 +0x28b fp=0xc0006c5fe0 sp=0xc0006c5f50 pc=0x8a14eb
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc0006c5fe8 sp=0xc0006c5fe0 pc=0x8d2801

goroutine 2 gp=0xc000106a80 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc000109fa8 sp=0xc000109f88 pc=0x8a18ee
runtime.goparkunlock(...)
        runtime/proc.go:408
runtime.forcegchelper()
        runtime/proc.go:326 +0xb8 fp=0xc000109fe0 sp=0xc000109fa8 pc=0x8a1778
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc000109fe8 sp=0xc000109fe0 pc=0x8d2801
created by runtime.init.6 in goroutine 1
        runtime/proc.go:314 +0x1a

goroutine 3 gp=0xc000106c40 m=nil [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc00010bf80 sp=0xc00010bf60 pc=0x8a18ee
runtime.goparkunlock(...)
        runtime/proc.go:408
runtime.bgsweep(0xc0000480e0)
        runtime/mgcsweep.go:318 +0xdf fp=0xc00010bfc8 sp=0xc00010bf80 pc=0x88ab5f
runtime.gcenable.gowrap1()
        runtime/mgc.go:203 +0x25 fp=0xc00010bfe0 sp=0xc00010bfc8 pc=0x87f405
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc00010bfe8 sp=0xc00010bfe0 pc=0x8d2801
created by runtime.gcenable in goroutine 1
        runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000106e00 m=nil [GC scavenge wait]:
runtime.gopark(0x10000?, 0x1d7ee88?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc00011bf78 sp=0xc00011bf58 pc=0x8a18ee
runtime.goparkunlock(...)
        runtime/proc.go:408
runtime.(*scavengerState).park(0x2ab1c40)
        runtime/mgcscavenge.go:425 +0x49 fp=0xc00011bfa8 sp=0xc00011bf78 pc=0x8884e9
runtime.bgscavenge(0xc0000480e0)
        runtime/mgcscavenge.go:658 +0x59 fp=0xc00011bfc8 sp=0xc00011bfa8 pc=0x888a99
runtime.gcenable.gowrap2()
        runtime/mgc.go:204 +0x25 fp=0xc00011bfe0 sp=0xc00011bfc8 pc=0x87f3a5
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc00011bfe8 sp=0xc00011bfe0 pc=0x8d2801
created by runtime.gcenable in goroutine 1
        runtime/mgc.go:204 +0xa5

goroutine 18 gp=0xc0002041c0 m=nil [finalizer wait]:
runtime.gopark(0xc00010de60?, 0x8879bc?, 0xc0?, 0x75?, 0x550011?)
        runtime/proc.go:402 +0xce fp=0xc00010de20 sp=0xc00010de00 pc=0x8a18ee
runtime.runfinq()
        runtime/mfinal.go:194 +0x107 fp=0xc00010dfe0 sp=0xc00010de20 pc=0x87e487
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc00010dfe8 sp=0xc00010dfe0 pc=0x8d2801
created by runtime.createfing in goroutine 1
        runtime/mfinal.go:164 +0x3d

goroutine 19 gp=0xc000204700 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc000117f50 sp=0xc000117f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc000117fe0 sp=0xc000117f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc000117fe8 sp=0xc000117fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 34 gp=0xc0001c9c00 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc0002e5f50 sp=0xc0002e5f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc0002e5fe0 sp=0xc0002e5f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc0002e5fe8 sp=0xc0002e5fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 35 gp=0xc0001c9dc0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc0002e7f50 sp=0xc0002e7f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc0002e7fe0 sp=0xc0002e7f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc0002e7fe8 sp=0xc0002e7fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 36 gp=0xc0002e8000 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc0002e1f50 sp=0xc0002e1f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc0002e1fe0 sp=0xc0002e1f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc0002e1fe8 sp=0xc0002e1fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 37 gp=0xc0002e81c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc0002e3f50 sp=0xc0002e3f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc0002e3fe0 sp=0xc0002e3f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc0002e3fe8 sp=0xc0002e3fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 20 gp=0xc0002048c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc000119f50 sp=0xc000119f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc000119fe0 sp=0xc000119f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc000119fe8 sp=0xc000119fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 38 gp=0xc0002e8380 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc0002eff50 sp=0xc0002eff30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc0002effe0 sp=0xc0002eff50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc0002effe8 sp=0xc0002effe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 5 gp=0xc000328000 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc00011df50 sp=0xc00011df30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc00011dfe0 sp=0xc00011df50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc00011dfe8 sp=0xc00011dfe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 6 gp=0xc0003281c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc0002ebf50 sp=0xc0002ebf30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc0002ebfe0 sp=0xc0002ebf50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc0002ebfe8 sp=0xc0002ebfe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 21 gp=0xc000204a80 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc000485f50 sp=0xc000485f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc000485fe0 sp=0xc000485f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc000485fe8 sp=0xc000485fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 22 gp=0xc000204c40 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc000487f50 sp=0xc000487f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc000487fe0 sp=0xc000487f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc000487fe8 sp=0xc000487fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 7 gp=0xc000328380 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc0002edf50 sp=0xc0002edf30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc0002edfe0 sp=0xc0002edf50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc0002edfe8 sp=0xc0002edfe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 8 gp=0xc000328540 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc000481f50 sp=0xc000481f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc000481fe0 sp=0xc000481f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc000481fe8 sp=0xc000481fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 9 gp=0xc000328700 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc000483f50 sp=0xc000483f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc000483fe0 sp=0xc000483f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc000483fe8 sp=0xc000483fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 10 gp=0xc0003288c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc00032ff50 sp=0xc00032ff30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc00032ffe0 sp=0xc00032ff50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc00032ffe8 sp=0xc00032ffe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 39 gp=0xc0002e8540 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc0002f1f50 sp=0xc0002f1f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc0002f1fe0 sp=0xc0002f1f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc0002f1fe8 sp=0xc0002f1fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 23 gp=0xc000204e00 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc00032bf50 sp=0xc00032bf30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc00032bfe0 sp=0xc00032bf50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc00032bfe8 sp=0xc00032bfe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 11 gp=0xc000328a80 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc000331f50 sp=0xc000331f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc000331fe0 sp=0xc000331f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc000331fe8 sp=0xc000331fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 40 gp=0xc0002e8700 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc0002f7f50 sp=0xc0002f7f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc0002f7fe0 sp=0xc0002f7f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc0002f7fe8 sp=0xc0002f7fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 24 gp=0xc000204fc0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc00032df50 sp=0xc00032df30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc00032dfe0 sp=0xc00032df50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc00032dfe8 sp=0xc00032dfe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 41 gp=0xc0002e88c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc0002f9f50 sp=0xc0002f9f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc0002f9fe0 sp=0xc0002f9f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc0002f9fe8 sp=0xc0002f9fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 42 gp=0xc0002e8a80 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc0002f3f50 sp=0xc0002f3f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc0002f3fe0 sp=0xc0002f3f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc0002f3fe8 sp=0xc0002f3fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 25 gp=0xc000205180 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc00048df50 sp=0xc00048df30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc00048dfe0 sp=0xc00048df50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc00048dfe8 sp=0xc00048dfe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 26 gp=0xc000205340 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc00048ff50 sp=0xc00048ff30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc00048ffe0 sp=0xc00048ff50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc00048ffe8 sp=0xc00048ffe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 12 gp=0xc000328c40 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc000489f50 sp=0xc000489f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc000489fe0 sp=0xc000489f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc000489fe8 sp=0xc000489fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 27 gp=0xc000205500 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc000495f50 sp=0xc000495f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc000495fe0 sp=0xc000495f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc000495fe8 sp=0xc000495fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 43 gp=0xc0002e8c40 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc0002f5f50 sp=0xc0002f5f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc0002f5fe0 sp=0xc0002f5f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc0002f5fe8 sp=0xc0002f5fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 28 gp=0xc0002056c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc000497f50 sp=0xc000497f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc000497fe0 sp=0xc000497f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc000497fe8 sp=0xc000497fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 44 gp=0xc0002e8e00 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc000491f50 sp=0xc000491f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc000491fe0 sp=0xc000491f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc000491fe8 sp=0xc000491fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 29 gp=0xc000205880 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc00049df50 sp=0xc00049df30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc00049dfe0 sp=0xc00049df50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc00049dfe8 sp=0xc00049dfe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 30 gp=0xc000205a40 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc00049ff50 sp=0xc00049ff30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc00049ffe0 sp=0xc00049ff50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc00049ffe8 sp=0xc00049ffe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 45 gp=0xc0002e8fc0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc000493f50 sp=0xc000493f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc000493fe0 sp=0xc000493f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc000493fe8 sp=0xc000493fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 46 gp=0xc0002e9180 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc000499f50 sp=0xc000499f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc000499fe0 sp=0xc000499f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc000499fe8 sp=0xc000499fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 47 gp=0xc0002e9340 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc00049bf50 sp=0xc00049bf30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc00049bfe0 sp=0xc00049bf50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc00049bfe8 sp=0xc00049bfe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 13 gp=0xc000328e00 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc00048bf50 sp=0xc00048bf30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc00048bfe0 sp=0xc00048bf50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc00048bfe8 sp=0xc00048bfe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 14 gp=0xc000328fc0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc000337f50 sp=0xc000337f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc000337fe0 sp=0xc000337f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc000337fe8 sp=0xc000337fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 15 gp=0xc000329180 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc000339f50 sp=0xc000339f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc000339fe0 sp=0xc000339f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc000339fe8 sp=0xc000339fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 31 gp=0xc000205c00 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc000333f50 sp=0xc000333f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc000333fe0 sp=0xc000333f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc000333fe8 sp=0xc000333fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 32 gp=0xc000205dc0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc000335f50 sp=0xc000335f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc000335fe0 sp=0xc000335f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc000335fe8 sp=0xc000335fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 33 gp=0xc0004a0000 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc0004a7f50 sp=0xc0004a7f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc0004a7fe0 sp=0xc0004a7f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc0004a7fe8 sp=0xc0004a7fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 48 gp=0xc0002e9500 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc0004a3f50 sp=0xc0004a3f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc0004a3fe0 sp=0xc0004a3f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc0004a3fe8 sp=0xc0004a3fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 49 gp=0xc0002e96c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc0004a5f50 sp=0xc0004a5f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc0004a5fe0 sp=0xc0004a5f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc0004a5fe8 sp=0xc0004a5fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 50 gp=0xc0002e9880 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc000505f50 sp=0xc000505f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc000505fe0 sp=0xc000505f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc000505fe8 sp=0xc000505fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 51 gp=0xc0002e9a40 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc000507f50 sp=0xc000507f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc000507fe0 sp=0xc000507f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc000507fe8 sp=0xc000507fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 16 gp=0xc000329340 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc000501f50 sp=0xc000501f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc000501fe0 sp=0xc000501f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc000501fe8 sp=0xc000501fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 52 gp=0xc0002e9c00 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc00050df50 sp=0xc00050df30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc00050dfe0 sp=0xc00050df50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc00050dfe8 sp=0xc00050dfe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 53 gp=0xc0002e9dc0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc00050ff50 sp=0xc00050ff30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc00050ffe0 sp=0xc00050ff50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc00050ffe8 sp=0xc00050ffe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 66 gp=0xc000329500 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc000503f50 sp=0xc000503f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc000503fe0 sp=0xc000503f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc000503fe8 sp=0xc000503fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 82 gp=0xc0004a01c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc0004a9f50 sp=0xc0004a9f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc0004a9fe0 sp=0xc0004a9f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc0004a9fe8 sp=0xc0004a9fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 54 gp=0xc0002fa000 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc000509f50 sp=0xc000509f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc000509fe0 sp=0xc000509f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc000509fe8 sp=0xc000509fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 55 gp=0xc0002fa1c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc00050bf50 sp=0xc00050bf30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc00050bfe0 sp=0xc00050bf50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc00050bfe8 sp=0xc00050bfe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 67 gp=0xc0003296c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc00033ff50 sp=0xc00033ff30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc00033ffe0 sp=0xc00033ff50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc00033ffe8 sp=0xc00033ffe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 68 gp=0xc000329880 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc000341f50 sp=0xc000341f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc000341fe0 sp=0xc000341f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc000341fe8 sp=0xc000341fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 83 gp=0xc0004a0380 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc00033bf50 sp=0xc00033bf30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc00033bfe0 sp=0xc00033bf50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc00033bfe8 sp=0xc00033bfe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 69 gp=0xc000329a40 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc000347f50 sp=0xc000347f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc000347fe0 sp=0xc000347f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc000347fe8 sp=0xc000347fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 70 gp=0xc000329c00 m=nil [GC worker (idle)]:
runtime.gopark(0xfe480efc858c?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc000349f50 sp=0xc000349f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc000349fe0 sp=0xc000349f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc000349fe8 sp=0xc000349fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 71 gp=0xc000329dc0 m=nil [GC worker (idle)]:
runtime.gopark(0xfe480fbb8ea0?, 0x1?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc000343f50 sp=0xc000343f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc000343fe0 sp=0xc000343f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc000343fe8 sp=0xc000343fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 84 gp=0xc0004a0540 m=nil [GC worker (idle)]:
runtime.gopark(0xfe480fbb8ea0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc00033df50 sp=0xc00033df30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc00033dfe0 sp=0xc00033df50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc00033dfe8 sp=0xc00033dfe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 72 gp=0xc00034a000 m=nil [GC worker (idle)]:
runtime.gopark(0x2b475c0?, 0x1?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc000345f50 sp=0xc000345f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc000345fe0 sp=0xc000345f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc000345fe8 sp=0xc000345fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 85 gp=0xc0004a0700 m=nil [GC worker (idle)]:
runtime.gopark(0xfe480fbb8ea0?, 0x1?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc0004aff50 sp=0xc0004aff30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc0004affe0 sp=0xc0004aff50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc0004affe8 sp=0xc0004affe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 73 gp=0xc00034a1c0 m=nil [GC worker (idle)]:
runtime.gopark(0xfe480efc858c?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc0004abf50 sp=0xc0004abf30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc0004abfe0 sp=0xc0004abf50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc0004abfe8 sp=0xc0004abfe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 86 gp=0xc0004a08c0 m=nil [GC worker (idle)]:
runtime.gopark(0xfe480fbb8ea0?, 0x1?, 0x78?, 0xb?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc0004b1f50 sp=0xc0004b1f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc0004b1fe0 sp=0xc0004b1f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc0004b1fe8 sp=0xc0004b1fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 87 gp=0xc0004a0a80 m=nil [GC worker (idle)]:
runtime.gopark(0xfe480fbb8ea0?, 0x1?, 0x18?, 0x19?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc0004b7f50 sp=0xc0004b7f30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc0004b7fe0 sp=0xc0004b7f50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc0004b7fe8 sp=0xc0004b7fe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 74 gp=0xc00034a380 m=nil [GC worker (idle)]:
runtime.gopark(0xfe480efc858c?, 0x3?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc0004adf50 sp=0xc0004adf30 pc=0x8a18ee
runtime.gcBgMarkWorker()
        runtime/mgc.go:1310 +0xe5 fp=0xc0004adfe0 sp=0xc0004adf50 pc=0x881545
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc0004adfe8 sp=0xc0004adfe0 pc=0x8d2801
created by runtime.gcBgMarkStartWorkers in goroutine 1
        runtime/mgc.go:1234 +0x1c

goroutine 75 gp=0xc0004a1180 m=nil [select]:
runtime.gopark(0xc0004b5f78?, 0x3?, 0x28?, 0xa?, 0xc0004b5f72?)
        runtime/proc.go:402 +0xce fp=0xc0004b5e18 sp=0xc0004b5df8 pc=0x8a18ee
runtime.selectgo(0xc0004b5f78, 0xc0004b5f6c, 0xc00022c780?, 0x0, 0x0?, 0x1)
        runtime/select.go:327 +0x725 fp=0xc0004b5f38 sp=0xc0004b5e18 pc=0x8b1d45
go.opencensus.io/stats/view.(*worker).start(0xc00022c780)
        go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0x9f fp=0xc0004b5fc8 sp=0xc0004b5f38 pc=0xf51b1f
go.opencensus.io/stats/view.init.0.gowrap1()
        go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x25 fp=0xc0004b5fe0 sp=0xc0004b5fc8 pc=0xf50e85
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc0004b5fe8 sp=0xc0004b5fe0 pc=0x8d2801
created by go.opencensus.io/stats/view.init.0 in goroutine 1
        go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x8d

goroutine 88 gp=0xc00051c1c0 m=nil [chan receive]:
runtime.gopark(0x0?, 0xc00043ddc8?, 0x96?, 0x5?, 0x0?)
        runtime/proc.go:402 +0xce fp=0xc00043dd90 sp=0xc00043dd70 pc=0x8a18ee
runtime.chanrecv(0xc0002863c0, 0xc00043dee0, 0x1)
        runtime/chan.go:583 +0x3cd fp=0xc00043de08 sp=0xc00043dd90 pc=0x86a9ed
runtime.chanrecv2(0xc0001820c0?, 0x0?)
        runtime/chan.go:447 +0x12 fp=0xc00043de30 sp=0xc00043de08 pc=0x86a612
github.com/getsentry/sentry-go.(*HTTPTransport).worker(0xc0004e0af0)
        github.com/getsentry/sentry-go@v0.27.0/transport.go:471 +0x13d fp=0xc00043dfc8 sp=0xc00043de30 pc=0xccaf5d
github.com/getsentry/sentry-go.(*HTTPTransport).Configure.func1.gowrap1()
        github.com/getsentry/sentry-go@v0.27.0/transport.go:343 +0x25 fp=0xc00043dfe0 sp=0xc00043dfc8 pc=0xcca3a5
runtime.goexit({})
        runtime/asm_amd64.s:1695 +0x1 fp=0xc00043dfe8 sp=0xc00043dfe0 pc=0x8d2801
created by github.com/getsentry/sentry-go.(*HTTPTransport).Configure.func1 in goroutine 1
        github.com/getsentry/sentry-go@v0.27.0/transport.go:343 +0x56

[RichieSams]

User's desktop CPU: AMD Ryzen 3970X - 32 core

[prattmic]

Someone on https://smartos.topicbox.com/groups/smartos-discuss/Tf8f450fd133908fb mentioned that they would get similar crashes just running go version, not just their larger, more complex programs. Can either of you reproduce that?

---

@distancesprinter

I was thinking of trying to compile on Windows instead of Linux docker container, just as an experiment, but I'm not sure whether the restic folks have a good recipe for me, given cross-compiling in docker seems to be the preferred methodology for obvious reasons.

Go has quite strong cross-compilation support, so I would be surprised if the build host mattered, especially as restic isn't using cgo. That said, their build process is minorly non-standard .

Here's the final build configuration:

$ go version -m restic_0.17.0_windows_amd64.exe
restic_0.17.0_windows_amd64.exe: go1.22.5
        path    github.com/restic/restic/cmd/restic
        mod     github.com/restic/restic        (devel) 
        dep     cloud.google.com/go     v0.115.0        h1:CnFSK6Xo3lDYRoBKEcAtia6VSC837/ZkJuRduSFnr14=
        dep     cloud.google.com/go/auth        v0.6.1  h1:T0Zw1XM5c1GlpN2HYr2s+m3vr1p2wy+8VN+Z1FKxW38=
        dep     cloud.google.com/go/auth/oauth2adapt    v0.2.2  h1:+TTV8aXpjeChS9M+aTtN/TjdQnzJvmzKFt//oWu7HX4=
        dep     cloud.google.com/go/compute/metadata    v0.3.0  h1:Tz+eQXMEqDIKRsmY3cHTL6FVaynIjX2QxYC4trgAKZc=
        dep     cloud.google.com/go/iam v1.1.8  h1:r7umDwhj+BQyz0ScZMp4QrGXjSTI3ZINnpgU2nlB/K0=
        dep     cloud.google.com/go/storage     v1.43.0 h1:CcxnSohZwizt4LCzQHWvBf1/kvtHUn7gk9QERXPyXFs=
        dep     github.com/Azure/azure-sdk-for-go/sdk/azcore    v1.12.0 h1:1nGuui+4POelzDwI7RG56yfQJHCnKvwfMoU7VsEp+Zg=
        dep     github.com/Azure/azure-sdk-for-go/sdk/azidentity        v1.7.0  h1:tfLQ34V6F7tVSwoTf/4lH5sE0o6eCJuNDTmH09nDpbc=
        dep     github.com/Azure/azure-sdk-for-go/sdk/internal  v1.9.0  h1:H+U3Gk9zY56G3u872L82bk4thcsy2Gghb9ExT4Zvm1o=
        dep     github.com/Azure/azure-sdk-for-go/sdk/storage/azblob    v1.3.2  h1:YUUxeiOWgdAQE3pXt2H7QXzZs0q8UBjgRbl56qo8GYM=
        dep     github.com/AzureAD/microsoft-authentication-library-for-go      v1.2.2  h1:XHOnouVk1mxXfQidrMEnLlPk9UMeRtyBTnEFtxkV0kU=
        dep     github.com/Backblaze/blazer     v0.6.1  h1:xC9HyC7OcxRzzmtfRiikIEvq4HZYWjU6caFwX2EXw1s=
        dep     github.com/cenkalti/backoff/v4  v4.3.0  h1:MyRJ/UdXutAwSAT+s3wNd7MfTIcy71VQueUuFK343L8=
        dep     github.com/cespare/xxhash/v2    v2.3.0  h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
        dep     github.com/cpuguy83/go-md2man/v2        v2.0.4  h1:wfIWP927BUkWJb2NmU/kNDYIBTh/ziUX91+lVfRxZq4=
        dep     github.com/elithrar/simple-scrypt       v1.3.0  h1:KIlOlxdoQf9JWKl5lMAJ28SY2URB0XTRDn2TckyzAZg=
        dep     github.com/felixge/httpsnoop    v1.0.4  h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg=
        dep     github.com/go-logr/logr v1.4.1  h1:pKouT5E8xu9zeFC39JXRDukb6JFQPXM5p5I91188VAQ=
        dep     github.com/go-logr/stdr v1.2.2  h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
        dep     github.com/go-ole/go-ole        v1.3.0  h1:Dt6ye7+vXGIKZ7Xtk4s6/xVdGDQynvom7xCFEdWr6uE=
        dep     github.com/golang-jwt/jwt/v5    v5.2.1  h1:OuVbFODueb089Lh128TAcimifWaLhJwVflnrgM17wHk=
        dep     github.com/golang/groupcache    v0.0.0-20210331224755-41bb18bfe9da      h1:oI5xCqsCo564l8iNU+DwB5epxmsaqB+rhGL0m5jtYqE=
        dep     github.com/golang/protobuf      v1.5.4  h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
        dep     github.com/google/s2a-go        v0.1.7  h1:60BLSyTrOV4/haCDW4zb1guZItoSq8foHCXrAnjBo/o=
        dep     github.com/google/uuid  v1.6.0  h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
        dep     github.com/googleapis/enterprise-certificate-proxy      v0.3.2  h1:Vie5ybvEvT75RniqhfFxPRy3Bf7vr3h0cechB90XaQs=
        dep     github.com/googleapis/gax-go/v2 v2.12.5 h1:8gw9KZK8TiVKB6q3zHY3SBzLnrGp6HQjyfYBYGmXdxA=
        dep     github.com/hashicorp/golang-lru/v2      v2.0.7  h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
        dep     github.com/inconshreveable/mousetrap    v1.1.0  h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
        dep     github.com/json-iterator/go     v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
        dep     github.com/klauspost/compress   v1.17.9 h1:6KIumPrER1LHsvBVuDa0r5xaG0Es51mhhB9BQB2qeMA=
        dep     github.com/klauspost/cpuid/v2   v2.2.6  h1:ndNyv040zDGIDh8thGkXYjnFtiN02M1PVVF+JE/48xc=
        dep     github.com/kr/fs        v0.1.0  h1:Jskdu9ieNAYnjxsi0LbQp1ulIKZV1LAFgK1tWhpZgl8=
        dep     github.com/kylelemons/godebug   v1.1.0  h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
        dep     github.com/minio/md5-simd       v1.1.2  h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34=
        dep     github.com/minio/minio-go/v7    v7.0.66 h1:bnTOXOHjOqv/gcMuiVbN9o2ngRItvqE774dG9nq0Dzw=
        dep     github.com/minio/sha256-simd    v1.0.1  h1:6kaan5IFmwTNynnKKpDHe6FWHohJOHhCPchzK49dzMM=
        dep     github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd      h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
        dep     github.com/modern-go/reflect2   v1.0.2  h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M=
        dep     github.com/ncw/swift/v2 v2.0.2  h1:jx282pcAKFhmoZBSdMcCRFn9VWkoBIRsCpe+yZq7vEk=
        dep     github.com/peterbourgon/unixtransport   v0.0.4  h1:UTF0FxXCAglvoZz9jaGPYjEg52DjBLDYGMJvJni6Tfw=
        dep     github.com/pkg/browser  v0.0.0-20240102092130-5ac0b6a4141c      h1:+mdjkGKdHQG3305AYmdv1U2eRNDiU2ErMBj1gwrq8eQ=
        dep     github.com/pkg/errors   v0.9.1  h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
        dep     github.com/pkg/sftp     v1.13.6 h1:JFZT4XbOU7l77xGSpOdW+pwIMqP044IyjXX6FGyEKFo=
        dep     github.com/restic/chunker       v0.4.0  h1:YUPYCUn70MYP7VO4yllypp2SjmsRhRJaad3xKu1QFRw=
        dep     github.com/rs/xid       v1.5.0  h1:mKX4bl4iPYJtEIxp6CYiUuLQ/8DYMoz0PUdtGgMFRVc=
        dep     github.com/russross/blackfriday/v2      v2.1.0  h1:JIOH55/0cWyOuilr9/qlrm0BSXldqnqwMsf35Ld67mk=
        dep     github.com/spf13/cobra  v1.8.1  h1:e5/vxKd/rZsfSJMUX1agtjeTDf+qv1/JdBF8gg5k9ZM=
        dep     github.com/spf13/pflag  v1.0.5  h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
        dep     go.opencensus.io        v0.24.0 h1:y73uSU6J157QMP2kn2r30vwW1A2W2WFwSCGnAVxeaD0=
        dep     go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc     v0.49.0 h1:4Pp6oUg3+e/6M4C0A/3kJ2VYa++dsWVTtGgLVj5xtHg=
        dep     go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp   v0.49.0 h1:jq9TW8u3so/bN+JPT166wjOI6/vQPF6Xe7nMNIltagk=
        dep     go.opentelemetry.io/otel        v1.24.0 h1:0LAOdjNmQeSTzGBzduGe/rU4tZhMwL5rWgtp9Ku5Jfo=
        dep     go.opentelemetry.io/otel/metric v1.24.0 h1:6EhoGWWK28x1fbpA4tYTOWBkPefTDQnb8WSGXlc88kI=
        dep     go.opentelemetry.io/otel/trace  v1.24.0 h1:CsKnnL4dUAr/0llH9FKuc698G04IrpWV0MQA/Y1YELI=
        dep     go.uber.org/automaxprocs        v1.5.3  h1:kWazyxZUrS3Gs4qUpbwo5kEIMGe/DAvi5Z4tl2NW4j8=
        dep     golang.org/x/crypto     v0.24.0 h1:mnl8DM0o513X8fdIkmyFE/5hTYxbwYOjDS/+rK6qpRI=
        dep     golang.org/x/net        v0.26.0 h1:soB7SVo0PWrY4vPW/+ay0jKDNScG2X9wFeYlXIvJsOQ=
        dep     golang.org/x/oauth2     v0.21.0 h1:tsimM75w1tF/uws5rbeHzIWxEqElMehnc+iW793zsZs=
        dep     golang.org/x/sync       v0.7.0  h1:YsImfSBoP9QPYL0xyKJPq0gcaJdG3rInoqxTWbfQu9M=
        dep     golang.org/x/sys        v0.22.0 h1:RI27ohtqKCnwULzJLqkv897zojh5/DwS/ENaMzUOaWI=
        dep     golang.org/x/term       v0.22.0 h1:BbsgPEJULsl2fV/AT3v15Mjva5yXKQDyKf+TbDz7QJk=
        dep     golang.org/x/text       v0.16.0 h1:a94ExnEXNtEwYLGJSIUxnWoxoRz/ZcCsV63ROupILh4=
        dep     golang.org/x/time       v0.5.0  h1:o7cqy6amK/52YcAKIPlM3a+Fpj35zvRj2TP+e1xFSfk=
        dep     google.golang.org/api   v0.187.0        h1:Mxs7VATVC2v7CY+7Xwm4ndkX71hpElcvx0D1Ji/p1eo=
        dep     google.golang.org/genproto      v0.0.0-20240624140628-dc46fd24d27d      h1:PksQg4dV6Sem3/HkBX+Ltq8T0ke0PKIRBNBatoDTVls=
        dep     google.golang.org/genproto/googleapis/api       v0.0.0-20240617180043-68d350f18fd4      h1:MuYw1wJzT+ZkybKfaOXKp5hJiZDn2iHaXRw0mRYdHSc=
        dep     google.golang.org/genproto/googleapis/rpc       v0.0.0-20240624140628-dc46fd24d27d      h1:k3zyW3BYYR30e8v3x0bTDdE9vpYFjZHK+HcyqkrppWk=
        dep     google.golang.org/grpc  v1.64.1 h1:LKtvyfbX3UGVPFcGqJ9ItpVWW6oN/2XqTxfAnwRRXiA=
        dep     google.golang.org/protobuf      v1.34.2 h1:6xV6lTsCfpGD21XK49h7MhtcApnLqkfYgPcdHftf6hg=
        dep     gopkg.in/ini.v1 v1.67.0 h1:Dgnx+6+nfE+IfzjUEISNeydPJh9AXNNsWbGP9KzCsOA=
        dep     gopkg.in/yaml.v3        v3.0.1  h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
        build   -buildmode=exe
        build   -compiler=gc
        build   -ldflags="-s -w"
        build   -tags=selfupdate
        build   DefaultGODEBUG=httplaxcontentlength=1,httpmuxgo121=1,panicnil=1,tls10server=1,tlsrsakex=1,tlsunsafeekm=1
        build   CGO_ENABLED=0
        build   GOARCH=amd64
        build   GOOS=windows
        build   GOAMD64=v1

The only non-standard thing I notice here is -ldflags="-s -w". Those linker flags strip the symbol table and debug symbols from the binary. That's not a particularly rare thing to do, but we might as well try without it.

If you install Go, you can build restic with:

$ go install github.com/restic/restic/cmd/restic@v0.17.0

(FWIW, note there is a v0.17.1 now)

This should install a restic binary to $HOME/go/bin.

To get the exact same build as restic's config:

$ CGO_ENABLED=0 go install -ldflags="-s -w" -tags=selfupdate github.com/restic/restic/cmd/restic@v0.17.0

I'd be curious if you can reproduce with this, as well as with the extra flags dropped.

(If it is easier to cross-compile from another OS, then GOOS=windows GOARCH=amd64 go install github.com/restic/restic/cmd/restic@v0.17.0 will get you the Windows binary in $HOME/go/bin/windows_amd64.)

[prattmic]

Oh, note that the restic binary is built with go1.22.5. Perhaps try that version as well as the latest release (go1.23.1)? You can install any version of Go and then just switch with GOTOOLCHAIN :)

$ GOTOOLCHAIN=go1.22.5 go install github.com/restic/restic/cmd/restic@v0.17.0
go: downloading go1.22.5 (linux/amd64)

[distancesprinter]

Someone on https://smartos.topicbox.com/groups/smartos-discuss/Tf8f450fd133908fb mentioned that they would get similar crashes just running go version, not just their larger, more complex programs. Can either of you reproduce that?

@prattmic Yes. I installed go1.23.1.windows-amd64.msi and ran go version. Output below:

Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

PS C:\Windows\system32> go version
fatal error: runtime: mcall called on m->g0 stack

runtime stack:
runtime.throw({0x1628213?, 0xc00002be1e?})
        runtime/panic.go:1067 +0x4d fp=0xc000077c30 sp=0xc000077c00 pc=0xf70a4d
runtime.badmcall(0xc000077c78?)
        runtime/proc.go:539 +0x1f fp=0xc000077c50 sp=0xc000077c30 pc=0xf3e01f
runtime.badmcall(0xf3c1b9)
        <autogenerated>:1 +0x25 fp=0xc000077c68 sp=0xc000077c50 pc=0xf7dda5
runtime: g 0: unexpected return pc for runtime.badmcall called from 0xc000077c80
stack: frame={sp:0xc000077c50, fp:0xc000077c68} stack=[0x544d204000,0x544d3ffb20)

goroutine 1 gp=0xc00004a000 m=nil [runnable, locked to thread]:
cmd/go/internal/modcmd.init.6()
        cmd/go/internal/modcmd/verify.go:41 +0x66 fp=0xc000077e20 sp=0xc000077e18 pc=0x14c01e6
runtime.doInit1(0x1aa8840)
        runtime/proc.go:7290 +0xed fp=0xc000077f50 sp=0xc000077e20 pc=0xf4b84d
runtime.doInit(...)
        runtime/proc.go:7257
runtime.main()
        runtime/proc.go:254 +0x325 fp=0xc000077fe0 sp=0xc000077f50 pc=0xf3d6c5
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc000077fe8 sp=0xc000077fe0 pc=0xf79061

goroutine 2 gp=0xc00004a700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00004dfa8 sp=0xc00004df88 pc=0xf70b6e
runtime.goparkunlock(...)
        runtime/proc.go:430
runtime.forcegchelper()
        runtime/proc.go:337 +0xb8 fp=0xc00004dfe0 sp=0xc00004dfa8 pc=0xf3d938
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00004dfe8 sp=0xc00004dfe0 pc=0xf79061
created by runtime.init.7 in goroutine 1
        runtime/proc.go:325 +0x1a

goroutine 3 gp=0xc00004aa80 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00004ff80 sp=0xc00004ff60 pc=0xf70b6e
runtime.goparkunlock(...)
        runtime/proc.go:430
runtime.bgsweep(0xc00005a000)
        runtime/mgcsweep.go:277 +0x94 fp=0xc00004ffc8 sp=0xc00004ff80 pc=0xf26934
runtime.gcenable.gowrap1()
        runtime/mgc.go:203 +0x25 fp=0xc00004ffe0 sp=0xc00004ffc8 pc=0xf1b005
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00004ffe8 sp=0xc00004ffe0 pc=0xf79061
created by runtime.gcenable in goroutine 1
        runtime/mgc.go:203 +0x66

goroutine 18 gp=0xc0000861c0 m=nil [GC scavenge wait]:
runtime.gopark(0xc00005a000?, 0x172b0c8?, 0x1?, 0x0?, 0xc0000861c0?)
        runtime/proc.go:424 +0xce fp=0xc00008ff78 sp=0xc00008ff58 pc=0xf70b6e
runtime.goparkunlock(...)
        runtime/proc.go:430
runtime.(*scavengerState).park(0x1b01400)
        runtime/mgcscavenge.go:425 +0x49 fp=0xc00008ffa8 sp=0xc00008ff78 pc=0xf24369
runtime.bgscavenge(0xc00005a000)
        runtime/mgcscavenge.go:653 +0x3c fp=0xc00008ffc8 sp=0xc00008ffa8 pc=0xf248dc
runtime.gcenable.gowrap2()
        runtime/mgc.go:204 +0x25 fp=0xc00008ffe0 sp=0xc00008ffc8 pc=0xf1afa5
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00008ffe8 sp=0xc00008ffe0 pc=0xf79061
created by runtime.gcenable in goroutine 1
        runtime/mgc.go:204 +0xa5

goroutine 19 gp=0xc000086380 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc000091e20 sp=0xc000091e00 pc=0xf70b6e
runtime.runfinq()
        runtime/mfinal.go:193 +0x107 fp=0xc000091fe0 sp=0xc000091e20 pc=0xf1a0c7
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc000091fe8 sp=0xc000091fe0 pc=0xf79061
created by runtime.createfing in goroutine 1
        runtime/mfinal.go:163 +0x3d

goroutine 4 gp=0xc00004ae00 m=nil [chan receive]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00008bf18 sp=0xc00008bef8 pc=0xf70b6e
runtime.chanrecv(0xc0000645b0, 0x0, 0x1)
        runtime/chan.go:639 +0x41e fp=0xc00008bf90 sp=0xc00008bf18 pc=0xf0a11e
runtime.chanrecv1(0x0?, 0x0?)
        runtime/chan.go:489 +0x12 fp=0xc00008bfb8 sp=0xc00008bf90 pc=0xf09cd2
runtime.unique_runtime_registerUniqueMapCleanup.func1(...)
        runtime/mgc.go:1732
runtime.unique_runtime_registerUniqueMapCleanup.gowrap1()
        runtime/mgc.go:1735 +0x2f fp=0xc00008bfe0 sp=0xc00008bfb8 pc=0xf1e0af
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00008bfe8 sp=0xc00008bfe0 pc=0xf79061
created by unique.runtime_registerUniqueMapCleanup in goroutine 1
        runtime/mgc.go:1730 +0x96
PS C:\Windows\system32>

I can try rebuilding restic, but since I'm seeing this with a simple go version I suspect that's not going to tell us much more. Please confirm if you feel it is worthwhile.

[prattmic]

I can try rebuilding restic, but since I'm seeing this with a simple go version I suspect that's not going to tell us much more. Please confirm if you feel it is worthwhile.

Agreed. If this reproduces with go version, no need to test restic.

[distancesprinter]

Here's another:

PS C:\Windows\system32> go version
go version go1.23.1 windows/amd64
PS C:\Windows\system32> go version
fatal: morestack on g0
Exception 0x80000003 0x0 0x0 0x1358ec1
PC=0x1358ec1

runtime.abort()
        runtime/asm_amd64.s:1162 +0x1 fp=0xc000153c40 sp=0xc000153c38 pc=0x1358ec1
runtime.morestack()
        runtime/asm_amd64.s:598 +0x3d fp=0xc000153c48 sp=0xc000153c40 pc=0x135713d

goroutine 1 gp=0xc00004a000 m=nil [runnable, locked to thread]:
cmd/go/internal/test.init()
        <autogenerated>:1 +0x1b fp=0xc000153e20 sp=0xc000153e18 pc=0x18b5e3b
runtime.doInit1(0x1e86140)
        runtime/proc.go:7290 +0xed fp=0xc000153f50 sp=0xc000153e20 pc=0x132b84d
runtime.doInit(...)
        runtime/proc.go:7257
runtime.main()
        runtime/proc.go:254 +0x325 fp=0xc000153fe0 sp=0xc000153f50 pc=0x131d6c5
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc000153fe8 sp=0xc000153fe0 pc=0x1359061

goroutine 2 gp=0xc00004a700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00004dfa8 sp=0xc00004df88 pc=0x1350b6e
runtime.goparkunlock(...)
        runtime/proc.go:430
runtime.forcegchelper()
        runtime/proc.go:337 +0xb8 fp=0xc00004dfe0 sp=0xc00004dfa8 pc=0x131d938
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00004dfe8 sp=0xc00004dfe0 pc=0x1359061
created by runtime.init.7 in goroutine 1
        runtime/proc.go:325 +0x1a

goroutine 3 gp=0xc00004aa80 m=nil [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc00004ff80 sp=0xc00004ff60 pc=0x1350b6e
runtime.goparkunlock(...)
        runtime/proc.go:430
runtime.bgsweep(0xc00005a000)
        runtime/mgcsweep.go:277 +0x94 fp=0xc00004ffc8 sp=0xc00004ff80 pc=0x1306934
runtime.gcenable.gowrap1()
        runtime/mgc.go:203 +0x25 fp=0xc00004ffe0 sp=0xc00004ffc8 pc=0x12fb005
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc00004ffe8 sp=0xc00004ffe0 pc=0x1359061
created by runtime.gcenable in goroutine 1
        runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc00004ac40 m=nil [GC scavenge wait]:
runtime.gopark(0xc00005a000?, 0x1b0b0c8?, 0x1?, 0x0?, 0xc00004ac40?)
        runtime/proc.go:424 +0xce fp=0xc000061f78 sp=0xc000061f58 pc=0x1350b6e
runtime.goparkunlock(...)
        runtime/proc.go:430
runtime.(*scavengerState).park(0x1ee1400)
        runtime/mgcscavenge.go:425 +0x49 fp=0xc000061fa8 sp=0xc000061f78 pc=0x1304369
runtime.bgscavenge(0xc00005a000)
        runtime/mgcscavenge.go:653 +0x3c fp=0xc000061fc8 sp=0xc000061fa8 pc=0x13048dc
runtime.gcenable.gowrap2()
        runtime/mgc.go:204 +0x25 fp=0xc000061fe0 sp=0xc000061fc8 pc=0x12fafa5
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc000061fe8 sp=0xc000061fe0 pc=0x1359061
created by runtime.gcenable in goroutine 1
        runtime/mgc.go:204 +0xa5

goroutine 5 gp=0xc00004afc0 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        runtime/proc.go:424 +0xce fp=0xc000063e20 sp=0xc000063e00 pc=0x1350b6e
runtime.runfinq()
        runtime/mfinal.go:193 +0x107 fp=0xc000063fe0 sp=0xc000063e20 pc=0x12fa0c7
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc000063fe8 sp=0xc000063fe0 pc=0x1359061
created by runtime.createfing in goroutine 1
        runtime/mfinal.go:163 +0x3d

goroutine 6 gp=0xc00004b180 m=nil [chan receive]:
runtime.gopark(0x0?, 0x12faf01?, 0x28?, 0x61?, 0x1e86128?)
        runtime/proc.go:424 +0xce fp=0xc000051f18 sp=0xc000051ef8 pc=0x1350b6e
runtime.chanrecv(0xc00006a690, 0x0, 0x1)
        runtime/chan.go:639 +0x41e fp=0xc000051f90 sp=0xc000051f18 pc=0x12ea11e
runtime.chanrecv1(0x131d780?, 0xc000051f76?)
        runtime/chan.go:489 +0x12 fp=0xc000051fb8 sp=0xc000051f90 pc=0x12e9cd2
runtime.unique_runtime_registerUniqueMapCleanup.func1(...)
        runtime/mgc.go:1732
runtime.unique_runtime_registerUniqueMapCleanup.gowrap1()
        runtime/mgc.go:1735 +0x2f fp=0xc000051fe0 sp=0xc000051fb8 pc=0x12fe0af
runtime.goexit({})
        runtime/asm_amd64.s:1700 +0x1 fp=0xc000051fe8 sp=0xc000051fe0 pc=0x1359061
created by unique.runtime_registerUniqueMapCleanup in goroutine 1
        runtime/mgc.go:1730 +0x96
rax     0x17
rbx     0x1ee3a80
rcx     0x1ee37a0
rdx     0x0
rdi     0xda1de72000
rsi     0x1ee37a0
rbp     0xc000153c58
rsp     0xc000153c38
r8      0xc0001538e8
r9      0xc000153ac0
r10     0x0
r11     0x246
r12     0xda1e1ffa80
r13     0xc
r14     0x1ee1a80
r15     0x3ffffff
rip     0x1358ec1
rflags  0x206
cs      0x33
fs      0x53
gs      0x2b
PS C:\Windows\system32> go version
go version go1.23.1 windows/amd64
PS C:\Windows\system32>

[prattmic]

The common theme behind these crashes is they are upset about values ultimately loaded from the G, which is ultimately stored in TLS.

I could imagine this being an OS or VMM bug corrupting the TLS, but there are some other possibilities we can investigate:

1. Asynchronous preemption ends up loading the G from TLS. Could you try disabling async preemption to see if this makes the problem go away, or become like frequent? GODEBUG=asyncpreemptoff=1 go version.

2. We changed the way we manage the TLS on Windows in Go 1.20 (https://go.dev/cl/431775) and Go 1.21 (https://go.dev/cl/486816). Could you test Go 1.19, 1.20, and 1.21 to see which, if any, of these fail?

cc @qmuntal since I mention your CLs and you probably know more about Windows TLS anyway. :)

[qmuntal]

cc @qmuntal since I mention your CLs and you probably know more about Windows TLS anyway.

Before CL 431775 loading the TLS was only one instruction. That CL added a second instruction to resolve a newly added indirection:

MOVQ TLS, BX

// is tranformed into

MOVQ runtime.tls_g(SB), BX
MOVQ 0(BX)(GS*1), BX

It is probably unsafe to add a preemption point in between those instructions. If that's the case, both instructions should be marked as unsafe somehow. I'm not sure if that's already done, @prattmic could you check? I don't fully understand how is async preemption implemented 😅

[prattmic]

Great question! Async preemption won't ever preempt assembly: https://cs.opensource.google/go/go/+/master:src/runtime/preempt.go;l=408;drc=af0c40311e2ee33ecd24971257606f42a49cf593;bpv=1;bpt=1

[cherrymui]

Good point, @qmuntal . https://cs.opensource.google/go/go/+/master:src/cmd/internal/obj/x86/asm6.go;l=2249-2262 marks two-instruction TLS access nonpreemptible, and CanUse1InsnTLS is false on Windows so this code does run. However, it seems after the rewrite it doesn't use the TLS pseudo-register, so useTLS doesn't match it?

[cherrymui]

Before register ABI, we load G from TLS in almost all function prologues. With register ABI, usually we load G from the register (R14), so TLS accesses are mostly from assembly functions. But there are still a small amount of ABI0 Go functions, which will load G from TLS. Maybe they are all in the runtime package therefore not async preemptible anyway?

And at call sites in normal Go functions to ABI0 functions, we generate code sequence to reload G from TLS to R14. These could be preemptible, if the marking didn't work as expected.

[cherrymui]

Yeah, for a call from a normal Go function to an ABI0 function, we generate a load from TLS https://cs.opensource.google/go/go/+/master:src/cmd/compile/internal/amd64/ssa.go;l=1080 . After the expansion in CL 431775, and the unsafe point marking, the code looks like

    0x000a 00010 (/tmp/pp/x.go:6)   CALL    main.f(SB)
    0x000f 00015 (/tmp/pp/x.go:6)   XORPS   X15, X15
    0x0013 00019 (/tmp/pp/x.go:6)   MOVQ    runtime.tls_g(SB), R14
    0x001a 00026 (/tmp/pp/x.go:6)   MOVQ    (R14)(GS*1), R14
    0x001e 00030 (/tmp/pp/x.go:6)   PCDATA  $0, $-2
    0x001e 00030 (/tmp/pp/x.go:6)   MOVQ    (R14)(TLS*2), R14
    0x0021 00033 (/tmp/pp/x.go:6)   PCDATA  $0, $-1

The "second", actually third after expansion, instruction, MOVQ (R14)(TLS*2), R14 is marked nonpreemptible (PCDATA -2). But not the MOVQ (R14)(GS*1), R14. If a preemption happens at this instruction, and a thread switch occurs, and GS is thread dependent, it will go wrong...

[distancesprinter]

1. Asynchronous preemption ends up loading the G from TLS. Could you try disabling async preemption to see if this makes the problem go away, or become like frequent? GODEBUG=asyncpreemptoff=1 go version.

Can't seem to trigger the error with go1.23.1 with GODEBUG=asyncpreemptoff=1:

Microsoft Windows [Version 10.0.17763.6189]
(c) 2018 Microsoft Corporation. All rights reserved.

C:\Windows\system32>set GODEBUG=asyncpreemptoff=1

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>go version
go version go1.23.1 windows/amd64

C:\Windows\system32>set
ALLUSERSPROFILE=C:\ProgramData
APPDATA=C:\Users\[redacted]\AppData\Roaming
CommonProgramFiles=C:\Program Files\Common Files
CommonProgramFiles(x86)=C:\Program Files (x86)\Common Files
CommonProgramW6432=C:\Program Files\Common Files
COMPUTERNAME=[redacted]
ComSpec=C:\Windows\system32\cmd.exe
DriverData=C:\Windows\System32\Drivers\DriverData
GODEBUG=asyncpreemptoff=1
GOPATH=C:\Users\[redacted]\go
HOMEDRIVE=C:
[redacted]
NUMBER_OF_PROCESSORS=4
OS=Windows_NT
Path=C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\Go\bin;[redacted]
PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
PROCESSOR_ARCHITECTURE=AMD64
PROCESSOR_IDENTIFIER=AMD64 Family 25 Model 1 Stepping 1, AuthenticAMD
PROCESSOR_LEVEL=25
PROCESSOR_REVISION=0101
ProgramData=C:\ProgramData
ProgramFiles=C:\Program Files
ProgramFiles(x86)=C:\Program Files (x86)
ProgramW6432=C:\Program Files
PROMPT=$P$G
PSModulePath=C:\Program Files\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules
PUBLIC=C:\Users\Public
SystemDrive=C:
SystemRoot=C:\Windows
[redacted]
windir=C:\Windows

C:\Windows\system32>

2. We changed the way we manage the TLS on Windows in Go 1.20 (https://go.dev/cl/431775) and Go 1.21 (https://go.dev/cl/486816). Could you test Go 1.19, 1.20, and 1.21 to see which, if any, of these fail?

Also have not been able to trigger the error with go1.19.13:

PS C:\Windows\system32> go version
go version go1.19.13 windows/amd64
PS C:\Windows\system32> go version
go version go1.19.13 windows/amd64
PS C:\Windows\system32> go version
go version go1.19.13 windows/amd64
PS C:\Windows\system32> go version
go version go1.19.13 windows/amd64
PS C:\Windows\system32> go version
go version go1.19.13 windows/amd64
PS C:\Windows\system32> go version
go version go1.19.13 windows/amd64
PS C:\Windows\system32> go version
go version go1.19.13 windows/amd64
PS C:\Windows\system32> go version
go version go1.19.13 windows/amd64
PS C:\Windows\system32> go version
go version go1.19.13 windows/amd64
PS C:\Windows\system32> go version
go version go1.19.13 windows/amd64
PS C:\Windows\system32>

[prattmic]

Thanks all, sounds like we found the bug (or at least a bug!). @qmuntal would you like to send a CL.

I remain confused why this fails so readily on these machines but very rarely otherwise. My best guess would be that the load from GS is particularly slow for some reason, making signals more likely to land there, but that is a bit of a stretch. Inside the Bhyve VM, perhaps the load from GS is causing a VM exit, which would be very slow? But that doesn't explain the bare metal cases.

[gopherbot]

Change https://go.dev/cl/612535 mentions this issue: cmd/internal/obj/x86: mark Windows TLS access sequence nonpreemptible

[cherrymui]

Could you try if CL 612535 makes a difference? Thanks.