search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.79k stars 2.42k forks source link

'skopeo copy' gets stuck uploading images to a Docker registry on Fedora >= 40 hosts #22575

Closed debarshiray closed 6 months ago

debarshiray commented 6 months ago

This was observed when trying to add Fedora 40 to the list of host operating systems on which the upstream Toolbx CI is run. Eventually, it started happening on Fedora Rawhide hosts too.

The Toolbx test suite sets up a local temporary Docker registry to test some behaviour of the toolbox(1) binary.

First, it sets up a TLS key pair:

$ mkdir /var/tmp/foo
$ openssl req \
    -newkey rsa:4096 \
    -nodes -sha256 \
    -keyout "/var/tmp/foo/domain.key" \
    -addext "subjectAltName= DNS:localhost" \
    -x509 \
    -days 365 \
    -subj "/" \
    -out "/var/tmp/foo/domain.crt"
...
$ mkdir --parents ~/.config/containers/certs.d/localhost\:50000
$ cp /var/tmp/foo/domain.crt ~/.config/containers/certs.d/localhost\:50000

Second, it runs a Docker registry in a container:

$ podman run \
    --detach \
    --rm \
    --name docker-registry \
    --privileged \
    --volume /var/tmp/foo:/certs \
    -e REGISTRY_HTTP_ADDR=0.0.0.0:443 \
    -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
    -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
    -p 50000:443 \
    docker.io/library/registry:2
...

Finally, it populates the Docker registry with an image:

$ podman pull registry.fedoraproject.org/fedora:34
...
$ skopeo copy containers-storage:registry.fedoraproject.org/fedora:34 docker://localhost:50000/fedora:34
Getting image source signatures
Copying blob 883e787c00d4 done   | 
<stuck>

That last step to upload an older Fedora image to populate the local temporary Docker registry always gets stuck on hosts running Fedora 40 and newer, but doesn't on hosts running older Fedora, such as 38 and 39.

For various odd requirements and quirks, the Toolbx test suite specifically uploads an older Fedora based image to simplify the implementation. That said, I reproduced it also with the fedora:35, fedora:36, fedora:37, fedora:38 and fedora:39 images on a Fedora 40 host. The fedora:40 image didn't get stuck on a Fedora 40 host.

All my tests were done using Skopeo 1.15.0 on Fedora 41, 40, 39 and 38 hosts. The corresponding RPMs were:

skopeo-1.15.0-1.fc41.x86_64
skopeo-1.15.0-1.fc40.x86_64
skopeo-1.15.0-1.fc39.x86_64
skopeo-1.15.0-1.fc38.x86_64
mtrmac commented 6 months ago

Thanks for your report.

This works for me fine with

# rpm -q podman skopeo
podman-5.0.1-1.fc40.x86_64
skopeo-1.15.0-1.fc40.x86_64

If this is reproducible, can you run Skopeo with GOTRACEBACK=all and trigger a backtrace, by sending SIGABRT or, maybe, in the console, pressing Ctrl-\ , please?

debarshiray commented 6 months ago

This is what I got:

$ GOTRACEBACK=all skopeo copy \
      containers-storage:registry.fedoraproject.org/fedora:34 \
      docker://localhost:50000/fedora:34
Getting image source signatures
Copying blob 883e787c00d4 done   | 
^\SIGQUIT: quitSIGQUIT: quit

PC=PC=0x55b74ad5904e0x5603dd90d04c m= m=00 sigcode= sigcode=128128

goroutine goroutine 01 gp= gp=0x55b74c6e9a600xc0000061c0 m= m=00 mp= mp=0x55b74c6eab800x5603df29eb80 [ [idlesyscall]:
, locked to thread]:
syscall.Syscall6runtime/internal/syscall.Syscall6((0xf7)
,   0x1/usr/lib/golang/src/runtime/internal/syscall/asm_linux_amd64.s, :0xfdc36,  +0xc00063f3980xe,  fp=0x10000040x7ffef2c876c8,  sp=0x00x7ffef2c876c0,  pc=0x00x55b74ad5904e)

    runtime/internal/syscall.EpollWait/usr/lib/golang/src/syscall/syscall_linux.go(:910x5 +?0x39,  fp={0xc00063f3600x7ffef2c87764 sp=?0xc00063f300,  pc=0x100x5603dd9cff79?
, 0x1os.(*Process).blockUntilWaitable?(}0xc000318f30, )
0xf2c879b0  ?/usr/lib/golang/src/os/wait_waitid.go, :0x7ffe32? +)
0x76     fp=/usr/lib/golang/src/runtime/internal/syscall/syscall_linux.go0xc00063f438: sp=550xc00063f360 + pc=0x450x5603dd9fe1f6 fp=
0x7ffef2c87718os.(*Process).wait sp=(0x7ffef2c876c80xc000318f30 pc=)
0x55b74ad58fc5  
/usr/lib/golang/src/os/exec_unix.goruntime.netpoll:(220xc00005e000 +?0x25)
 fp=    0xc00063f498/usr/lib/golang/src/runtime/netpoll_epoll.go sp=:0xc00063f438118 pc= +0x5603dd9f73650xd2
 fp=0x7ffef2c87da8os.(*Process).Wait sp=(0x7ffef2c87718... pc=)
0x55b74ad8bd32  
/usr/lib/golang/src/os/exec.go:134
runtime.findRunnableos/exec.(*Cmd).Wait(()
0xc000191340    )
/usr/lib/golang/src/runtime/proc.go :/usr/lib/golang/src/os/exec/exec.go3470: +8970x8c6 + fp=0x450x7ffef2c87f20 fp= sp=0xc00063f4f80x7ffef2c87da8 sp= pc=0xc00063f4980x55b74ad98426 pc=
0x5603ddfb25a5
runtime.schedulegithub.com/containers/storage/pkg/unshare.(*Cmd).Run(()
0xc000301570    )
/usr/lib/golang/src/runtime/proc.go :/builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/storage/pkg/unshare/unshare_linux.go3868: +3680xb1 + fp=0x300x7ffef2c87f58 fp= sp=0xc00063f5100x7ffef2c87f20 sp= pc=0xc00063f4f80x55b74ad99991 pc=
0x5603ddfc5fb0
github.com/containers/storage/pkg/unshare.ExecRunnableruntime.park_m(({0xc000582c400x5603dea0ab40)
?   , /usr/lib/golang/src/runtime/proc.go0xc000301570:?4036} +, 0x1ec0x3 fp=?0x7ffef2c87fb0)
 sp=    0x7ffef2c87f58/builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/storage/pkg/unshare/unshare_linux.go pc=:0x55b74ad99f6c616
 +0x53runtime.mcall fp=(0xc00063f5c8)
 sp=    0xc00063f510/usr/lib/golang/src/runtime/asm_amd64.s pc=:0x5603ddfc7393458
 +0x50github.com/containers/storage/pkg/unshare.MaybeReexecUsingUserNamespace fp=(0x7ffef2c87fc80x1 sp=)
0x7ffef2c87fb0   pc=/builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/storage/pkg/unshare/unshare_linux.go0x55b74adc36d0:
604
 +goroutine 0xdae1 fp= gp=0xc00063f7e00xc0000061c0 sp= m=nil0xc00063f5c8 [ pc=semacquire0x5603ddfc712e]:

runtime.goparkmain.maybeReexec((0x55b74ad64c18)
?   , /builddir/build/BUILD/skopeo-1.15.0/cmd/skopeo/unshare_linux.go0xc0006b65a8:?36,  +0x00x193? fp=, 0xc00063f8500x0 sp=?0xc00063f7e0,  pc=0x55b74c74ebc00x5603de46d7b3?
)
main.reexecIfNecessaryForImages (/usr/lib/golang/src/runtime/proc.go:{4020xc00059f080 +?0xce,  fp=0xc00063faa00xc00040a588? sp=, 0xc00040a5680x5603dd91f674 pc=?0x55b74ad9266e}
)
    runtime.goparkunlock/builddir/build/BUILD/skopeo-1.15.0/cmd/skopeo/unshare_linux.go(:...50)
 +  0x38/usr/lib/golang/src/runtime/proc.go fp=:0xc00063f888408 sp=
0xc00063f850 pc=runtime.semacquire10x5603de46d878(
0xc000684458, 0x0main.(*copyOptions).run, (0x10xc0002e8b40, , 0x0{, 0xc00059f0800x12, )
0x2 , /usr/lib/golang/src/runtime/sema.go0x2:}160,  +{0x22c0x5603dea09cc8 fp=, 0xc00040a5f00xc000124058 sp=}0xc00040a588)
 pc=    0x55b74ada4b6c/builddir/build/BUILD/skopeo-1.15.0/cmd/skopeo/copy.go
:129sync.runtime_Semacquire +(0xfb0x55b74ad9bd6b fp=?0xc00063fc20)
 sp=    0xc00063f888/usr/lib/golang/src/runtime/sema.go pc=:0x5603de4542bb62
 +0x25main.(*copyOptions).run-fm fp=(0xc00040a628{ sp=0xc00059f0800xc00040a5f0? pc=, 0x55b74adc18850x0
?, 0x0?}sync.(*WaitGroup).Wait, ({0x55b74b60feb40x5603dea09cc8??)
,   0xc000124058/usr/lib/golang/src/sync/waitgroup.go?:}116)
 +  0x48<autogenerated> fp=:0xc00040a6501 sp= +0xc00040a6280x45 pc= fp=0x55b74ade4ce80xc00063fc60
 sp=0xc00063fc20 pc=0x5603de4707e5github.com/containers/image/v5/copy.(*imageCopier).copyLayers.func2.deferwrap2
(main.copyCmd.commandAction.func1)
(   /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/image/v5/copy/single.go:0xc000144908493,  +{0x250xc00059f080 fp=, 0xc00040a6680x2 sp=, 0xc00040a6500x2 pc=}0x55b74b6102e5)

    /builddir/build/BUILD/skopeo-1.15.0/cmd/skopeo/utils.gogithub.com/containers/image/v5/copy.(*imageCopier).copyLayers.func2:(580xc00005e200 +, 0x570xc000684450 fp=, 0xc00063fcb0{ sp=0xc0007390e00xc00063fc60,  pc=0x10x5603de454017, 
0x55b74ad6d33a?}, 0xc0006b6910, {github.com/spf13/cobra.(*Command).execute0x55b74be62210(, 0xc0001449080x55b74c74c680, }{, 0xc00015a0200xc000739140, , 0x20xc0006b6a18, )
0x2 }/builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/image/v5/copy/single.go)
:   506/builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/spf13/cobra/command.go +:0x403983 fp= +0xc00040a7f00xaca sp= fp=0xc00040a6680xc00063fe38 pc= sp=0x55b74b6101630xc00063fcb0
 pc=0x5603ddafc54a
github.com/containers/image/v5/copy.(*imageCopier).copyLayersgithub.com/spf13/cobra.(*Command).ExecuteC((0xc00005e2000xc000144608, )
{   0x55b74be62210/builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/spf13/cobra/command.go, :0x55b74c74c6801115} +)
0x3ff    fp=/builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/image/v5/copy/single.go0xc00063ff10: sp=5070xc00063fe38 + pc=0x4790x5603ddafce1f fp=
0xc00040abd0 sp=0xc00040a7f0github.com/spf13/cobra.(*Command).Execute pc=(0x55b74b60f679...
)
    /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/spf13/cobra/command.go:github.com/containers/image/v5/copy.(*copier).copySingleImage1039(
0xc0000b9a40main.main, ({)
0x55b74be62210  , /builddir/build/BUILD/skopeo-1.15.0/cmd/skopeo/main.go0x55b74c74c680:}131,  +0xc000738ae00x56,  fp=0x00xc00063ff50,  sp={0xc00063ff100xcf pc=?0x5603de45e816, 
0x0?runtime.main, (0x0)
?   }/usr/lib/golang/src/runtime/proc.go)
:   271/builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/image/v5/copy/single.go +:0x29d210 fp= +0xc00063ffe00xf05 sp= fp=0xc00063ff500xc00040b1f0 pc= sp=0x5603dd94623d0xc00040abd0
 pc=runtime.goexit0x55b74b60cc05(
{})
    github.com/containers/image/v5/copy.Image/usr/lib/golang/src/runtime/asm_amd64.s(:{16950x55b74be62210 +, 0x10x55b74c74c680 fp=}0xc00063ffe8,  sp=0xc000012ab00xc00063ffe0,  pc={0x5603dd9797210x55b74be683e0
, 
0xc000012c90goroutine }2,  gp={0xc000006c400x55b74be688c0 m=nil,  [0xc00015c600force gc (idle)}]:
, runtime.gopark0xc0000061c0(?0x0)
?   , /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/image/v5/copy/copy.go0x0:?291,  +0x00x1225? fp=, 0xc00040b6200x0 sp=?0xc00040b1f0,  pc=0x00x55b74b600185?
)
main.(*copyOptions).run.func3   (/usr/lib/golang/src/runtime/proc.go)
:   402/builddir/build/BUILD/skopeo-1.15.0/cmd/skopeo/copy.go +:0xce286 fp= +0xc00008efa80x3de sp= fp=0xc00008ef880xc00040b7a0 pc= sp=0x5603dd94666e0xc00040b620
 pc=runtime.goparkunlock0x55b74b8a19fe(
...)
    /usr/lib/golang/src/runtime/proc.go:408
github.com/containers/common/pkg/retry.IfNecessaryruntime.forcegchelper(({)
0x55b74be62210  , /usr/lib/golang/src/runtime/proc.go0x55b74c74c680:}326,  +0xc00040bb300xb8,  fp=0xc0000127f80xc00008efe0)
 sp=    0xc00008efa8/builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/common/pkg/retry/retry.go pc=:0x5603dd9464f841
 +runtime.goexit0x57( fp={0xc00040b888} sp=)
0xc00040b7a0     pc=/usr/lib/golang/src/runtime/asm_amd64.s0x55b74b13a377:
1695 +0x1 fp=0xc00008efe8main.(*copyOptions).run sp=(0xc00008efe00xc0002f6640 pc=, 0x5603dd979721{
0xc00068f0c0created by , runtime.init.70x2 in goroutine , 10x2
}   , /usr/lib/golang/src/runtime/proc.go{:0x55b74be55cc8314,  +0xc0001240680x1a}
)

    goroutine /builddir/build/BUILD/skopeo-1.15.0/cmd/skopeo/copy.go3: gp=2850xc000007180 + m=nil0x12b8 [ fp=GC sweep wait0xc00040bc20]:
 sp=runtime.gopark0xc00040b888( pc=0x10x55b74b8a1478?
, 0x0main.(*copyOptions).run-fm?(, 0x0?{, 0xc00068f0c00x0??, , 0x00x0??, )
0x0 ?/usr/lib/golang/src/runtime/proc.go}:, 402{ +0x55b74be55cc80xce? fp=, 0xc00008f7800xc000124068 sp=?0xc00008f760} pc=)
0x5603dd94666e  
<autogenerated>:runtime.goparkunlock1( +...0x45)
 fp=    0xc00040bc60/usr/lib/golang/src/runtime/proc.go sp=:0xc00040bc20408 pc=
0x55b74b8bc7e5runtime.bgsweep
(0xc0000b8000main.copyCmd.commandAction.func1)
(   /usr/lib/golang/src/runtime/mgcsweep.go:318 +0xc00014a9080xdf,  fp={0xc00008f7c80xc00068f0c0 sp=, 0xc00008f7800x2 pc=, 0x5603dd9310bf0x2
}runtime.gcenable.gowrap1)
(   )
/builddir/build/BUILD/skopeo-1.15.0/cmd/skopeo/utils.go :/usr/lib/golang/src/runtime/mgc.go58: +2030x57 + fp=0x250xc00040bcb0 fp= sp=0xc00008f7e00xc00040bc60 sp= pc=0xc00008f7c80x55b74b8a0017 pc=
0x5603dd9259a5
runtime.goexit({})
    /usr/lib/golang/src/runtime/asm_amd64.s:github.com/spf13/cobra.(*Command).execute1695( +0xc00014a9080x1,  fp={0xc00008f7e80xc00015e020 sp=, 0xc00008f7e00x2 pc=, 0x5603dd9797210x2
}created by )
runtime.gcenable     in goroutine /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/spf13/cobra/command.go1:
983  +/usr/lib/golang/src/runtime/mgc.go0xaca: fp=2030xc00040be38 + sp=0x660xc00040bcb0
 pc=
0x55b74af4854agoroutine 
4github.com/spf13/cobra.(*Command).ExecuteC gp=(0xc0000073400xc00014a608 m=nil)
 [  GC scavenge wait/builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/spf13/cobra/command.go]:
:runtime.gopark1115( +0x100000x3ff? fp=, 0xc00040bf100x5603de736588 sp=?0xc00040be38,  pc=0x00x55b74af48e1f?
, github.com/spf13/cobra.(*Command).Execute0x0(?..., )
0x0 ?/builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/spf13/cobra/command.go)
:   1039/usr/lib/golang/src/runtime/proc.go
:402main.main +(0xce)
 fp=    0xc00008ff78/builddir/build/BUILD/skopeo-1.15.0/cmd/skopeo/main.go sp=:0xc00008ff58131 pc= +0x5603dd94666e0x56
 fp=0xc00040bf50runtime.goparkunlock sp=(0xc00040bf10... pc=)
0x55b74b8aa816  
/usr/lib/golang/src/runtime/proc.go:runtime.main408(
)
runtime.(*scavengerState).park  (/usr/lib/golang/src/runtime/proc.go0x5603df29bd80:)
271  +/usr/lib/golang/src/runtime/mgcscavenge.go0x29d: fp=4250xc00040bfe0 + sp=0x490xc00040bf50 fp= pc=0xc00008ffa80x55b74ad9223d sp=
0xc00008ff78runtime.goexit pc=(0x5603dd92ea69{
}runtime.bgscavenge)
(   0xc0000b8000/usr/lib/golang/src/runtime/asm_amd64.s)
:   1695/usr/lib/golang/src/runtime/mgcscavenge.go +:0x1658 fp= +0xc00040bfe80x59 sp= fp=0xc00040bfe00xc00008ffc8 pc= sp=0x55b74adc57210xc00008ffa8
 pc=
0x5603dd92f019goroutine 
2runtime.gcenable.gowrap2 gp=(0xc000006c40)
 m=nil   [/usr/lib/golang/src/runtime/mgc.goforce gc (idle):]:
204 +runtime.gopark0x25( fp=0x00xc00008ffe0? sp=, 0xc00008ffc80x0 pc=?0x5603dd925945, 
0x0runtime.goexit?(, {0x0}?)
,   0x0/usr/lib/golang/src/runtime/asm_amd64.s?:)
1695     +/usr/lib/golang/src/runtime/proc.go0x1: fp=4020xc00008ffe8 + sp=0xce0xc00008ffe0 fp= pc=0xc00008efa80x5603dd979721 sp=
0xc00008ef88created by  pc=runtime.gcenable0x55b74ad9266e in goroutine 
1
runtime.goparkunlock    (/usr/lib/golang/src/runtime/mgc.go...:)
204  +/usr/lib/golang/src/runtime/proc.go0xa5:
408

goroutine runtime.forcegchelper18( gp=)
0xc000102700     m=nil/usr/lib/golang/src/runtime/proc.go [:finalizer wait326]:
 +runtime.gopark0xb8( fp=0xc00008e6480xc00008efe0? sp=, 0xc00008efa80x5603dd919065 pc=?0x55b74ad924f8, 
0xa8runtime.goexit?(, {0x1}?)
,   0xc0000061c0/usr/lib/golang/src/runtime/asm_amd64.s?:)
1695     +/usr/lib/golang/src/runtime/proc.go0x1: fp=4020xc00008efe8 + sp=0xce0xc00008efe0 fp= pc=0xc00008e6200x55b74adc5721 sp=
0xc00008e600created by  pc=runtime.init.70x5603dd94666e in goroutine 
1
runtime.runfinq (/usr/lib/golang/src/runtime/proc.go)
    :/usr/lib/golang/src/runtime/mfinal.go314: +1940x1a +
0x107
 fp=goroutine 0xc00008e7e03 sp= gp=0xc00008e6200xc000007180 pc= m=nil0x5603dd9249e7 [
GC sweep waitruntime.goexit]:
({runtime.gopark}()
0x1 ?/usr/lib/golang/src/runtime/asm_amd64.s, :0x01695? +, 0x10x0 fp=?0xc00008e7e8,  sp=0x00xc00008e7e0? pc=, 0x5603dd9797210x0
?created by )
runtime.createfing   in goroutine /usr/lib/golang/src/runtime/proc.go1:
402  +/usr/lib/golang/src/runtime/mfinal.go0xce: fp=1640xc00008f780 + sp=0x3d0xc00008f760
 pc=
0x55b74ad9266egoroutine 
19runtime.goparkunlock gp=(0xc00038c540... m=nil)
 [  GC worker (idle)/usr/lib/golang/src/runtime/proc.go]:
:runtime.gopark408(
0x0?runtime.bgsweep, (0x00xc0000b8000?)
,   0x0/usr/lib/golang/src/runtime/mgcsweep.go?:, 3180x0 +?0xdf,  fp=0x00xc00008f7c8? sp=)
0xc00008f780     pc=/usr/lib/golang/src/runtime/proc.go0x55b74ad7d0bf:
402runtime.gcenable.gowrap1 +(0xce)
 fp=    0xc00008a750/usr/lib/golang/src/runtime/mgc.go sp=:0xc00008a730203 pc= +0x5603dd94666e0x25
 fp=0xc00008f7e0runtime.gcBgMarkWorker sp=(0xc00008f7c8)
 pc=    0x55b74ad719a5/usr/lib/golang/src/runtime/mgc.go
:runtime.goexit1310( +{0xe5} fp=)
0xc00008a7e0     sp=/usr/lib/golang/src/runtime/asm_amd64.s0xc00008a750: pc=16950x5603dd927aa5 +
0x1runtime.goexit fp=(0xc00008f7e8{ sp=}0xc00008f7e0)
 pc=    0x55b74adc5721/usr/lib/golang/src/runtime/asm_amd64.s
:created by 1695runtime.gcenable + in goroutine 0x11 fp=
0xc00008a7e8     sp=/usr/lib/golang/src/runtime/mgc.go0xc00008a7e0: pc=2030x5603dd979721 +
0x66created by 
runtime.gcBgMarkStartWorkers
 in goroutine goroutine 14
 gp=    0xc000007340/usr/lib/golang/src/runtime/mgc.go m=nil: [1234GC scavenge wait +]:
0x1c
runtime.gopark
(goroutine 0x31b9c5? gp=, 0xc000007c000x77c543 m=nil? [, GC worker (idle)0x0]:
?runtime.gopark, (0x00x0??, , 0x00x0??)
,   0x0/usr/lib/golang/src/runtime/proc.go?:, 4020x0 +?0xce,  fp=0x00xc00008ff78? sp=)
0xc00008ff58     pc=/usr/lib/golang/src/runtime/proc.go0x55b74ad9266e:
402runtime.goparkunlock +(0xce... fp=)
0xc000090750     sp=/usr/lib/golang/src/runtime/proc.go0xc000090730: pc=4080x5603dd94666e

runtime.(*scavengerState).parkruntime.gcBgMarkWorker((0x55b74c6e7d80)
)
        /usr/lib/golang/src/runtime/mgc.go/usr/lib/golang/src/runtime/mgcscavenge.go::1310425 + +0xe50x49 fp= fp=0xc0000907e00xc00008ffa8 sp= sp=0xc0000907500xc00008ff78 pc= pc=0x5603dd927aa50x55b74ad7aa69

runtime.goexitruntime.bgscavenge(({0xc0000b8000})
)
        /usr/lib/golang/src/runtime/mgcscavenge.go/usr/lib/golang/src/runtime/asm_amd64.s::6581695 + +0x590x1 fp= fp=0xc00008ffc80xc0000907e8 sp= sp=0xc00008ffa80xc0000907e0 pc= pc=0x55b74ad7b0190x5603dd979721

runtime.gcenable.gowrap2created by (runtime.gcBgMarkStartWorkers)
 in goroutine   1/usr/lib/golang/src/runtime/mgc.go
:   204/usr/lib/golang/src/runtime/mgc.go +:0x251234 fp= +0xc00008ffe00x1c sp=
0xc00008ffc8
 pc=goroutine 0x55b74ad7194534
 gp=runtime.goexit0xc000500000( m=nil{ [}GC worker (idle))
]:
    runtime.gopark/usr/lib/golang/src/runtime/asm_amd64.s(:0x01695? +, 0x10x0 fp=?0xc00008ffe8,  sp=0x00xc00008ffe0? pc=, 0x55b74adc57210x0
?created by , runtime.gcenable0x0 in goroutine ?1)

        /usr/lib/golang/src/runtime/proc.go/usr/lib/golang/src/runtime/mgc.go::402204 + +0xce0xa5 fp=
0xc000506750
 sp=goroutine 0xc00050673018 pc= gp=0x5603dd94666e0xc000102700
 m=nilruntime.gcBgMarkWorker [(finalizer wait)
]:
    /usr/lib/golang/src/runtime/mgc.goruntime.gopark:(13100x0 +?0xe5,  fp=0x55b74be4f7580xc0005067e0? sp=, 0xc0005067500x0 pc=?0x5603dd927aa5, 
0x20runtime.goexit?(, {0x2000000020}?)
)
        /usr/lib/golang/src/runtime/asm_amd64.s/usr/lib/golang/src/runtime/proc.go::1695402 + +0x10xce fp= fp=0xc0005067e80xc00009ee20 sp= sp=0xc0005067e00xc00009ee00 pc= pc=0x5603dd9797210x55b74ad9266e

created by runtime.gcBgMarkStartWorkersruntime.runfinq in goroutine (1)

    /usr/lib/golang/src/runtime/mfinal.go   :/usr/lib/golang/src/runtime/mgc.go194: +12340x107 + fp=0x1c0xc00009efe0
 sp=
0xc00009ee20goroutine  pc=60x55b74ad709e7 gp=
0xc000007dc0runtime.goexit m=nil( [{GC worker (idle)}]:
)
runtime.gopark  (/usr/lib/golang/src/runtime/asm_amd64.s0x0:?1695,  +0x00x1? fp=, 0xc00009efe80x0 sp=?0xc00009efe0,  pc=0x00x55b74adc5721?
, created by 0x0runtime.createfing? in goroutine )
1   
/usr/lib/golang/src/runtime/proc.go :/usr/lib/golang/src/runtime/mfinal.go402: +1640xce + fp=0x3d0xc000090f50
 sp=
0xc000090f30goroutine  pc=190x5603dd94666e gp=
0xc00038e540runtime.gcBgMarkWorker m=nil( [)
GC worker (idle)    ]:
/usr/lib/golang/src/runtime/mgc.go:runtime.gopark1310( +0x00xe5? fp=, 0xc000090fe00x0 sp=?0xc000090f50,  pc=0x00x5603dd927aa5?
, runtime.goexit0x0(?{, }0x0)
?   )
/usr/lib/golang/src/runtime/asm_amd64.s :/usr/lib/golang/src/runtime/proc.go1695: +4020x1 + fp=0xce0xc000090fe8 fp= sp=0xc00008a7500xc000090fe0 sp= pc=0xc00008a7300x5603dd979721 pc=
0x55b74ad9266ecreated by 
runtime.gcBgMarkStartWorkersruntime.gcBgMarkWorker in goroutine (1)

        /usr/lib/golang/src/runtime/mgc.go/usr/lib/golang/src/runtime/mgc.go::13101234 + +0xe50x1c fp=
0xc00008a7e0
 sp=goroutine 0xc00008a75035 pc= gp=0x55b74ad73aa50xc0005001c0
 m=nilruntime.goexit [(GC worker (idle){]:
}runtime.gopark)
(   0x0/usr/lib/golang/src/runtime/asm_amd64.s?:, 16950x0 +?0x1,  fp=0x00xc00008a7e8? sp=, 0xc00008a7e00x0 pc=?0x55b74adc5721, 
0x0created by ?runtime.gcBgMarkStartWorkers)
 in goroutine   1/usr/lib/golang/src/runtime/proc.go
:   402/usr/lib/golang/src/runtime/mgc.go +:0xce1234 fp= +0xc000506f500x1c sp=
0xc000506f30
 pc=goroutine 0x5603dd94666e5
 gp=runtime.gcBgMarkWorker0xc000007c00( m=nil)
 [  GC worker (idle)/usr/lib/golang/src/runtime/mgc.go]:
:runtime.gopark1310( +0x00xe5? fp=, 0xc000506fe00x0 sp=?0xc000506f50,  pc=0x00x5603dd927aa5?
, runtime.goexit0x0(?{, }0x0)
?   )
/usr/lib/golang/src/runtime/asm_amd64.s :/usr/lib/golang/src/runtime/proc.go1695: +4020x1 + fp=0xce0xc000506fe8 fp= sp=0xc0000907500xc000506fe0 sp= pc=0xc0000907300x5603dd979721 pc=
0x55b74ad9266ecreated by 
runtime.gcBgMarkStartWorkersruntime.gcBgMarkWorker in goroutine (1)

        /usr/lib/golang/src/runtime/mgc.go/usr/lib/golang/src/runtime/mgc.go::13101234 + +0xe50x1c fp=
0xc0000907e0
 sp=goroutine 0xc0000907507 pc= gp=0x55b74ad73aa50xc0000d0000
 m=nilruntime.goexit [(GC worker (idle){]:
}runtime.gopark)
(   0x0/usr/lib/golang/src/runtime/asm_amd64.s?:, 16950x0 +?0x1,  fp=0x00xc0000907e8? sp=, 0xc0000907e00x0 pc=?0x55b74adc5721, 
0x0created by ?runtime.gcBgMarkStartWorkers)
 in goroutine   1/usr/lib/golang/src/runtime/proc.go
:   402/usr/lib/golang/src/runtime/mgc.go +:0xce1234 fp= +0xc0000917500x1c sp=
0xc000091730
 pc=goroutine 0x5603dd94666e6
 gp=runtime.gcBgMarkWorker0xc000007dc0( m=nil)
 [  GC worker (idle)/usr/lib/golang/src/runtime/mgc.go]:
:runtime.gopark1310( +0x00xe5? fp=, 0xc0000917e00x0 sp=?0xc000091750,  pc=0x00x5603dd927aa5?
, runtime.goexit0x0(?{, }0x0)
?   )
/usr/lib/golang/src/runtime/asm_amd64.s :/usr/lib/golang/src/runtime/proc.go1695: +4020x1 + fp=0xce0xc0000917e8 fp= sp=0xc000090f500xc0000917e0 sp= pc=0xc000090f300x5603dd979721 pc=
0x55b74ad9266ecreated by 
runtime.gcBgMarkStartWorkersruntime.gcBgMarkWorker in goroutine (1)

        /usr/lib/golang/src/runtime/mgc.go/usr/lib/golang/src/runtime/mgc.go::13101234 + +0xe50x1c fp=
0xc000090fe0
 sp=goroutine 0xc000090f5020 pc= gp=0x55b74ad73aa50xc00038c700
 m=nilruntime.goexit [(GC worker (idle){]:
}runtime.gopark)
(   0x0/usr/lib/golang/src/runtime/asm_amd64.s?:, 16950x0 +?0x1,  fp=0x00xc000090fe8? sp=, 0xc000090fe00x0 pc=?0x55b74adc5721, 
0x0created by ?runtime.gcBgMarkStartWorkers)
 in goroutine   1/usr/lib/golang/src/runtime/proc.go
:   402/usr/lib/golang/src/runtime/mgc.go +:0xce1234 fp= +0xc00008af500x1c sp=0xc00008af30
 pc=
0x5603dd94666egoroutine 
7runtime.gcBgMarkWorker gp=(0xc0000d0000)
 m=nil   [/usr/lib/golang/src/runtime/mgc.goGC worker (idle):]:
1310runtime.gopark +(0xe50x0 fp=?0xc00008afe0,  sp=0x00xc00008af50? pc=, 0x5603dd927aa50x0
?runtime.goexit, (0x0{?}, )
0x0 ?/usr/lib/golang/src/runtime/asm_amd64.s)
:   1695/usr/lib/golang/src/runtime/proc.go +:0x1402 fp= +0xc00008afe80xce sp= fp=0xc00008afe00xc000091750 pc= sp=0x5603dd9797210xc000091730
 pc=created by 0x55b74ad9266eruntime.gcBgMarkStartWorkers
 in goroutine runtime.gcBgMarkWorker1(
)
        /usr/lib/golang/src/runtime/mgc.go/usr/lib/golang/src/runtime/mgc.go::12341310 + +0x1c0xe5
 fp=
0xc0000917e0goroutine  sp=360xc000091750 gp= pc=0xc0005003800x55b74ad73aa5 m=nil
 [runtime.goexitGC worker (idle)(]:
{runtime.gopark}()
0x0 ?/usr/lib/golang/src/runtime/asm_amd64.s, :0x01695? +, 0x10x0 fp=?0xc0000917e8,  sp=0x00xc0000917e0? pc=, 0x55b74adc57210x0
?created by )
runtime.gcBgMarkStartWorkers     in goroutine /usr/lib/golang/src/runtime/proc.go1:
402  +/usr/lib/golang/src/runtime/mgc.go0xce: fp=12340xc000507750 + sp=0x1c0xc000507730
 pc=
0x5603dd94666egoroutine 
8runtime.gcBgMarkWorker gp=(0xc0000d01c0)
 m=nil   [/usr/lib/golang/src/runtime/mgc.goGC worker (idle):]:
1310runtime.gopark +(0xe50x0 fp=?0xc0005077e0,  sp=0x00xc000507750? pc=, 0x5603dd927aa50x0
?runtime.goexit, (0x0{?}, )
0x0 ?/usr/lib/golang/src/runtime/asm_amd64.s)
:   1695/usr/lib/golang/src/runtime/proc.go +:0x1402 fp= +0xc0005077e80xce sp= fp=0xc0005077e00xc000091f50 pc= sp=0x5603dd9797210xc000091f30
 pc=created by 0x55b74ad9266eruntime.gcBgMarkStartWorkers
 in goroutine runtime.gcBgMarkWorker1(
)
        /usr/lib/golang/src/runtime/mgc.go/usr/lib/golang/src/runtime/mgc.go::12341310 + +0x1c0xe5
 fp=
0xc000091fe0goroutine  sp=80xc000091f50 gp= pc=0xc0000d01c00x55b74ad73aa5 m=nil
 [runtime.goexitGC worker (idle)(]:
{runtime.gopark}()
0x0 ?/usr/lib/golang/src/runtime/asm_amd64.s, :0x01695? +, 0x10x0 fp=?0xc000091fe8,  sp=0x00xc000091fe0? pc=, 0x55b74adc57210x0
?created by )
runtime.gcBgMarkStartWorkers     in goroutine /usr/lib/golang/src/runtime/proc.go1:
402  +/usr/lib/golang/src/runtime/mgc.go0xce: fp=12340xc000091f50 + sp=0x1c0xc000091f30
 pc=
0x5603dd94666egoroutine 
9runtime.gcBgMarkWorker gp=(0xc0000d0380)
 m=nil   [/usr/lib/golang/src/runtime/mgc.goGC worker (idle):]:
1310runtime.gopark +(0xe50x0 fp=?0xc000091fe0,  sp=0x00xc000091f50? pc=, 0x5603dd927aa50x0
?runtime.goexit, (0x0{?}, )
0x0 ?/usr/lib/golang/src/runtime/asm_amd64.s)
:   1695/usr/lib/golang/src/runtime/proc.go +:0x1402 fp= +0xc000091fe80xce sp= fp=0xc000091fe00xc0000d6750 pc= sp=0x5603dd9797210xc0000d6730
 pc=created by 0x55b74ad9266eruntime.gcBgMarkStartWorkers
 in goroutine runtime.gcBgMarkWorker1(
)
        /usr/lib/golang/src/runtime/mgc.go/usr/lib/golang/src/runtime/mgc.go::12341310 + +0x1c0xe5
 fp=
0xc0000d67e0goroutine  sp=370xc0000d6750 gp= pc=0xc0005005400x55b74ad73aa5 m=nil
 [runtime.goexitGC worker (idle)(]:
{runtime.gopark}()
0x0 ?/usr/lib/golang/src/runtime/asm_amd64.s, :0x01695? +, 0x10x0 fp=?0xc0000d67e8,  sp=0x00xc0000d67e0? pc=, 0x55b74adc57210x0
?created by )
runtime.gcBgMarkStartWorkers     in goroutine /usr/lib/golang/src/runtime/proc.go1:
402  +/usr/lib/golang/src/runtime/mgc.go0xce: fp=12340xc000507f50 + sp=0x1c0xc000507f30
 pc=
0x5603dd94666egoroutine 
10runtime.gcBgMarkWorker gp=(0xc0000d0540)
 m=nil   [/usr/lib/golang/src/runtime/mgc.goGC worker (idle):]:
1310runtime.gopark +(0xe50x0 fp=?0xc000507fe0,  sp=0x00xc000507f50? pc=, 0x00x5603dd927aa5?
, runtime.goexit0x0(?{, }0x0)
?   )
/usr/lib/golang/src/runtime/asm_amd64.s :/usr/lib/golang/src/runtime/proc.go1695: +4020x1 + fp=0xce0xc000507fe8 fp= sp=0xc0000d6f500xc000507fe0 sp= pc=0xc0000d6f300x5603dd979721 pc=
0x55b74ad9266ecreated by 
runtime.gcBgMarkStartWorkersruntime.gcBgMarkWorker in goroutine (1)

        /usr/lib/golang/src/runtime/mgc.go/usr/lib/golang/src/runtime/mgc.go::13101234 + +0xe50x1c fp=
0xc0000d6fe0
 sp=goroutine 0xc0000d6f509 pc= gp=0x55b74ad73aa50xc0000d0380
 m=nilruntime.goexit [(GC worker (idle){]:
}runtime.gopark)
(   0x0/usr/lib/golang/src/runtime/asm_amd64.s?:, 16950x0 +?0x1,  fp=0x00xc0000d6fe8? sp=, 0xc0000d6fe00x0 pc=?0x55b74adc5721, 
0x0created by ?runtime.gcBgMarkStartWorkers)
 in goroutine   1/usr/lib/golang/src/runtime/proc.go
:   402/usr/lib/golang/src/runtime/mgc.go +:0xce1234 fp= +0xc0005027500x1c sp=
0xc000502730
 pc=goroutine 0x5603dd94666e20
 gp=runtime.gcBgMarkWorker0xc00038e700( m=nil)
 [  GC worker (idle)/usr/lib/golang/src/runtime/mgc.go]:
:runtime.gopark1310( +0x47a830b4960xe5? fp=, 0xc0005027e00x3 sp=?0xc000502750,  pc=0x420x5603dd927aa5?
, runtime.goexit0x62(?{, }0x0)
?   )
/usr/lib/golang/src/runtime/asm_amd64.s :/usr/lib/golang/src/runtime/proc.go1695: +4020x1 + fp=0xce0xc0005027e8 fp= sp=0xc00008af500xc0005027e0 sp= pc=0xc00008af300x5603dd979721 pc=
0x55b74ad9266ecreated by 
runtime.gcBgMarkStartWorkersruntime.gcBgMarkWorker in goroutine (1)

        /usr/lib/golang/src/runtime/mgc.go/usr/lib/golang/src/runtime/mgc.go::13101234 + +0xe50x1c fp=
0xc00008afe0
 sp=goroutine 0xc00008af5038 pc= gp=0x55b74ad73aa50xc000500700
 m=nilruntime.goexit [(GC worker (idle){]:
}runtime.gopark)
(   0x0/usr/lib/golang/src/runtime/asm_amd64.s?:, 16950x0 +?0x1,  fp=0x00xc00008afe8? sp=, 0xc00008afe00x0 pc=?0x55b74adc5721, 
0x0created by ?runtime.gcBgMarkStartWorkers)
 in goroutine   1/usr/lib/golang/src/runtime/proc.go
:   402/usr/lib/golang/src/runtime/mgc.go +:0xce1234 fp= +0xc0005087500x1c sp=
0xc000508730
 pc=goroutine 0x5603dd94666e11
 gp=runtime.gcBgMarkWorker0xc0000d0700( m=nil)
 [  GC worker (idle)/usr/lib/golang/src/runtime/mgc.go]:
:runtime.gopark1310( +0x47a8308bf70xe5? fp=, 0xc0005087e00x1 sp=?0xc000508750,  pc=0x580x5603dd927aa5?
, runtime.goexit0x35(?{, }0x0)
?   )
/usr/lib/golang/src/runtime/asm_amd64.s :/usr/lib/golang/src/runtime/proc.go1695: +4020x1 + fp=0xce0xc0005087e8 fp= sp=0xc0000d77500xc0005087e0 sp= pc=0xc0000d77300x5603dd979721 pc=
0x55b74ad9266ecreated by 
runtime.gcBgMarkStartWorkersruntime.gcBgMarkWorker in goroutine (1)

        /usr/lib/golang/src/runtime/mgc.go/usr/lib/golang/src/runtime/mgc.go::13101234 + +0xe50x1c fp=
0xc0000d77e0
 sp=goroutine 0xc0000d775010 pc= gp=0x55b74ad73aa50xc0000d0540
 m=nilruntime.goexit [(GC worker (idle){]:
}runtime.gopark)
(   0x47341e73de/usr/lib/golang/src/runtime/asm_amd64.s?:, 16950x0 +?0x1,  fp=0x00xc0000d77e8? sp=, 0xc0000d77e00x0 pc=?0x55b74adc5721, 
0x0created by ?runtime.gcBgMarkStartWorkers)
 in goroutine   1/usr/lib/golang/src/runtime/proc.go
:   402/usr/lib/golang/src/runtime/mgc.go +:0xce1234 fp= +0xc000502f500x1c sp=
0xc000502f30
 pc=goroutine 0x5603dd94666e12
 gp=runtime.gcBgMarkWorker0xc0000d08c0( m=nil)
    /usr/lib/golang/src/runtime/mgc.go:1310 +0xe5 fp=0xc000502fe0 sp=0xc000502f50 pc=0x5603dd927aa5 [
runtime.goexitGC worker (idle)(]:
{}runtime.gopark)
(   0x4788753a88/usr/lib/golang/src/runtime/asm_amd64.s?:, 16950x3 +?0x1,  fp=0x5c0xc000502fe8? sp=, 0xc000502fe00xb5 pc=?0x5603dd979721, 
0x0created by ?runtime.gcBgMarkStartWorkers)
 in goroutine   1/usr/lib/golang/src/runtime/proc.go
:   402/usr/lib/golang/src/runtime/mgc.go +:0xce1234 fp= +0xc0000d7f500x1c sp=
0xc0000d7f30
 pc=goroutine 0x55b74ad9266e39
 gp=runtime.gcBgMarkWorker0xc0005008c0( m=nil)
 [  GC worker (idle)/usr/lib/golang/src/runtime/mgc.go]:
:runtime.gopark1310( +0x47341e3cb00xe5? fp=, 0xc0000d7fe00x0 sp=?0xc0000d7f50,  pc=0x00x55b74ad73aa5?
, runtime.goexit0x0(?{, }0x0)
?   )
/usr/lib/golang/src/runtime/asm_amd64.s :/usr/lib/golang/src/runtime/proc.go1695: +4020x1 + fp=0xce0xc0000d7fe8 fp= sp=0xc000508f500xc0000d7fe0 sp= pc=0xc000508f300x55b74adc5721 pc=
0x5603dd94666ecreated by 
runtime.gcBgMarkStartWorkersruntime.gcBgMarkWorker in goroutine (1)

        /usr/lib/golang/src/runtime/mgc.go/usr/lib/golang/src/runtime/mgc.go::13101234 + +0xe50x1c fp=
0xc000508fe0
 sp=goroutine 0xc000508f5034 pc= gp=0x5603dd927aa50xc000502000
 m=nilruntime.goexit [(GC worker (idle){]:
}runtime.gopark)
(   0x55b74c74ebc0/usr/lib/golang/src/runtime/asm_amd64.s?:, 16950x1 +?0x1,  fp=0xc30xc000508fe8? sp=, 0xc000508fe00xf6 pc=?0x5603dd979721, 
0x0created by ?runtime.gcBgMarkStartWorkers)
 in goroutine   1/usr/lib/golang/src/runtime/proc.go
:   402/usr/lib/golang/src/runtime/mgc.go +:0xce1234 fp= +0xc0000d27500x1c sp=
0xc0000d2730
 pc=goroutine 0x55b74ad9266e11
 gp=runtime.gcBgMarkWorker0xc0000d0700( m=nil)
 [  GC worker (idle)/usr/lib/golang/src/runtime/mgc.go]:
:runtime.gopark1310( +0x47341e513c0xe5? fp=, 0xc0000d27e00x0 sp=?0xc0000d2750,  pc=0x00x55b74ad73aa5?
, runtime.goexit0x0(?{, }0x0)
?   )
/usr/lib/golang/src/runtime/asm_amd64.s :/usr/lib/golang/src/runtime/proc.go1695: +4020x1 + fp=0xce0xc0000d27e8 fp= sp=0xc0005037500xc0000d27e0 sp= pc=0xc0005037300x55b74adc5721 pc=
0x5603dd94666ecreated by 
runtime.gcBgMarkStartWorkersruntime.gcBgMarkWorker in goroutine (1)

        /usr/lib/golang/src/runtime/mgc.go/usr/lib/golang/src/runtime/mgc.go::13101234 + +0xe50x1c fp=
0xc0005037e0
 sp=goroutine 0xc00050375035 pc= gp=0x5603dd927aa50xc0005021c0
 m=nilruntime.goexit [(GC worker (idle){]:
}runtime.gopark)
(   0x4791027495/usr/lib/golang/src/runtime/asm_amd64.s?:, 16950x1 +?0x1 fp=, 0xc0005037e80x68 sp=?0xc0005037e0,  pc=0x800x5603dd979721?
, created by 0x0runtime.gcBgMarkStartWorkers? in goroutine )
1   
/usr/lib/golang/src/runtime/proc.go :/usr/lib/golang/src/runtime/mgc.go402: +12340xce + fp=0x1c0xc0000d2f50
 sp=
0xc0000d2f30goroutine  pc=400x55b74ad9266e gp=
0xc000500a80runtime.gcBgMarkWorker m=nil( [)
GC worker (idle)    ]:
/usr/lib/golang/src/runtime/mgc.goruntime.gopark:(13100x5603df302bc0 +?0xe5,  fp=0x10xc0000d2fe0? sp=, 0xc0000d2f500x9d pc=?0x55b74ad73aa5, 
0x72runtime.goexit?(, {0x0}?)
)
        /usr/lib/golang/src/runtime/asm_amd64.s/usr/lib/golang/src/runtime/proc.go::1695402 + +0x10xce fp= fp=0xc0000d2fe80xc000509750 sp= sp=0xc0000d2fe00xc000509730 pc= pc=0x55b74adc57210x5603dd94666e

created by runtime.gcBgMarkWorkerruntime.gcBgMarkStartWorkers( in goroutine )
1   
/usr/lib/golang/src/runtime/mgc.go  :/usr/lib/golang/src/runtime/mgc.go1310: +12340xe5 + fp=0x1c0xc0005097e0
 sp=
0xc000509750goroutine  pc=210x5603dd927aa5 gp=
0xc00038e8c0runtime.goexit m=nil( [{GC worker (idle)}]:
)
runtime.gopark  (/usr/lib/golang/src/runtime/asm_amd64.s0x47a8303642:?1695,  +0x10x1? fp=, 0xc0005097e80x5 sp=?0xc0005097e0 pc=0x5603dd979721, 
0x4acreated by ?runtime.gcBgMarkStartWorkers,  in goroutine 0x01?
)
        /usr/lib/golang/src/runtime/mgc.go/usr/lib/golang/src/runtime/proc.go::1234402 + +0x1c0xce
 fp=
0xc00008b750goroutine  sp=120xc00008b730 gp= pc=0xc0005016c00x55b74ad9266e m=nil
 [runtime.gcBgMarkWorkerselect(, locked to thread)
]:
    runtime.gopark/usr/lib/golang/src/runtime/mgc.go(:0xc0005057a81310? +, 0xe50x2 fp=?0xc00008b7e0,  sp=0x90xc00008b750? pc=, 0x55b74ad73aa50x69
?runtime.goexit, (0xc000505794{?})
)
        /usr/lib/golang/src/runtime/proc.go/usr/lib/golang/src/runtime/asm_amd64.s::4021695 + +0xce0x1 fp= fp=0xc0005056380xc00008b7e8 sp= sp=0xc0005056180xc00008b7e0 pc= pc=0x5603dd94666e0x55b74adc5721

created by runtime.selectgoruntime.gcBgMarkStartWorkers( in goroutine 0xc0005057a81, 
0xc000505790    , /usr/lib/golang/src/runtime/mgc.go0x0:?1234,  +0x00x1c, 
0x0
?goroutine , 360x1 gp=)
0xc000502380     m=nil/usr/lib/golang/src/runtime/select.go [:GC worker (idle)327]:
 +runtime.gopark0x725( fp=0x47a8309b3a0xc000505758? sp=, 0xc0005056380x1 pc=?0x5603dd957b25, 
0x6d?runtime.ensureSigM.func1, (0x3c)
?   , /usr/lib/golang/src/runtime/signal_unix.go0x0:?1034)
 +  0x1a5/usr/lib/golang/src/runtime/proc.go fp=:0xc0005057e0402 sp= +0xc0005057580xce pc= fp=0x5603dd970b850xc0000d3750
 sp=runtime.goexit0xc0000d3730( pc={0x55b74ad9266e}
)
runtime.gcBgMarkWorker  (/usr/lib/golang/src/runtime/asm_amd64.s)
:   1695/usr/lib/golang/src/runtime/mgc.go +:0x11310 fp= +0xc0005057e80xe5 sp= fp=0xc0005057e00xc0000d37e0 pc= sp=0x5603dd9797210xc0000d3750
 pc=created by 0x55b74ad73aa5runtime.ensureSigM
 in goroutine runtime.goexit1(
{   }/usr/lib/golang/src/runtime/signal_unix.go)
:   1017/usr/lib/golang/src/runtime/asm_amd64.s +:0xc81695
 +0x1
 fp=goroutine 0xc0000d37e850 sp= gp=0xc0000d37e00xc0000d0c40 pc= m=0x55b74adc572110
 mp=created by 0xc000640008runtime.gcBgMarkStartWorkers [ in goroutine syscall1]:

    runtime.notetsleepg/usr/lib/golang/src/runtime/mgc.go(:0x5603df301b601234,  +0xffffffffffffffff0x1c)

/usr/lib/golang/src/runtime/lock_futex.gogoroutine :22246 gp= +0xc00038ea800x29 m=nil fp= [0xc0004067a0GC worker (idle) sp=]:
0xc000406778runtime.gopark pc=(0x5603dd9176890x4788753a6a
?, os/signal.signal_recv0x1(?)
,   0x3e/usr/lib/golang/src/runtime/sigqueue.go?:, 1520x85 +?0x29,  fp=0x00xc0004067c0? sp=)
0xc0004067a0     pc=/usr/lib/golang/src/runtime/proc.go0x5603dd975fe9:
402os/signal.loop +(0xce)
 fp=    0xc00008bf50/usr/lib/golang/src/os/signal/signal_unix.go sp=:0xc00008bf3023 pc= +0x55b74ad9266e0x13
 fp=runtime.gcBgMarkWorker0xc0004067e0( sp=)
0xc0004067c0     pc=/usr/lib/golang/src/runtime/mgc.go0x5603ddfb5e93:
1310runtime.goexit +(0xe5{ fp=}0xc00008bfe0)
 sp=    0xc00008bf50/usr/lib/golang/src/runtime/asm_amd64.s pc=:0x55b74ad73aa51695
 +runtime.goexit0x1( fp={0xc0004067e8} sp=)
0xc0004067e0     pc=/usr/lib/golang/src/runtime/asm_amd64.s0x5603dd979721:
1695created by  +os/signal.Notify.func1.10x1 in goroutine  fp=10xc00008bfe8
 sp=    0xc00008bfe0/usr/lib/golang/src/os/signal/signal.go pc=:0x55b74adc5721151
 +created by 0x1fruntime.gcBgMarkStartWorkers
 in goroutine 
1goroutine 
53   gp=/usr/lib/golang/src/runtime/mgc.go0xc0000d1500: m=nil1234 [ +chan receive0x1c]:

runtime.gopark
(goroutine 0x037? gp=, 0xc0005025400x0 m=nil? [, GC worker (idle)0x0]:
?runtime.gopark, (0x00x55b74c74ebc0??, , 0x00x1??)
,   0x13/usr/lib/golang/src/runtime/proc.go?:, 4020x16 +?0xce,  fp=0x00xc000407ef8? sp=)
0xc000407ed8     pc=/usr/lib/golang/src/runtime/proc.go0x5603dd94666e:
402 +0xceruntime.chanrecv fp=(0xc0000d3f500xc000158d80 sp=, 0xc0000d3f300xc000407fc0 pc=, 0x55b74ad9266e0x1
)
runtime.gcBgMarkWorker  (/usr/lib/golang/src/runtime/chan.go)
:   583/usr/lib/golang/src/runtime/mgc.go +:0x3bf1310 fp= +0xc000407f700xe5 sp= fp=0xc000407ef80xc0000d3fe0 pc= sp=0x5603dd9117bf0xc0000d3f50
 pc=runtime.chanrecv20x55b74ad73aa5(
0x0runtime.goexit?(, {0x0}?)
)
        /usr/lib/golang/src/runtime/asm_amd64.s/usr/lib/golang/src/runtime/chan.go::1695447 + +0x10x12 fp= fp=0xc0000d3fe80xc000407f98 sp= sp=0xc0000d3fe00xc000407f70 pc= pc=0x55b74adc57210x5603dd9113f2

created by github.com/containers/storage/pkg/unshare.MaybeReexecUsingUserNamespace.func2.1runtime.gcBgMarkStartWorkers( in goroutine )
1   
/builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/storage/pkg/unshare/unshare_linux.go   :/usr/lib/golang/src/runtime/mgc.go589: +12340x5b + fp=0x1c0xc000407fe0
 sp=
0xc000407f98goroutine  pc=260x5603ddfc72db gp=
0xc0005828c0runtime.goexit m=nil( [{select}]:
)
runtime.gopark  (/usr/lib/golang/src/runtime/asm_amd64.s0xc00008c788:?1695,  +0x20x1? fp=, 0xc000407fe80x60 sp=?0xc000407fe0,  pc=0x00x5603dd979721?
, created by 0xc00008c784github.com/containers/storage/pkg/unshare.MaybeReexecUsingUserNamespace.func2? in goroutine )
1   
/usr/lib/golang/src/runtime/proc.go :/builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/storage/pkg/unshare/unshare_linux.go402: +5880xce + fp=0x6e0xc00008c630
 sp=
0xc00008c610rax     pc=0xf70x55b74ad9266e

rbx    0x1runtime.selectgo
(rcx    0xc00008c7880x5603dd90d04e, 
0xc00008c780rdx    , 0xc00063f3980x0
?rdi    , 0x10x0
, rsi    0x00xfdc?
, rbp    0x10xc00063f2f0)

    rsp    /usr/lib/golang/src/runtime/select.go0xc00063f2b0:
327r8      +0x00x725
 fp=r9     0xc00008c7500x0 sp=
0xc00008c630r10     pc=0x10000040x55b74ada3b25

r11    0x216database/sql.(*DB).connectionOpener
(r12    0xc00059d4500xc00063f3e0, 
{r13    0x55b74be624e80x5990, 
0xc000565ef0r14    }0xc0000061c0)

    r15    /usr/lib/golang/src/database/sql/sql.go0x7ffffffffffff:
1246rip     +0x5603dd90d04c0x87
 fp=rflags 0xc00008c7b80x216 sp=
0xc00008c750cs      pc=0x330x55b74b261607

fs     database/sql.OpenDB.gowrap10x0(
)
gs      0x0/usr/lib/golang/src/database/sql/sql.go
:
-----

824 +0x28 fp=0xc00008c7e0 sp=0xc00008c7b8 pc=0x55b74b25f928
runtime.goexit({})
    /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00008c7e8 sp=0xc00008c7e0 pc=0x55b74adc5721
created by database/sql.OpenDB in goroutine SIGQUIT: quit1

PC= 0x5603dd97b503/usr/lib/golang/src/database/sql/sql.go m=:2824 sigcode= +00x14c

goroutine goroutine 027 gp= gp=0xc000006a800xc000582a80 m= m=nil2 [ mp=select0xc000094808]:
 [runtime.goparkidle(]:
0xc00008cf78?, 0x2?runtime.futex, (0x00x5603df2a1a00?, , 0x800xce, ?0x0, , 0xc00008cf340x7f5013fffbf0?, )
0x0 , /usr/lib/golang/src/runtime/proc.go0x0:)
402  +/usr/lib/golang/src/runtime/sys_linux_amd64.s0xce: fp=5580xc00008cdd8 + sp=0x230xc00008cdb8 fp= pc=0x7f5013fffbc00x55b74ad9266e sp=
0x7f5013fffbb8 pc=runtime.selectgo0x5603dd97b503(
0xc00008cf78, 0xc00008cf30runtime.futexsleep, (0x00x473584bc8a??, , 0x00x13fffc40, ?0x0, ?0x5603dd917210, ?0x1)
)
        /usr/lib/golang/src/runtime/os_linux.go/usr/lib/golang/src/runtime/select.go::75327 + +0xa70x725 fp= fp=0x7f5013fffc100xc00008cef8 sp= sp=0x7f5013fffbc00xc00008cdd8 pc= pc=0x5603dd9400a70x55b74ada3b25

runtime.notetsleep_internalgithub.com/vbauerster/mpb/v8.(*pState).autoRefreshListener((0x5603df2a1a000xc0002de1e0, , 0xdf84758000xc00010ec60)
)
        /usr/lib/golang/src/runtime/lock_futex.go/builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/vbauerster/mpb/v8/progress.go::212324 + +0xb50x13f fp= fp=0x7f5013fffc500xc00008cfc0 sp= sp=0x7f5013fffc100xc00008cef8 pc= pc=0x5603dd9174d50x55b74b5f231f

runtime.notetsleepgithub.com/vbauerster/mpb/v8.NewWithContext.gowrap2((0x473584ba26)
?   , /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/vbauerster/mpb/v8/progress.go0x7f5000000002:?107)
 +  0x25/usr/lib/golang/src/runtime/lock_futex.go fp=:0xc00008cfe0235 sp= +0xc00008cfc00x29 pc= fp=0x55b74b5f05850x7f5013fffc70
 sp=runtime.goexit0x7f5013fffc50( pc={0x5603dd917609}
)
    /usr/lib/golang/src/runtime/asm_amd64.sruntime.sysmon:(1695)
 +  0x1/usr/lib/golang/src/runtime/proc.go fp=:0xc00008cfe86001 sp= +0xc00008cfe00x1c9 pc= fp=0x55b74adc57210x7f5013fffcd8
 sp=created by 0x7f5013fffc70github.com/vbauerster/mpb/v8.NewWithContext pc= in goroutine 0x5603dd952c691

runtime.mstart1 (/builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/vbauerster/mpb/v8/progress.go)
:   107/usr/lib/golang/src/runtime/proc.go +:0x3951729
 +
0x95goroutine  fp=280x7f5013fffd00 gp= sp=0xc000582c400x7f5013fffcd8 m=nil pc= [0x5603dd949455select
]:
runtime.mstart0runtime.gopark(()
0xc00009ff08    ?/usr/lib/golang/src/runtime/proc.go, :0x51686? +, 0x760x20 fp=?0x7f5013fffd30,  sp=0xea0x7f5013fffd00? pc=, 0x5603dd9493960xc00009feb6
?)
runtime.mstart  (/usr/lib/golang/src/runtime/proc.go)
:   402/usr/lib/golang/src/runtime/asm_amd64.s +:0xce394 fp= +0xc00009fd580x5 sp= fp=0xc00009fd380x7f5013fffd38 pc= sp=0x55b74ad9266e0x7f5013fffd30
 pc=runtime.selectgo0x5603dd977645(
0xc00009ff08rax    , 0xfffffffffffffffc0xc00009feac
, rbx    0x00x0?
, rcx    0x00x5603dd97b503, 
0x0rdx    ?0x0, 
0x1rdi    )
0x5603df2a1a00  
/usr/lib/golang/src/runtime/select.gorsi    :0x80327
 +rbp    0x7250x7f5013fffc00 fp=
0xc00009fe78rsp     sp=0x7f5013fffbb80xc00009fd58
 pc=r8     0x55b74ada3b250x0

r9     github.com/vbauerster/mpb/v8.(*Progress).serve0x0(
0xc0003ecfa0r10    , 0x7f5013fffbf00xc0002de1e0
, r11    0x00x246?
)
r12     0x7f5013fffbf0/builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/vbauerster/mpb/v8/progress.go
:r13    2740xffffffffffffff78 +
0x1dcr14     fp=0xc000006a800xc00009ffb8
 sp=r15    0xc00009fe780x7ffd38510500 pc=
0x55b74b5f1c7crip    
0x5603dd97b503github.com/vbauerster/mpb/v8.NewWithContext.gowrap3
(rflags )
0x246   
/builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/vbauerster/mpb/v8/progress.gocs     :0x33114
 +fs     0x280x0 fp=
0xc00009ffe0gs      sp=0x00xc00009ffb8
 pc=
-----

0x55b74b5f0528
runtime.goexit({})
    /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00009ffe8 sp=0xc00009ffe0 pc=0x55b74adc5721
created by github.com/vbauerster/mpb/v8.NewWithContext in goroutine 1
    /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/vbauerster/mpb/v8/progress.go:114 +0x4fa

goroutine 29 gp=0xc000582e00 m=nil [chan receive]:
runtime.gopark(0xc000486780?, 0x0?, 0xc8?, 0x7d?, 0x55b74ad63210?)
    /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc0000e7d90 sp=0xc0000e7d70 pc=0x55b74ad9266e
runtime.chanrecv(0xc00010ea20, 0xc0000e7f70, 0x1)
    /usr/lib/golang/src/runtime/chan.go:583 +0x3bf fp=0xc0000e7e08 sp=0xc0000e7d90 pc=0x55b74ad5d7bf
runtime.chanrecv2(0x55b74be639c0?, 0xc000732048?)
    /usr/lib/golang/src/runtime/chan.go:447 +0x12 fp=0xc0000e7e30 sp=0xc0000e7e08 pc=0x55b74ad5d3f2
github.com/vbauerster/mpb/v8.heapManager.run(0xc00010ea20)
    /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/vbauerster/mpb/v8/heap_manager.go:47 +0x96 fp=0xc0000e7fc8 sp=0xc0000e7e30 pc=0x55b74b5eed56
github.com/vbauerster/mpb/v8.NewWithContext.gowrap4()
    /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/vbauerster/mpb/v8/progress.go:115 +0x25 fp=0xc0000e7fe0 sp=0xc0000e7fc8 pc=0x55b74b5f04c5
runtime.goexit({})
    /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000e7fe8 sp=0xc0000e7fe0 pc=0x55b74adc5721
created by github.com/vbauerster/mpb/v8.NewWithContext in goroutine 1
    /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/vbauerster/mpb/v8/progress.go:115 +0x545

goroutine 30 gp=0xc000582fc0 m=nil [select]:
runtime.gopark(0xc000057b20?, 0x6?, 0x10?, 0x78?, 0xc00005794c?)
    /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc0000577e8 sp=0xc0000577c8 pc=0x55b74ad9266e
runtime.selectgo(0xc000057b20, 0xc000057940, 0x55b74b9c7ac8?, 0x0, 0xc0003ea7b0?, 0x1)
    /usr/lib/golang/src/runtime/select.go:327 +0x725 fp=0xc000057908 sp=0xc0000577e8 pc=0x55b74ada3b25
net/http.(*persistConn).roundTrip(0xc0005a1200, 0xc000040040)
    /usr/lib/golang/src/net/http/transport.go:2674 +0x979 fp=0xc000057b90 sp=0xc000057908 pc=0x55b74b1161d9
net/http.(*Transport).roundTrip(0xc0002f7680, 0xc000154120)
    /usr/lib/golang/src/net/http/transport.go:608 +0x79a fp=0xc000057db8 sp=0xc000057b90 pc=0x55b74b10a11a
net/http.(*Transport).RoundTrip(0xf?, 0x55b74be56520?)
    /usr/lib/golang/src/net/http/roundtrip.go:17 +0x13 fp=0xc000057dd8 sp=0xc000057db8 pc=0x55b74b0ef2f3
net/http.send(0xc000154120, {0x55b74be56520, 0xc0002f7680}, {0x101SIGQUIT: quit?
, PC=0x55b74c6eab800x5603dd97b501? m=, 30x0 sigcode=?0}
)

    goroutine /usr/lib/golang/src/net/http/client.go0: gp=2590xc000006fc0 + m=0x5e43 fp= mp=0xc000057fb80xc000095008 sp= [0xc000057dd8idle pc=]:
0x55b74b0acc04
net/http.(*Client).send(0xc0003ea660runtime.futex, (0xc0001541200xc000095148, , {0x800x140, ?0x0, , 0x550x0?, , 0x00x0, ?0x0})
)
        /usr/lib/golang/src/runtime/sys_linux_amd64.s/usr/lib/golang/src/net/http/client.go::557180 + +0x210x98 fp= fp=0x7f500b5ffa380xc000058030 sp= sp=0x7f500b5ffa300xc000057fb8 pc= pc=0x5603dd97b5010x55b74b0ac4b8

runtime.futexsleep(net/http.(*Client).do0x7f500b5ffab0(?0xc0003ea660, , 0xdd9172100xc000154120?)
,   0x5603dd953f87/usr/lib/golang/src/net/http/client.go?:)
724  +/usr/lib/golang/src/runtime/os_linux.go0x8dc: fp=690xc000058220 + sp=0x300xc000058030 fp= pc=0x7f500b5ffa880x55b74b0ae7dc sp=
0x7f500b5ffa38 pc=net/http.(*Client).Do0x5603dd940030(
...runtime.notesleep)
(   0xc000095148/usr/lib/golang/src/net/http/client.go)
:   590/usr/lib/golang/src/runtime/lock_futex.go
:170 +0x87 fp=0x7f500b5ffac0 sp=0x7f500b5ffa88github.com/containers/image/v5/docker.(*dockerClient).makeRequestToResolvedURLOnce pc=(0x5603dd9173a70xc0002f7400
, {0x55b74be62210runtime.mPark, (0x55b74c74c680...})
,   {/usr/lib/golang/src/runtime/proc.go0x55b74b9bc538:, 17610x5
}runtime.stopm, (0xc0006be1b0)
,   0xc000058690/usr/lib/golang/src/runtime/proc.go, :{27820x55b74be564c0 +, 0x8c0xc0003ea2a0 fp=}0x7f500b5ffaf0,  sp=0xffffffffffffffff0x7f500b5ffac0,  pc=...0x5603dd94ad4c)

    /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/image/v5/docker/docker_client.go:632 +runtime.findRunnable0x405( fp=)
0xc000058378     sp=/usr/lib/golang/src/runtime/proc.go0xc000058220: pc=35120x55b74b6cb105 +
0xd5f fp=github.com/containers/image/v5/docker.(*dockerClient).makeRequestToResolvedURL0x7f500b5ffc68( sp=0xc0002f74000x7f500b5ffaf0,  pc={0x5603dd94c8bf0x55b74be62210
, runtime.schedule0x55b74c74c680(})
,   {/usr/lib/golang/src/runtime/proc.go0x55b74b9bc538:, 38680x5 +}0xb1,  fp=0xc0006be1b00x7f500b5ffca0,  sp=0xc0000586900x7f500b5ffc68,  pc={0x5603dd94d9910x55b74be564c0
, 0xc0003ea2a0runtime.park_m}(, 0xc0000073400xffffffffffffffff)
,   .../usr/lib/golang/src/runtime/proc.go)
:   4036/builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/image/v5/docker/docker_client.go +:0x1ec559 fp= +0x7f500b5ffcf80x10d sp= fp=0x7f500b5ffca0 pc=0xc0000584880x5603dd94df6c sp=
0xc000058378 pc=runtime.mcall0x55b74b6ca8ed(
)
    /usr/lib/golang/src/runtime/asm_amd64.s:github.com/containers/image/v5/docker.(*dockerImageDestination).PutBlobWithOptions.func1458( +{0x500x55b74be55d88 fp=, 0x7f500b5ffd100xc0000f80a0 sp=}0x7f500b5ffcf8,  pc=0xc0000588f00x5603dd9776d0, 
0xc000058900rax    , 0xca0xc00005e100
, rbx    {0x00x55b74be62210
, rcx    0x55b74c74c6800x5603dd97b503}
, rdx    0xc0000588a80x0, 
0xc000058bd0rdi    )
0xc000095148    
/builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/image/v5/docker/docker_image_dest.gorsi    :0x80189
 +rbp    0x2940x7f500b5ffa78 fp=
0xc000058710rsp     sp=0x7f500b5ffa300xc000058488
 pc=r8     0x55b74b6d38b40x0

r9     0x0github.com/containers/image/v5/docker.(*dockerImageDestination).PutBlobWithOptions
(r10    0xc00005e1000x0, 
{r11    0x55b74be622100x286, 
0x55b74c74c680r12    }0x0, 
{r13    0x55b74be561680x1?
, r14    0xc0003501b00xc000006fc0?
}r15    , 0xd{
{rip    0x55b74bb861c00x5603dd97b501, 
0x0rflags }0x286, 
0xffffffffffffffffcs     , 0x33{
0x0fs     , 0x00x0
, gs     ...0x0}
, 
-----

...}, ...)
    /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/image/v5/docker/docker_image_dest.go:203 +0x858 fp=0xc000058bd0 sp=0xc000058710 pc=0x55b74b6d2d18
SIGQUIT: quitgithub.com/containers/image/v5/copy.(*imageCopier).copyBlobFromStream
(PC=0xc00005e2000x5603dd97b501,  m={40x55b74be62210 sigcode=, 00x55b74c74c680
}
, goroutine {00x7fd03d048820 gp=, 0xc0000076c0_ m=}4,  mp={0xc000095808{ [0xc000559ae0idle, ]:
0x47}, 0xaf2a800, runtime.futex{(0x00xc000095948, , 0x00x80, , ...0x0}, , 0x0..., }0x0, , ...0x0)
)
        /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/image/v5/copy/blob.go/usr/lib/golang/src/runtime/sys_linux_amd64.s::107557 + +0x96e0x21 fp= fp=0xc0000591380x7f50135ffa38 sp= sp=0xc000058bd00x7f50135ffa30 pc= pc=0x55b74b5fba8e0x5603dd97b501

github.com/containers/image/v5/copy.(*imageCopier).copyLayerFromStreamruntime.futexsleep((0xc00005e2000x7f50135ffab0, ?{, 0x55b74be622100xdd917210, ?0x55b74c74c680, }0x5603dd953f87, ?{)
0x7fd03d048820  , /usr/lib/golang/src/runtime/os_linux.go0xc0004d4060:}69,  +{0x30{ fp=0xc000559ae00x7f50135ffa88,  sp=0x470x7f50135ffa38} pc=, 0x5603dd9400300xaf2a800
, {runtime.notesleep0x0(, 0xc0000959480x0)
,   .../usr/lib/golang/src/runtime/lock_futex.go}:, 170... +}0x87,  fp=...0x7f50135ffac0)
 sp=    0x7f50135ffa88/builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/image/v5/copy/single.go pc=:0x5603dd9173a7903
 +0x2f7runtime.mPark fp=(0xc0000593a8... sp=)
0xc000059138     pc=/usr/lib/golang/src/runtime/proc.go0x55b74b615737:
1761
github.com/containers/image/v5/copy.(*imageCopier).copyLayer.func3runtime.stopm((0xc00005e200)
,   0x55b74c74c680/usr/lib/golang/src/runtime/proc.go?:, 2782{ +{0x8c0xc000559ae0 fp=, 0x7f50135ffaf00x47 sp=}0x7f50135ffac0,  pc=0xaf2a8000x5603dd94ad4c, 
{0x0, 0x0, 0x0}runtime.findRunnable, (0x0)
,   {/usr/lib/golang/src/runtime/proc.go0x55b74b9f6999:, 3512... +}0xd5f,  fp=...0x7f50135ffc68} sp=, 0x7f50135ffaf0... pc=)
0x5603dd94c8bf  
/builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/image/v5/copy/single.goruntime.schedule:(812)
 +  0x45d/usr/lib/golang/src/runtime/proc.go fp=:0xc0000597a03868 sp= +0xc0000593a80xb1 pc= fp=0x55b74b61365d0x7f50135ffca0
 sp=0x7f50135ffc68 pc=0x5603dd94d991github.com/containers/image/v5/copy.(*imageCopier).copyLayer
(0xc00005e200runtime.park_m, ({0xc0000d15000x55b74be62210)
,   0x55b74c74c680/usr/lib/golang/src/runtime/proc.go}:, 4036{ +{0x1ec0xc000559ae0 fp=, 0x7f50135ffcf80x47 sp=}0x7f50135ffca0,  pc=0xaf2a8000x5603dd94df6c, 
{0x0runtime.mcall, (0x0)
,   0x0/usr/lib/golang/src/runtime/asm_amd64.s}:, 4580x0 +, 0x50... fp=}0x7f50135ffd10,  sp=...0x7f50135ffcf8)
 pc=    0x5603dd9776d0/builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/image/v5/copy/single.go
:rax    8440xca +
0xacfrbx     fp=0x00xc000059ca0
 sp=rcx    0xc0000597a00x5603dd97b503 pc=
0x55b74b612c8frdx    
0x0
github.com/containers/image/v5/copy.(*imageCopier).copyLayers.func1rdi    (0xc0000959480x0
, rsi    {0x80{
0xc000559ae0rbp    , 0x7f50135ffa780x47
}rsp    , 0x7f50135ffa300xaf2a800
, r8     {0x00x0
, r9     0x00x0, 
0x0r10    }0x0, 
0x0r11    , 0x286{
0x55b74b9f6999r12    , 0x10x2c
}r13    , 0x1...
}r14    , 0xc0000076c0...
)
r15     0x3/builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/image/v5/copy/single.go
:rip    4590x5603dd97b501 +
0x3b2rflags  fp=0x2860xc000059f48
 sp=cs     0xc000059ca00x33 pc=
0x55b74b610732fs     
0x0github.com/containers/image/v5/copy.(*imageCopier).copyLayers.func2.gowrap3
(gs     )
0x0 
/builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/image/v5/copy/single.go
-----

:502 +0x66 fp=0xc000059fe0 sp=0xc000059f48 pc=0x55b74b610286
runtime.goexit({}SIGQUIT: quit)

    PC=/usr/lib/golang/src/runtime/asm_amd64.s0x5603dd97b501: m=16955 + sigcode=0x10 fp=
0xc000059fe8
 sp=goroutine 0xc000059fe00 pc= gp=0x55b74adc57210xc0001021c0
 m=created by 5github.com/containers/image/v5/copy.(*imageCopier).copyLayers.func2 mp= in goroutine 0xc0001000081 [
idle    ]:
/builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/image/v5/copy/single.go:502runtime.futex +(0x1540xc000100148
, 0x80
, goroutine 0x065,  gp=0x00xc000583dc0,  m=nil0x0 [, IO wait0x0]:
)
    runtime.gopark/usr/lib/golang/src/runtime/sys_linux_amd64.s(:0x50c1a5dd299f7a6557? +, 0x210x53a5c9fb021fa485 fp=?0x7f5012bffb88,  sp=0xa60x7f5012bffb80? pc=, 0x5603dd97b5010xf7
?runtime.futexsleep, (0xb0x7f5012bffc00??)
,   0x1021c0/usr/lib/golang/src/runtime/proc.go?:, 4020x7f5012bffc00 +?0xce)
 fp=    0xc0000a16f0/usr/lib/golang/src/runtime/os_linux.go sp=:0xc0000a16d069 pc= +0x55b74ad9266e0x30
 fp=0x7f5012bffbd8 sp=runtime.netpollblock0x7f5012bffb88( pc=0x55b74ae19ef80x5603dd940030?
, runtime.notesleep0x4ad5a906(?0xc000100148, )
0xb7    ?/usr/lib/golang/src/runtime/lock_futex.go)
:   170/usr/lib/golang/src/runtime/netpoll.go +:0x87573 fp= +0x7f5012bffc100xf7 sp= fp=0x7f5012bffbd80xc0000a1728 pc= sp=0x5603dd9173a70xc0000a16f0
 pc=runtime.mPark0x55b74ad8b397(
...)
internal/poll.runtime_pollWait  (/usr/lib/golang/src/runtime/proc.go0x7fd03dac8e70:, 17610x72
)
runtime.stoplockedm (/usr/lib/golang/src/runtime/netpoll.go)
:   345/usr/lib/golang/src/runtime/proc.go +:0x853026 fp= +0xc0000a17480x73 sp= fp=0xc0000a17280x7f5012bffc68 pc= sp=0x55b74adbfdc50x7f5012bffc10
 pc=internal/poll.(*pollDesc).wait0x5603dd94b5f3(
0xc0001a0e00runtime.schedule?(, )
0xc0004d9000    ?/usr/lib/golang/src/runtime/proc.go, :0x03847)
 +  0x3a/usr/lib/golang/src/internal/poll/fd_poll_runtime.go fp=:0x7f5012bffca084 sp= +0x7f5012bffc680x27 pc= fp=0x5603dd94d91a0xc0000a1770
 sp=0xc0000a1748runtime.park_m pc=(0x55b74ae36e270xc0005016c0
)
    internal/poll.(*pollDesc).waitRead/usr/lib/golang/src/runtime/proc.go(:...4036)
 +  0x1ec/usr/lib/golang/src/internal/poll/fd_poll_runtime.go fp=:0x7f5012bffcf889 sp=
0x7f5012bffca0 pc=internal/poll.(*FD).Read0x5603dd94df6c(
0xc0001a0e00runtime.mcall, ({)
0xc0004d9000    , /usr/lib/golang/src/runtime/asm_amd64.s0x800:, 4580x800 +}0x50)
 fp=    0x7f5012bffd10/usr/lib/golang/src/internal/poll/fd_unix.go sp=:0x7f5012bffcf8164 pc= +0x5603dd9776d00x27a
 fp=rax    0xc0000a18080xca sp=
0xc0000a1770rbx     pc=0x00x55b74ae3811a

rcx    0x5603dd97b503
net.(*netFD).Readrdx    (0x00xc0001a0e00
, rdi    {0xc0001001480xc0004d9000
?rsi    , 0x800x55b74c74ebc0
?rbp    , 0x7f5012bffbc80xc000552000
?rsp    }0x7f5012bffb80)

    r8     /usr/lib/golang/src/net/fd_posix.go0x0:
55r9      +0x00x25
 fp=r10    0xc0000a18500x0 sp=
0xc0000a1808r11     pc=0x2860x55b74aecc9a5

r12    0x7f5012bffba0net.(*conn).Read
(r13    0xc0005c80a80xc000505794, 
{r14    0xc0004d90000xc0001021c0?
, r15    0xc0000a19300x1fbfff?
, rip    0x55b74ad64e250x5603dd97b501?
}rflags )
0x286   
/usr/lib/golang/src/net/net.gocs     :0x33179
 +fs     0x450x0 fp=
0xc0000a1898gs      sp=0x00xc0000a1850
 pc=
-----

0x55b74aeddb05
net.(*TCPConn).Read(0x1?, {0xc0004d9000?, 0x0?, 0x1000a1930?})
    <autogenerated>:1 +SIGQUIT: quit0x25
 fp=PC=0xc0000a18c80x5603dd97b501 sp= m=0xc0000a18986 pc= sigcode=0x55b74aef04450

goroutine crypto/tls.(*atLeastReader).Read0( gp=0xc0005520000xc000102540,  m={60xc0004d9000 mp=?0xc000100808,  [0x0idle?]:
, 0xc000552000?runtime.futex}()
0x5603df3019d8  , /usr/lib/golang/src/crypto/tls/conn.go0x80:, 8060x0 +, 0x3b0x0 fp=, 0xc0000a19100x0 sp=, 0xc0000a18c80x0 pc=)
0x55b74b03c35b  
/usr/lib/golang/src/runtime/sys_linux_amd64.sbytes.(*Buffer).ReadFrom:(5570xc00013bb30 +, 0x21{ fp=0x55b74be579400x7f50121ffc30,  sp=0xc0005520000x7f50121ffc28} pc=)
0x5603dd97b501  
/usr/lib/golang/src/bytes/buffer.goruntime.futexsleep:(2110x7f50121ffca8 +?0x98,  fp=0xdd9172100xc0000a1968? sp=, 0xc0000a19100x8 pc=?0x55b74ae7a118)

    /usr/lib/golang/src/runtime/os_linux.gocrypto/tls.(*Conn).readFromUntil:(690xc00013b888 +, 0x30{ fp=0x55b74be568800x7f50121ffc80,  sp=0xc0005c80a80x7f50121ffc30} pc=, 0x5603dd9400300xc0000a1978
?runtime.notesleep)
(   0x5603df3019d8/usr/lib/golang/src/crypto/tls/conn.go)
:   828/usr/lib/golang/src/runtime/lock_futex.go +:0xde170 fp= +0xc0000a19a00x87 sp= fp=0xc0000a19680x7f50121ffcb8 pc= sp=0x55b74b03c53e0x7f50121ffc80
 pc=0x5603dd9173a7
crypto/tls.(*Conn).readRecordOrCCS(runtime.templateThread0xc00013b888(, )
0x0 )
/usr/lib/golang/src/runtime/proc.go :/usr/lib/golang/src/crypto/tls/conn.go2760: +6260x71 + fp=0x3cf0x7f50121ffcd8 fp= sp=0xc0000a1c200x7f50121ffcb8 sp= pc=0xc0000a19a00x5603dd94ac31 pc=
0x55b74b03964f
runtime.mstart1(crypto/tls.(*Conn).readRecord)
(   .../usr/lib/golang/src/runtime/proc.go)
:   1729/usr/lib/golang/src/crypto/tls/conn.go +:0x95588 fp=
0x7f50121ffd00 sp=crypto/tls.(*Conn).Read0x7f50121ffcd8( pc=0xc00013b8880x5603dd949455, 
{0xc0004b7000runtime.mstart0, (0x1000)
,   /usr/lib/golang/src/runtime/proc.go0x2:?1686} +)
0x76     fp=/usr/lib/golang/src/crypto/tls/conn.go0x7f50121ffd30: sp=13700x7f50121ffd00 + pc=0x1560x5603dd949396 fp=
0xc0000a1c90runtime.mstart sp=(0xc0000a1c20)
 pc=    0x55b74b03fef6/usr/lib/golang/src/runtime/asm_amd64.s
:394net/http.(*persistConn).Read +(0x50xc0005a1200 fp=, 0x7f50121ffd38{ sp=0xc0004b70000x7f50121ffd30? pc=, 0x5603dd9776450x55b74ada47c0
?rax    , 0xca0xc0000a1f28
?rbx    }0x0)

    rcx    /usr/lib/golang/src/net/http/transport.go0x5603dd97b503:
1976rdx     +0x00x4a
 fp=rdi    0xc0000a1cf00x5603df3019d8 sp=
0xc0000a1c90rsi     pc=0x800x55b74b1128ea

rbp    0x7f50121ffc70bufio.(*Reader).fill
(rsp    0xc00015cf600x7f50121ffc28)

    r8     /usr/lib/golang/src/bufio/bufio.go0x0:
110r9      +0x00x103
 fp=r10    0xc0000a1d280x0 sp=
0xc0000a1cf0r11     pc=0x2860x55b74aef5163

r12    bufio.(*Reader).Peek0xc000102540(
0xc00015cf60r13    , 0xffffffffffffff780x1
)
r14     0xc000102540/usr/lib/golang/src/bufio/bufio.go
:r15    1480x7ffd385105a0 +
0x53rip     fp=0x5603dd97b5010xc0000a1d48
 sp=rflags 0xc0000a1d280x286 pc=
0x55b74aef5293cs     
0x33
net/http.(*persistConn).readLoopfs     (0x00xc0005a1200
)
gs      0x0/usr/lib/golang/src/net/http/transport.go
:
-----

2140 +0x1b9 fp=0xc0000a1fc8 sp=0xc0000a1d48 pc=0x55b74b113679
net/http.(*Transport).dialConn.gowrap2()
    /usr/lib/golang/src/net/http/transport.go:1798 +0x25 fp=0xc0000a1fe0 sp=0xc0000a1fc8SIGQUIT: quit pc=
0x55b74b111e05PC=
0x5603dd97b501runtime.goexit m=(7{ sigcode=}0)

/usr/lib/golang/src/runtime/asm_amd64.sgoroutine :01695 gp= +0xc000007a400x1 m= fp=70xc0000a1fe8 mp= sp=0xc0000c60080xc0000a1fe0 [ pc=idle0x55b74adc5721]:

created by runtime.futexnet/http.(*Transport).dialConn( in goroutine 0xc0000c614857, 
0x80,   0x0/usr/lib/golang/src/net/http/transport.go, :0x01798,  +0x00x152f, 
0x0
)
goroutine   66/usr/lib/golang/src/runtime/sys_linux_amd64.s gp=:0xc000502700557 m=nil + [0x21select fp=]:
0x7f50117ffa38runtime.gopark sp=(0x7f50117ffa300xc0003e7f48 pc=?0x5603dd97b501, 
0x2runtime.futexsleep?(, 0x7f50117ffab00xc0??, , 0xdd9172100x7d??, , 0x5603dd953f870xc0003e7ef4??)
)
        /usr/lib/golang/src/runtime/os_linux.go/usr/lib/golang/src/runtime/proc.go::69402 + +0x300xce fp= fp=0x7f50117ffa880xc0003e7d98 sp= sp=0x7f50117ffa380xc0003e7d78 pc= pc=0x5603dd9400300x55b74ad9266e

runtime.notesleepruntime.selectgo((0xc0000c61480xc0003e7f48)
,   0xc0003e7ef0/usr/lib/golang/src/runtime/lock_futex.go, :0xc00015fcc0170? +, 0x870x0 fp=, 0x7f50117ffac00xc0003ea390 sp=?0x7f50117ffa88,  pc=0x10x5603dd9173a7)

    /usr/lib/golang/src/runtime/select.goruntime.mPark:(327... +)
0x725    fp=/usr/lib/golang/src/runtime/proc.go0xc0003e7eb8: sp=17610xc0003e7d98
 pc=runtime.stopm0x55b74ada3b25(
)
net/http.(*persistConn).writeLoop   (/usr/lib/golang/src/runtime/proc.go0xc0005a1200:)
2782     +/usr/lib/golang/src/net/http/transport.go0x8c: fp=24430x7f50117ffaf0 + sp=0xf00x7f50117ffac0 fp= pc=0xc0003e7fc80x5603dd94ad4c sp=
0xc0003e7eb8 pc=0x55b74b115150
net/http.(*Transport).dialConn.gowrap3()
runtime.findRunnable    (/usr/lib/golang/src/net/http/transport.go)
:   1799/usr/lib/golang/src/runtime/proc.go +:0x253512 fp= +0xc0003e7fe00xd5f sp= fp=0xc0003e7fc80x7f50117ffc68 pc= sp=0x55b74b111da50x7f50117ffaf0
 pc=runtime.goexit0x5603dd94c8bf(
{runtime.schedule}()
)
        /usr/lib/golang/src/runtime/asm_amd64.s/usr/lib/golang/src/runtime/proc.go::16953868 + +0x10xb1 fp= fp=0xc0003e7fe80x7f50117ffca0 sp= sp=0xc0003e7fe00x7f50117ffc68 pc= pc=0x55b74adc57210x5603dd94d991

created by net/http.(*Transport).dialConnruntime.park_m in goroutine (570xc0005008c0
)
        /usr/lib/golang/src/net/http/transport.go/usr/lib/golang/src/runtime/proc.go::17994036 + +0x15850x1ec
 fp=0x7f50117ffcf8 sp=
0x7f50117ffca0rax     pc=0xfffffffffffffffc0x5603dd94df6c

rbx    runtime.mcall0x4(
)
rcx     0x55b74ad5904e/usr/lib/golang/src/runtime/asm_amd64.s
:rdx    4580x80 +
0x50rdi     fp=0x40x7f50117ffd10
 sp=rsi    0x7f50117ffcf80x7ffef2c87764 pc=
0x5603dd9776d0rbp    
0x7ffef2c87708rax    
0xcarsp    
0x7ffef2c876c0rbx    
0x0r8     
0x0rcx    
0x5603dd97b503r9     
0x0rdx    
0x0r10    
0x95rdi    
0xc0000c6148r11    
0x246rsi    
0x80r12    
0x7ffef2c87798rbp    
0x7f50117ffa78r13    
0x1rsp    
0x7f50117ffa30r14    
0x55b74c6e9a60r8     
0x0r15    
0x9r9     
0x0rip    
0x55b74ad5904er10    
0x0rflags 
0x246r11    
0x286cs     
0x33r12    
0x1fs     
0x0r13    
0x1gs     
0x0r14    
0xc000007a40
-----

r15    0x9
rip    0x5603dd97b501
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x55b74adc6f37 m=2 sigcode=0

goroutine 0 gp=0xc000006a80SIGQUIT: quit m=
2PC= mp=0x5603dd97b5010xc000094808 m= [8idle sigcode=]:
0

goroutine 0 gp=runtime.usleep0xc0005821c0( m=0x2808)
 mp=    0xc000580008/usr/lib/golang/src/runtime/sys_linux_amd64.s [:idle135]:
 +0x37runtime.futex fp=(0x7fd03d9ffc700xc000580148 sp=, 0x7fd03d9ffc500x80 pc=, 0x55b74adc6f370x0
, 0x0runtime.sysmon, (0x0)
,   0x0/usr/lib/golang/src/runtime/proc.go)
:   5965/usr/lib/golang/src/runtime/sys_linux_amd64.s +:0xa5557 fp= +0x7fd03d9ffcd80x21 sp= fp=0x7fd03d9ffc700x7f5010dffa38 pc= sp=0x55b74ad9eb450x7f5010dffa30
 pc=0x5603dd97b501runtime.mstart1
(runtime.futexsleep)
(   0x7f5010dffab0/usr/lib/golang/src/runtime/proc.go?:, 17290xdd917210 +?0x95,  fp=0x5603dd953f870x7fd03d9ffd00? sp=)
0x7fd03d9ffcd8   pc=/usr/lib/golang/src/runtime/os_linux.go0x55b74ad95455:
69runtime.mstart0 +(0x30)
 fp=    0x7f5010dffa88/usr/lib/golang/src/runtime/proc.go sp=:0x7f5010dffa381686 pc= +0x5603dd9400300x76
 fp=runtime.notesleep0x7fd03d9ffd30(0xc000580148 sp=)
0x7fd03d9ffd00   pc=/usr/lib/golang/src/runtime/lock_futex.go0x55b74ad95396:
170runtime.mstart +(0x87)
 fp=    0x7f5010dffac0/usr/lib/golang/src/runtime/asm_amd64.s sp=:0x7f5010dffa88394 pc= +0x5603dd9173a70x5
 fp=runtime.mPark0x7fd03d9ffd38( sp=...0x7fd03d9ffd30)
 pc=    0x55b74adc3645/usr/lib/golang/src/runtime/proc.go
:rax    17610xfffffffffffffffc

runtime.stopmrbx    (0x280)

    rcx    /usr/lib/golang/src/runtime/proc.go0x55b74adc6f37:
2782rdx     +0x00x8c
 fp=rdi    0x7f5010dffaf00x7fd03d9ffc50 sp=
0x7f5010dffac0rsi     pc=0x00x5603dd94ad4c

rbp    0x7fd03d9ffc60
rsp    runtime.findRunnable0x7fd03d9ffc50(
)
r8      0x15ce08/usr/lib/golang/src/runtime/proc.go
:r9     35120x9da084bccb +
0xd5fr10     fp=0x00x7f5010dffc68
 sp=r11    0x7f5010dffaf00x212 pc=
0x5603dd94c8bfr12    
0x7fd03d9ffc50runtime.schedule
(r13    )
0xffffffffffffff78  
/usr/lib/golang/src/runtime/proc.gor14    :0xc000006a803868
 +r15    0xb10x7ffef2c87b30 fp=
0x7f5010dffca0rip     sp=0x55b74adc6f370x7f5010dffc68
 pc=rflags 0x5603dd94d9910x212

runtime.park_mcs     (0x330xc0000d0540
)
fs      0x0/usr/lib/golang/src/runtime/proc.go
:gs     40360x0 +
0x1ec
-----

 fp=0x7f5010dffcf8 sp=0x7f5010dffca0 pc=0x5603dd94df6c
runtime.mcall()
    /usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=SIGQUIT: quit0x7f5010dffd10
 sp=PC=0x7f5010dffcf80x55b74adc7501 pc= m=0x5603dd9776d03
 sigcode=rax    00xca

goroutine 
0rbx     gp=0x00xc000006fc0
 m=rcx    30x5603dd97b503 mp=
0xc000095008rdx     [0x0idle
]:
rdi    0xc000580148
rsi    runtime.futex0x80(
0xc000095148rbp    , 0x7f5010dffa780x80
, rsp    0x00x7f5010dffa30, 
0x0r8     , 0x00x0
, r9     0x00x0)

    r10    /usr/lib/golang/src/runtime/sys_linux_amd64.s0x0:
557r11     +0x2860x21
 fp=r12    0x7fd03cfffa780x7f5010dffaa8 sp=
0x7fd03cfffa70r13     pc=0x10x55b74adc7501

r14    0xc0005821c0runtime.futexsleep
(r15    0x7fd03cfffaf00x3?
, rip    0x4ad632100x5603dd97b501?
, rflags 0x55b74ad9ff870x286?
)
cs      0x33/usr/lib/golang/src/runtime/os_linux.go
:fs     690x0 +
0x30gs      fp=0x00x7fd03cfffac8
 sp=
-----

0x7fd03cfffa78 pc=0x55b74ad8c030
runtime.notesleep(0xc000095148)
    /usr/lib/golang/src/runtime/lock_futex.go:170 +0x87 fp=0x7fd03cfffb00 sp=0x7fd03cfffac8 pc=0x55b74ad633a7
runtime.mPark(...)
    /usr/lib/golang/src/runtime/proc.go:1761
runtime.stopm()
    /usr/lib/golang/src/runtime/proc.go:2782 +0x8c fp=SIGQUIT: quit0x7fd03cfffb30
 sp=PC=0x7fd03cfffb000x5603dd97b501 pc= m=0x55b74ad96d4c9
 sigcode=0

goroutine 0 gp=0xc0000d0a80runtime.findRunnable m=(9)
 mp=    0xc0000c6808/usr/lib/golang/src/runtime/proc.go [:idle3512]:
 +0xd5fruntime.futex fp=(0x7fd03cfffca80xc0000c6948 sp=, 0x7fd03cfffb300x80 pc=, 0x55b74ad988bf0x0
, runtime.schedule0x0(, 0x0, 0x0)
)
        /usr/lib/golang/src/runtime/sys_linux_amd64.s/usr/lib/golang/src/runtime/proc.go::5573868 + +0x210xb1 fp= fp=0x7f500abffb880x7fd03cfffce0 sp= sp=0x7f500abffb800x7fd03cfffca8 pc= pc=0x5603dd97b5010x55b74ad99991

runtime.futexsleepruntime.goexit0((0x7f500abffc000xc000e5a380??, )
0xdd917210  ?/usr/lib/golang/src/runtime/proc.go, :0x5603000000014181? +)
0x18     fp=/usr/lib/golang/src/runtime/os_linux.go0x7fd03cfffcf8: sp=690x7fd03cfffce0 + pc=0x300x55b74ad9a838 fp=
0x7f500abffbd8 sp=runtime.mcall0x7f500abffb88( pc=)
0x5603dd940030  
/usr/lib/golang/src/runtime/asm_amd64.sruntime.notesleep:(4580xc0000c6948 +)
0x50     fp=/usr/lib/golang/src/runtime/lock_futex.go0x7fd03cfffd10: sp=1700x7fd03cfffcf8 + pc=0x870x55b74adc36d0 fp=
0x7f500abffc10rax     sp=0xca0x7f500abffbd8
 pc=rbx    0x5603dd9173a70x0

rcx    runtime.mPark0x55b74adc7503(
...rdx    )
0x0 
/usr/lib/golang/src/runtime/proc.gordi    :0xc0000951481761

rsi    runtime.stopm0x80(
)
rbp     0x7fd03cfffab8/usr/lib/golang/src/runtime/proc.go
:rsp    27820x7fd03cfffa70 +
0x8cr8      fp=0x00x7f500abffc40
 sp=r9     0x7f500abffc100x0 pc=
0x5603dd94ad4cr10    
0x0runtime.startlockedm
(r11    0x5603dd94bb120x286?
)
r12     0x6/usr/lib/golang/src/runtime/proc.go
:r13    30540x1 +
0x5er14     fp=0xc000006fc00x7f500abffc68
 sp=r15    0x7f500abffc400x9 pc=
0x5603dd94b85erip    
0x55b74adc7501runtime.schedule
(rflags )
0x286   
/usr/lib/golang/src/runtime/proc.gocs     :0x333914
 +fs     0x6a0x0 fp=
0x7f500abffca0gs      sp=0x00x7f500abffc68
 pc=
-----

0x5603dd94d94a
runtime.park_m(0xc0000d0540)
    /usr/lib/golang/src/runtime/proc.go:4036 +0x1ec fp=0x7f500abffcf8 sp=0x7f500abffca0 pc=0x5603dd94df6cSIGQUIT: quit

runtime.mcallPC=(0x55b74adc7501)
 m= 4/usr/lib/golang/src/runtime/asm_amd64.s sigcode=:0458
 +
0x50goroutine  fp=00x7f500abffd10 gp= sp=0xc0000076c00x7f500abffcf8 m= pc=40x5603dd9776d0 mp=
0xc000095808rax     [0xcaidle
]:
rbx    0x0runtime.futex
(rcx    0xc0000959480x5603dd97b503, 
0x80rdx    , 0x00x0
, rdi    0x00xc0000c6948, 
0x0rsi    , 0x800x0
)
rbp     0x7f500abffbc8/usr/lib/golang/src/runtime/sys_linux_amd64.s
:rsp    5570x7f500abffb80 +
0x21r8      fp=0x00x7fd037dffa38
 sp=r9     0x7fd037dffa300x0 pc=
0x55b74adc7501r10    
0x0runtime.futexsleep
(r11    0x7fd037dffab00x286?
, r12    0x4ad632100x7f500abffba8?
, r13    0x55b74ad9ff870x1?
)
r14     0xc0000d0a80/usr/lib/golang/src/runtime/os_linux.go
:r15    690x9 +
0x30rip     fp=0x5603dd97b5010x7fd037dffa88
 sp=rflags 0x7fd037dffa380x286 pc=
0x55b74ad8c030cs     
0x33runtime.notesleep
(fs     0xc0000959480x0)

    gs     /usr/lib/golang/src/runtime/lock_futex.go0x0:
170
-----

 +0x87 fp=0x7fd037dffac0 sp=0x7fd037dffa88 pc=0x55b74ad633a7
runtime.mPark(...)
    /usr/lib/golang/src/runtime/proc.go:1761
runtime.stopm()
    /usr/lib/golang/src/runtime/proc.go:2782 +0x8c fp=0x7fd037dffaf0 sp=SIGQUIT: quit0x7fd037dffac0
 pc=PC=0x55b74ad96d4c0x5603dd97b501
 m=10 sigcode=0

goroutine runtime.findRunnable50( gp=)
0xc0000d0c40     m=/usr/lib/golang/src/runtime/proc.go10: mp=35120xc000640008 + [0xd5fsyscall fp=]:
0x7fd037dffc68 sp=runtime.notetsleepg0x7fd037dffaf0( pc=0x5603df301b600x55b74ad988bf, 
0xffffffffffffffffruntime.schedule)
(   )
/usr/lib/golang/src/runtime/lock_futex.go   :/usr/lib/golang/src/runtime/proc.go246: +38680x29 + fp=0xb10xc0004067a0 fp= sp=0x7fd037dffca00xc000406778 sp= pc=0x7fd037dffc680x5603dd917689 pc=
0x55b74ad99991os/signal.signal_recv
()
runtime.park_m  (/usr/lib/golang/src/runtime/sigqueue.go0xc00038f6c0:)
152  +/usr/lib/golang/src/runtime/proc.go0x29: fp=40360xc0004067c0 + sp=0x1ec0xc0004067a0 fp= pc=0x7fd037dffcf80x5603dd975fe9 sp=
0x7fd037dffca0 pc=os/signal.loop0x55b74ad99f6c(
)
runtime.mcall   (/usr/lib/golang/src/os/signal/signal_unix.go)
:   23/usr/lib/golang/src/runtime/asm_amd64.s +:0x13458 fp= +0xc0004067e00x50 sp= fp=0xc0004067c00x7fd037dffd10 pc= sp=0x5603ddfb5e930x7fd037dffcf8
 pc=runtime.goexit0x55b74adc36d0(
{rax    }0xca)

    rbx    /usr/lib/golang/src/runtime/asm_amd64.s0x0:
1695rcx     +0x55b74adc75030x1
 fp=rdx    0xc0004067e80x0 sp=
0xc0004067e0rdi     pc=0xc0000959480x5603dd979721

rsi    created by 0x80os/signal.Notify.func1.1
 in goroutine rbp    10x7fd037dffa78

    rsp    /usr/lib/golang/src/os/signal/signal.go0x7fd037dffa30:
151r8      +0x00x1f

r9     rax    0x00xca

r10    rbx    0x00x0

r11    rcx    0x2860x5603dd97b503

r12    rdx    0x190x0

r13    rdi    0x10x5603df301b60

r14    rsi    0xc0000076c00x80

r15    rbp    0xf0xc000406728

rip    rsp    0x55b74adc75010xc0004066e0

rflags r8     0x2860x0

cs     r9     0x330x0

fs     r10    0x00x0

gs     r11    0x00x286

-----

r12    0x0
r13    0x1
r14    0xc0000d0c40
r15    0x9SIGQUIT: quit

rip    PC=0x5603dd97b5010x55b74adc7501
 m=rflags 50x286 sigcode=
0cs     
0x33

goroutine fs     00x0 gp=
0xc0001021c0gs      m=0x05
 mp=0xc000100008 [idle]:
runtime.futex(0xc000100148, 0x80, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7fd0373ffa38 sp=0x7fd0373ffa30 pc=0x55b74adc7501
runtime.futexsleep(0x7fd0373ffab0?, 0x4ad63210?, 0x55b74ad9ff87?)
    /usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0x7fd0373ffa88 sp=0x7fd0373ffa38 pc=0x55b74ad8c030
runtime.notesleep(0xc000100148)
    /usr/lib/golang/src/runtime/lock_futex.go:170 +0x87 fp=0x7fd0373ffac0 sp=0x7fd0373ffa88 pc=0x55b74ad633a7
runtime.mPark(...)
    /usr/lib/golang/src/runtime/proc.go:1761
runtime.stopm()
    /usr/lib/golang/src/runtime/proc.go:2782 +0x8c fp=0x7fd0373ffaf0 sp=0x7fd0373ffac0 pc=0x55b74ad96d4c
runtime.findRunnable()
    /usr/lib/golang/src/runtime/proc.go:3512 +0xd5f fp=0x7fd0373ffc68 sp=0x7fd0373ffaf0 pc=0x55b74ad988bf
runtime.schedule()
    /usr/lib/golang/src/runtime/proc.go:3868 +0xb1 fp=0x7fd0373ffca0 sp=0x7fd0373ffc68 pc=0x55b74ad99991
runtime.park_m(0xc000502700)
    /usr/lib/golang/src/runtime/proc.go:4036 +0x1ec fp=0x7fd0373ffcf8 sp=0x7fd0373ffca0 pc=0x55b74ad99f6c
runtime.mcall()
    /usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0x7fd0373ffd10 sp=0x7fd0373ffcf8 pc=0x55b74adc36d0
rax    0xca
rbx    0x0
rcx    0x55b74adc7503
rdx    0x0
rdi    0xc000100148
rsi    0x80
rbp    0x7fd0373ffa78
rsp    0x7fd0373ffa30
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x6
r13    0x1
r14    0xc0001021c0
r15    0x9
rip    0x55b74adc7501
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x55b74adc7501 m=6 sigcode=0

goroutine 0 gp=0xc000102540 m=6 mp=0xc000100808 [idle]:
runtime.futex(0x55b74c74d9d8, 0x80, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7fd0369ffc30 sp=0x7fd0369ffc28 pc=0x55b74adc7501
runtime.futexsleep(0x7fd0369ffca8?, 0x4ad63210?, 0x8?)
    /usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0x7fd0369ffc80 sp=0x7fd0369ffc30 pc=0x55b74ad8c030
runtime.notesleep(0x55b74c74d9d8)
    /usr/lib/golang/src/runtime/lock_futex.go:170 +0x87 fp=0x7fd0369ffcb8 sp=0x7fd0369ffc80 pc=0x55b74ad633a7
runtime.templateThread()
    /usr/lib/golang/src/runtime/proc.go:2760 +0x71 fp=0x7fd0369ffcd8 sp=0x7fd0369ffcb8 pc=0x55b74ad96c31
runtime.mstart1()
    /usr/lib/golang/src/runtime/proc.go:1729 +0x95 fp=0x7fd0369ffd00 sp=0x7fd0369ffcd8 pc=0x55b74ad95455
runtime.mstart0()
    /usr/lib/golang/src/runtime/proc.go:1686 +0x76 fp=0x7fd0369ffd30 sp=0x7fd0369ffd00 pc=0x55b74ad95396
runtime.mstart()
    /usr/lib/golang/src/runtime/asm_amd64.s:394 +0x5 fp=0x7fd0369ffd38 sp=0x7fd0369ffd30 pc=0x55b74adc3645
rax    0xca
rbx    0x0
rcx    0x55b74adc7503
rdx    0x0
rdi    0x55b74c74d9d8
rsi    0x80
rbp    0x7fd0369ffc70
rsp    0x7fd0369ffc28
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0xc000102540
r13    0xffffffffffffff78
r14    0xc000102540
r15    0x7ffef2c87bd0
rip    0x55b74adc7501
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x55b74adc7501 m=7 sigcode=0

goroutine 0 gp=0xc000007a40 m=7 mp=0xc0000c6008 [idle]:
runtime.futex(0xc0000c6148, 0x80, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7fd035fffa78 sp=0x7fd035fffa70 pc=0x55b74adc7501
runtime.futexsleep(0x7fd035fffaf0?, 0x4ad63210?, 0x55b74ad9ff87?)
    /usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0x7fd035fffac8 sp=0x7fd035fffa78 pc=0x55b74ad8c030
runtime.notesleep(0xc0000c6148)
    /usr/lib/golang/src/runtime/lock_futex.go:170 +0x87 fp=0x7fd035fffb00 sp=0x7fd035fffac8 pc=0x55b74ad633a7
runtime.mPark(...)
    /usr/lib/golang/src/runtime/proc.go:1761
runtime.stopm()
    /usr/lib/golang/src/runtime/proc.go:2782 +0x8c fp=0x7fd035fffb30 sp=0x7fd035fffb00 pc=0x55b74ad96d4c
runtime.findRunnable()
    /usr/lib/golang/src/runtime/proc.go:3512 +0xd5f fp=0x7fd035fffca8 sp=0x7fd035fffb30 pc=0x55b74ad988bf
runtime.schedule()
    /usr/lib/golang/src/runtime/proc.go:3868 +0xb1 fp=0x7fd035fffce0 sp=0x7fd035fffca8 pc=0x55b74ad99991
runtime.goexit0(0xc0000fc380?)
    /usr/lib/golang/src/runtime/proc.go:4181 +0x18 fp=0x7fd035fffcf8 sp=0x7fd035fffce0 pc=0x55b74ad9a838
runtime.mcall()
    /usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0x7fd035fffd10 sp=0x7fd035fffcf8 pc=0x55b74adc36d0
rax    0xca
rbx    0x0
rcx    0x55b74adc7503
rdx    0x0
rdi    0xc0000c6148
rsi    0x80
rbp    0x7fd035fffab8
rsp    0x7fd035fffa70
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x6
r13    0x1
r14    0xc000007a40
r15    0xf
rip    0x55b74adc7501
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x55b74adc7501 m=8 sigcode=0

goroutine 0 gp=0xc0005821c0 m=8 mp=0xc000580008 [idle]:
runtime.futex(0xc000580148, 0x80, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7fd0355ffa38 sp=0x7fd0355ffa30 pc=0x55b74adc7501
runtime.futexsleep(0x7fd0355ffab0?, 0x4ad63210?, 0x1355ffae0?)
    /usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0x7fd0355ffa88 sp=0x7fd0355ffa38 pc=0x55b74ad8c030
runtime.notesleep(0xc000580148)
    /usr/lib/golang/src/runtime/lock_futex.go:170 +0x87 fp=0x7fd0355ffac0 sp=0x7fd0355ffa88 pc=0x55b74ad633a7
runtime.mPark(...)
    /usr/lib/golang/src/runtime/proc.go:1761
runtime.stopm()
    /usr/lib/golang/src/runtime/proc.go:2782 +0x8c fp=0x7fd0355ffaf0 sp=0x7fd0355ffac0 pc=0x55b74ad96d4c
runtime.findRunnable()
    /usr/lib/golang/src/runtime/proc.go:3512 +0xd5f fp=0x7fd0355ffc68 sp=0x7fd0355ffaf0 pc=0x55b74ad988bf
runtime.schedule()
    /usr/lib/golang/src/runtime/proc.go:3868 +0xb1 fp=0x7fd0355ffca0 sp=0x7fd0355ffc68 pc=0x55b74ad99991
runtime.park_m(0xc000582c40)
    /usr/lib/golang/src/runtime/proc.go:4036 +0x1ec fp=0x7fd0355ffcf8 sp=0x7fd0355ffca0 pc=0x55b74ad99f6c
runtime.mcall()
    /usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0x7fd0355ffd10 sp=0x7fd0355ffcf8 pc=0x55b74adc36d0
rax    0xca
rbx    0x0
rcx    0x55b74adc7503
rdx    0x0
rdi    0xc000580148
rsi    0x80
rbp    0x7fd0355ffa78
rsp    0x7fd0355ffa30
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x182
r13    0x1
r14    0xc0005821c0
r15    0x7
rip    0x55b74adc7501
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x55b74adc7501 m=9 sigcode=0

goroutine 0 gp=0xc00038ee00 m=9 mp=0xc0004d0808 [idle]:
runtime.futex(0xc0004d0948, 0x80, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7fd034bffa78 sp=0x7fd034bffa70 pc=0x55b74adc7501
runtime.futexsleep(0x7fd034bffaf0?, 0x4ad63210?, 0x55b74ad9ff87?)
    /usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0x7fd034bffac8 sp=0x7fd034bffa78 pc=0x55b74ad8c030
runtime.notesleep(0xc0004d0948)
    /usr/lib/golang/src/runtime/lock_futex.go:170 +0x87 fp=0x7fd034bffb00 sp=0x7fd034bffac8 pc=0x55b74ad633a7
runtime.mPark(...)
    /usr/lib/golang/src/runtime/proc.go:1761
runtime.stopm()
    /usr/lib/golang/src/runtime/proc.go:2782 +0x8c fp=0x7fd034bffb30 sp=0x7fd034bffb00 pc=0x55b74ad96d4c
runtime.findRunnable()
    /usr/lib/golang/src/runtime/proc.go:3512 +0xd5f fp=0x7fd034bffca8 sp=0x7fd034bffb30 pc=0x55b74ad988bf
runtime.schedule()
    /usr/lib/golang/src/runtime/proc.go:3868 +0xb1 fp=0x7fd034bffce0 sp=0x7fd034bffca8 pc=0x55b74ad99991
runtime.goexit0(0xc00051a1c0?)
    /usr/lib/golang/src/runtime/proc.go:4181 +0x18 fp=0x7fd034bffcf8 sp=0x7fd034bffce0 pc=0x55b74ad9a838
runtime.mcall()
    /usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0x7fd034bffd10 sp=0x7fd034bffcf8 pc=0x55b74adc36d0
rax    0xca
rbx    0x0
rcx    0x55b74adc7503
rdx    0x0
rdi    0xc0004d0948
rsi    0x80
rbp    0x7fd034bffab8
rsp    0x7fd034bffa70
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x19
r13    0x1
r14    0xc00038ee00
r15    0x3
rip    0x55b74adc7501
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x55b74adc7501 m=10 sigcode=0

goroutine 0 gp=0xc0006421c0 m=10 mp=0xc000640008 [idle]:
runtime.futex(0xc000640148, 0x80, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7fd02bdffa78 sp=0x7fd02bdffa70 pc=0x55b74adc7501
runtime.futexsleep(0x7fd02bdffaf0?, 0x4ad63210?, 0x55b74ad9ff87?)
    /usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0x7fd02bdffac8 sp=0x7fd02bdffa78 pc=0x55b74ad8c030
runtime.notesleep(0xc000640148)
    /usr/lib/golang/src/runtime/lock_futex.go:170 +0x87 fp=0x7fd02bdffb00 sp=0x7fd02bdffac8 pc=0x55b74ad633a7
runtime.mPark(...)
    /usr/lib/golang/src/runtime/proc.go:1761
runtime.stopm()
    /usr/lib/golang/src/runtime/proc.go:2782 +0x8c fp=0x7fd02bdffb30 sp=0x7fd02bdffb00 pc=0x55b74ad96d4c
runtime.findRunnable()
    /usr/lib/golang/src/runtime/proc.go:3512 +0xd5f fp=0x7fd02bdffca8 sp=0x7fd02bdffb30 pc=0x55b74ad988bf
runtime.schedule()
    /usr/lib/golang/src/runtime/proc.go:3868 +0xb1 fp=0x7fd02bdffce0 sp=0x7fd02bdffca8 pc=0x55b74ad99991
runtime.goexit0(0xc00063f500?)
    /usr/lib/golang/src/runtime/proc.go:4181 +0x18 fp=0x7fd02bdffcf8 sp=0x7fd02bdffce0 pc=0x55b74ad9a838
runtime.mcall()
    /usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0x7fd02bdffd10 sp=0x7fd02bdffcf8 pc=0x55b74adc36d0
rax    0xca
rbx    0x0
rcx    0x55b74adc7503
rdx    0x0
rdi    0xc000640148
rsi    0x80
rbp    0x7fd02bdffab8
rsp    0x7fd02bdffa70
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x5
r13    0x1
r14    0xc0006421c0
r15    0x5
rip    0x55b74adc7501
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x55b74adc7501 m=11 sigcode=0

goroutine 0 gp=0xc0000d0c40 m=11 mp=0xc000580808 [idle]:
runtime.futex(0xc000580948, 0x80, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7fd02b3ffa38 sp=0x7fd02b3ffa30 pc=0x55b74adc7501
runtime.futexsleep(0x7fd02b3ffab0?, 0x4ad63210?, 0x0?)
    /usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0x7fd02b3ffa88 sp=0x7fd02b3ffa38 pc=0x55b74ad8c030
runtime.notesleep(0xc000580948)
    /usr/lib/golang/src/runtime/lock_futex.go:170 +0x87 fp=0x7fd02b3ffac0 sp=0x7fd02b3ffa88 pc=0x55b74ad633a7
runtime.mPark(...)
    /usr/lib/golang/src/runtime/proc.go:1761
runtime.stopm()
    /usr/lib/golang/src/runtime/proc.go:2782 +0x8c fp=0x7fd02b3ffaf0 sp=0x7fd02b3ffac0 pc=0x55b74ad96d4c
runtime.findRunnable()
    /usr/lib/golang/src/runtime/proc.go:3512 +0xd5f fp=0x7fd02b3ffc68 sp=0x7fd02b3ffaf0 pc=0x55b74ad988bf
runtime.schedule()
    /usr/lib/golang/src/runtime/proc.go:3868 +0xb1 fp=0x7fd02b3ffca0 sp=0x7fd02b3ffc68 pc=0x55b74ad99991
runtime.park_m(0xc000502700)
    /usr/lib/golang/src/runtime/proc.go:4036 +0x1ec fp=0x7fd02b3ffcf8 sp=0x7fd02b3ffca0 pc=0x55b74ad99f6c
runtime.mcall()
    /usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0x7fd02b3ffd10 sp=0x7fd02b3ffcf8 pc=0x55b74adc36d0
rax    0xca
rbx    0x0
rcx    0x55b74adc7503
rdx    0x0
rdi    0xc000580948
rsi    0x80
rbp    0x7fd02b3ffa78
rsp    0x7fd02b3ffa30
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x7fd02b3ffad0
r13    0x1
r14    0xc0000d0c40
r15    0x7
rip    0x55b74adc7501
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x55b74adc7501 m=12 sigcode=0

goroutine 0 gp=0xc00038e380 m=12 mp=0xc000640808 [idle]:
runtime.futex(0xc000640948, 0x80, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7fd02a9ffa38 sp=0x7fd02a9ffa30 pc=0x55b74adc7501
runtime.futexsleep(0x7fd02a9ffab0?, 0x4ad63210?, 0x55b74ad9ff87?)
    /usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0x7fd02a9ffa88 sp=0x7fd02a9ffa38 pc=0x55b74ad8c030
runtime.notesleep(0xc000640948)
    /usr/lib/golang/src/runtime/lock_futex.go:170 +0x87 fp=0x7fd02a9ffac0 sp=0x7fd02a9ffa88 pc=0x55b74ad633a7
runtime.mPark(...)
    /usr/lib/golang/src/runtime/proc.go:1761
runtime.stopm()
    /usr/lib/golang/src/runtime/proc.go:2782 +0x8c fp=0x7fd02a9ffaf0 sp=0x7fd02a9ffac0 pc=0x55b74ad96d4c
runtime.findRunnable()
    /usr/lib/golang/src/runtime/proc.go:3512 +0xd5f fp=0x7fd02a9ffc68 sp=0x7fd02a9ffaf0 pc=0x55b74ad988bf
runtime.schedule()
    /usr/lib/golang/src/runtime/proc.go:3868 +0xb1 fp=0x7fd02a9ffca0 sp=0x7fd02a9ffc68 pc=0x55b74ad99991
runtime.park_m(0xc00038f6c0)
    /usr/lib/golang/src/runtime/proc.go:4036 +0x1ec fp=0x7fd02a9ffcf8 sp=0x7fd02a9ffca0 pc=0x55b74ad99f6c
runtime.mcall()
    /usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0x7fd02a9ffd10 sp=0x7fd02a9ffcf8 pc=0x55b74adc36d0
rax    0xca
rbx    0x0
rcx    0x55b74adc7503
rdx    0x0
rdi    0xc000640948
rsi    0x80
rbp    0x7fd02a9ffa78
rsp    0x7fd02a9ffa30
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x6
r13    0x1
r14    0xc00038e380
r15    0xd
rip    0x55b74adc7501
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x55b74adc7501 m=13 sigcode=0

goroutine 0 gp=0xc00038f180 m=13 mp=0xc000641008 [idle]:
runtime.futex(0xc000641148, 0x80, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7fd029fffa38 sp=0x7fd029fffa30 pc=0x55b74adc7501
runtime.futexsleep(0x7fd029fffab0?, 0x4ad63210?, 0x129fffae0?)
    /usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0x7fd029fffa88 sp=0x7fd029fffa38 pc=0x55b74ad8c030
runtime.notesleep(0xc000641148)
    /usr/lib/golang/src/runtime/lock_futex.go:170 +0x87 fp=0x7fd029fffac0 sp=0x7fd029fffa88 pc=0x55b74ad633a7
runtime.mPark(...)
    /usr/lib/golang/src/runtime/proc.go:1761
runtime.stopm()
    /usr/lib/golang/src/runtime/proc.go:2782 +0x8c fp=0x7fd029fffaf0 sp=0x7fd029fffac0 pc=0x55b74ad96d4c
runtime.findRunnable()
    /usr/lib/golang/src/runtime/proc.go:3512 +0xd5f fp=0x7fd029fffc68 sp=0x7fd029fffaf0 pc=0x55b74ad988bf
runtime.schedule()
    /usr/lib/golang/src/runtime/proc.go:3868 +0xb1 fp=0x7fd029fffca0 sp=0x7fd029fffc68 pc=0x55b74ad99991
runtime.park_m(0xc000502700)
    /usr/lib/golang/src/runtime/proc.go:4036 +0x1ec fp=0x7fd029fffcf8 sp=0x7fd029fffca0 pc=0x55b74ad99f6c
runtime.mcall()
    /usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0x7fd029fffd10 sp=0x7fd029fffcf8 pc=0x55b74adc36d0
rax    0xca
rbx    0x0
rcx    0x55b74adc7503
rdx    0x0
rdi    0xc000641148
rsi    0x80
rbp    0x7fd029fffa78
rsp    0x7fd029fffa30
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x6
r13    0x1
r14    0xc00038f180
r15    0x7
rip    0x55b74adc7501
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x55b74adc7501 m=14 sigcode=0

goroutine 0 gp=0xc000582540 m=14 mp=0xc000641808 [idle]:
runtime.futex(0xc000641948, 0x80, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7fd0295ffa38 sp=0x7fd0295ffa30 pc=0x55b74adc7501
runtime.futexsleep(0x7fd0295ffab0?, 0x4ad63210?, 0x55b74ad9ff87?)
    /usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0x7fd0295ffa88 sp=0x7fd0295ffa38 pc=0x55b74ad8c030
runtime.notesleep(0xc000641948)
    /usr/lib/golang/src/runtime/lock_futex.go:170 +0x87 fp=0x7fd0295ffac0 sp=0x7fd0295ffa88 pc=0x55b74ad633a7
runtime.mPark(...)
    /usr/lib/golang/src/runtime/proc.go:1761
runtime.stopm()
    /usr/lib/golang/src/runtime/proc.go:2782 +0x8c fp=0x7fd0295ffaf0 sp=0x7fd0295ffac0 pc=0x55b74ad96d4c
runtime.findRunnable()
    /usr/lib/golang/src/runtime/proc.go:3512 +0xd5f fp=0x7fd0295ffc68 sp=0x7fd0295ffaf0 pc=0x55b74ad988bf
runtime.schedule()
    /usr/lib/golang/src/runtime/proc.go:3868 +0xb1 fp=0x7fd0295ffca0 sp=0x7fd0295ffc68 pc=0x55b74ad99991
runtime.park_m(0xc000582e00)
    /usr/lib/golang/src/runtime/proc.go:4036 +0x1ec fp=0x7fd0295ffcf8 sp=0x7fd0295ffca0 pc=0x55b74ad99f6c
runtime.mcall()
    /usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0x7fd0295ffd10 sp=0x7fd0295ffcf8 pc=0x55b74adc36d0
rax    0xca
rbx    0x0
rcx    0x55b74adc7503
rdx    0x0
rdi    0xc000641948
rsi    0x80
rbp    0x7fd0295ffa78
rsp    0x7fd0295ffa30
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x7fd0295ffaa8
r13    0x0
r14    0xc000582540
r15    0xf
rip    0x55b74adc7501
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x55b74adc7501 m=15 sigcode=0

goroutine 0 gp=0xc000642540 m=15 mp=0xc000581008 [idle]:
runtime.futex(0xc000581148, 0x80, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7fd028bffa78 sp=0x7fd028bffa70 pc=0x55b74adc7501
runtime.futexsleep(0x7fd028bffaf0?, 0x4ad63210?, 0x55b74ad9ff87?)
    /usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0x7fd028bffac8 sp=0x7fd028bffa78 pc=0x55b74ad8c030
runtime.notesleep(0xc000581148)
    /usr/lib/golang/src/runtime/lock_futex.go:170 +0x87 fp=0x7fd028bffb00 sp=0x7fd028bffac8 pc=0x55b74ad633a7
runtime.mPark(...)
    /usr/lib/golang/src/runtime/proc.go:1761
runtime.stopm()
    /usr/lib/golang/src/runtime/proc.go:2782 +0x8c fp=0x7fd028bffb30 sp=0x7fd028bffb00 pc=0x55b74ad96d4c
runtime.findRunnable()
    /usr/lib/golang/src/runtime/proc.go:3512 +0xd5f fp=0x7fd028bffca8 sp=0x7fd028bffb30 pc=0x55b74ad988bf
runtime.schedule()
    /usr/lib/golang/src/runtime/proc.go:3868 +0xb1 fp=0x7fd028bffce0 sp=0x7fd028bffca8 pc=0x55b74ad99991
runtime.goexit0(0xc00111aa80?)
    /usr/lib/golang/src/runtime/proc.go:4181 +0x18 fp=0x7fd028bffcf8 sp=0x7fd028bffce0 pc=0x55b74ad9a838
runtime.mcall()
    /usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0x7fd028bffd10 sp=0x7fd028bffcf8 pc=0x55b74adc36d0
rax    0xca
rbx    0x0
rcx    0x55b74adc7503
rdx    0x0
rdi    0xc000581148
rsi    0x80
rbp    0x7fd028bffab8
rsp    0x7fd028bffa70
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x19
r13    0x1
r14    0xc000642540
r15    0x7
rip    0x55b74adc7501
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x55b74adc7501 m=16 sigcode=0

goroutine 0 gp=0xc0006b81c0 m=16 mp=0xc000581808 [idle]:
runtime.futex(0xc000581948, 0x80, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7fd01fdffa38 sp=0x7fd01fdffa30 pc=0x55b74adc7501
runtime.futexsleep(0x7fd01fdffab0?, 0x4ad63210?, 0x55b74ad9ff87?)
    /usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0x7fd01fdffa88 sp=0x7fd01fdffa38 pc=0x55b74ad8c030
runtime.notesleep(0xc000581948)
    /usr/lib/golang/src/runtime/lock_futex.go:170 +0x87 fp=0x7fd01fdffac0 sp=0x7fd01fdffa88 pc=0x55b74ad633a7
runtime.mPark(...)
    /usr/lib/golang/src/runtime/proc.go:1761
runtime.stopm()
    /usr/lib/golang/src/runtime/proc.go:2782 +0x8c fp=0x7fd01fdffaf0 sp=0x7fd01fdffac0 pc=0x55b74ad96d4c
runtime.findRunnable()
    /usr/lib/golang/src/runtime/proc.go:3512 +0xd5f fp=0x7fd01fdffc68 sp=0x7fd01fdffaf0 pc=0x55b74ad988bf
runtime.schedule()
    /usr/lib/golang/src/runtime/proc.go:3868 +0xb1 fp=0x7fd01fdffca0 sp=0x7fd01fdffc68 pc=0x55b74ad99991
runtime.park_m(0xc000502700)
    /usr/lib/golang/src/runtime/proc.go:4036 +0x1ec fp=0x7fd01fdffcf8 sp=0x7fd01fdffca0 pc=0x55b74ad99f6c
runtime.mcall()
    /usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0x7fd01fdffd10 sp=0x7fd01fdffcf8 pc=0x55b74adc36d0
rax    0xca
rbx    0x0
rcx    0x55b74adc7503
rdx    0x0
rdi    0xc000581948
rsi    0x80
rbp    0x7fd01fdffa78
rsp    0x7fd01fdffa30
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x19
r13    0x1
r14    0xc0006b81c0
r15    0x1
rip    0x55b74adc7501
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x55b74adc7501 m=17 sigcode=0

goroutine 0 gp=0xc0000fc1c0 m=17 mp=0xc0004d1808 [idle]:
runtime.futex(0xc0004d1948, 0x80, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7fd01f3ffa38 sp=0x7fd01f3ffa30 pc=0x55b74adc7501
runtime.futexsleep(0x7fd01f3ffab0?, 0x4ad63210?, 0x55b74ad9ff87?)
    /usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0x7fd01f3ffa88 sp=0x7fd01f3ffa38 pc=0x55b74ad8c030
runtime.notesleep(0xc0004d1948)
    /usr/lib/golang/src/runtime/lock_futex.go:170 +0x87 fp=0x7fd01f3ffac0 sp=0x7fd01f3ffa88 pc=0x55b74ad633a7
runtime.mPark(...)
    /usr/lib/golang/src/runtime/proc.go:1761
runtime.stopm()
    /usr/lib/golang/src/runtime/proc.go:2782 +0x8c fp=0x7fd01f3ffaf0 sp=0x7fd01f3ffac0 pc=0x55b74ad96d4c
runtime.findRunnable()
    /usr/lib/golang/src/runtime/proc.go:3512 +0xd5f fp=0x7fd01f3ffc68 sp=0x7fd01f3ffaf0 pc=0x55b74ad988bf
runtime.schedule()
    /usr/lib/golang/src/runtime/proc.go:3868 +0xb1 fp=0x7fd01f3ffca0 sp=0x7fd01f3ffc68 pc=0x55b74ad99991
runtime.park_m(0xc000582c40)
    /usr/lib/golang/src/runtime/proc.go:4036 +0x1ec fp=0x7fd01f3ffcf8 sp=0x7fd01f3ffca0 pc=0x55b74ad99f6c
runtime.mcall()
    /usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0x7fd01f3ffd10 sp=0x7fd01f3ffcf8 pc=0x55b74adc36d0
rax    0xca
rbx    0x0
rcx    0x55b74adc7503
rdx    0x0
rdi    0xc0004d1948
rsi    0x80
rbp    0x7fd01f3ffa78
rsp    0x7fd01f3ffa30
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0xc000070508
r13    0x1
r14    0xc0000fc1c0
r15    0xf
rip    0x55b74adc7501
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x55b74adc7501 m=18 sigcode=0

goroutine 0 gp=0xc0006b8540 m=18 mp=0xc0000c6808 [idle]:
runtime.futex(0xc0000c6948, 0x80, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7fd01e9ffa78 sp=0x7fd01e9ffa70 pc=0x55b74adc7501
runtime.futexsleep(0x7fd01e9ffaf0?, 0x4ad63210?, 0x55b74ad9ff87?)
    /usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0x7fd01e9ffac8 sp=0x7fd01e9ffa78 pc=0x55b74ad8c030
runtime.notesleep(0xc0000c6948)
    /usr/lib/golang/src/runtime/lock_futex.go:170 +0x87 fp=0x7fd01e9ffb00 sp=0x7fd01e9ffac8 pc=0x55b74ad633a7
runtime.mPark(...)
    /usr/lib/golang/src/runtime/proc.go:1761
runtime.stopm()
    /usr/lib/golang/src/runtime/proc.go:2782 +0x8c fp=0x7fd01e9ffb30 sp=0x7fd01e9ffb00 pc=0x55b74ad96d4c
runtime.findRunnable()
    /usr/lib/golang/src/runtime/proc.go:3512 +0xd5f fp=0x7fd01e9ffca8 sp=0x7fd01e9ffb30 pc=0x55b74ad988bf
runtime.schedule()
    /usr/lib/golang/src/runtime/proc.go:3868 +0xb1 fp=0x7fd01e9ffce0 sp=0x7fd01e9ffca8 pc=0x55b74ad99991
runtime.goexit0(0xc0000fc540?)
    /usr/lib/golang/src/runtime/proc.go:4181 +0x18 fp=0x7fd01e9ffcf8 sp=0x7fd01e9ffce0 pc=0x55b74ad9a838
runtime.mcall()
    /usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0x7fd01e9ffd10 sp=0x7fd01e9ffcf8 pc=0x55b74adc36d0
rax    0xca
rbx    0x0
rcx    0x55b74adc7503
rdx    0x0
rdi    0xc0000c6948
rsi    0x80
rbp    0x7fd01e9ffab8
rsp    0x7fd01e9ffa70
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x19
r13    0x1
r14    0xc0006b8540
r15    0xf
rip    0x55b74adc7501
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x55b74adc7501 m=19 sigcode=0

goroutine 0 gp=0xc001088700 m=19 mp=0xc000101008 [idle]:
runtime.futex(0xc000101148, 0x80, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7fd01dfffa38 sp=0x7fd01dfffa30 pc=0x55b74adc7501
runtime.futexsleep(0x7fd01dfffab0?, 0x4ad63210?, 0x55b74ad9ff87?)
    /usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0x7fd01dfffa88 sp=0x7fd01dfffa38 pc=0x55b74ad8c030
runtime.notesleep(0xc000101148)
    /usr/lib/golang/src/runtime/lock_futex.go:170 +0x87 fp=0x7fd01dfffac0 sp=0x7fd01dfffa88 pc=0x55b74ad633a7
runtime.mPark(...)
    /usr/lib/golang/src/runtime/proc.go:1761
runtime.stopm()
    /usr/lib/golang/src/runtime/proc.go:2782 +0x8c fp=0x7fd01dfffaf0 sp=0x7fd01dfffac0 pc=0x55b74ad96d4c
runtime.findRunnable()
    /usr/lib/golang/src/runtime/proc.go:3512 +0xd5f fp=0x7fd01dfffc68 sp=0x7fd01dfffaf0 pc=0x55b74ad988bf
runtime.schedule()
    /usr/lib/golang/src/runtime/proc.go:3868 +0xb1 fp=0x7fd01dfffca0 sp=0x7fd01dfffc68 pc=0x55b74ad99991
runtime.park_m(0xc000502700)
    /usr/lib/golang/src/runtime/proc.go:4036 +0x1ec fp=0x7fd01dfffcf8 sp=0x7fd01dfffca0 pc=0x55b74ad99f6c
runtime.mcall()
    /usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0x7fd01dfffd10 sp=0x7fd01dfffcf8 pc=0x55b74adc36d0
rax    0xca
rbx    0x0
rcx    0x55b74adc7503
rdx    0x0
rdi    0xc000101148
rsi    0x80
rbp    0x7fd01dfffa78
rsp    0x7fd01dfffa30
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x19
r13    0x1
r14    0xc001088700
r15    0x9
rip    0x55b74adc7501
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
Aborted (core dumped)
mtrmac commented 6 months ago

That looks very confusing. 17 different backtraces for “goroutine 0”, several different kinds of output interleaves. Is that a single copy?

PC=PC=0x55b74ad5904e0x5603dd90d04c m= m=00 sigcode= sigcode=128128

is clearly from 2 different processes. containers/storage/pkg/unshare.MaybeReexecUsingUserNamespace is creating a child, I guess we see 2 interleaved backtraces.

debarshiray commented 6 months ago

I don't think I fumbled the paste by pressing ctrl+v or something similar. I had copied the logs from a freshly spawned terminal without any previous skopeo copy invocations in the scroll back to avoid any mishaps.

debarshiray commented 6 months ago

For various odd requirements and quirks, the Toolbx test suite specifically uploads an older Fedora based image to simplify the implementation. That said, I reproduced it also with the fedora:35, fedora:36, fedora:37, fedora:38 and fedora:39 images on a Fedora 40 host. The fedora:40 image didn't get stuck on a Fedora 40 host.

This part about only older images getting stuck isn't true. I can now reproduce a fedora:40 image getting stuck. It was probably some timing issue.

However, the part about only Fedora 40 and above hosts showing this problem is likely still true because of the observed behaviour of the Toolbx CI.

mtrmac commented 6 months ago

I don't think I fumbled the paste by pressing ctrl+v or something similar.

No, I think it’s the process structure. Sending a control character sends the signal to every process in the process group, and all of them output the traceback at once.

Could you try kill -QUIT the processes individually? Maybe starting with the innermost child.

debarshiray commented 6 months ago

Okay! I did a kill -QUIT <PID> against the skopeo-in-a-user-namespace process, and got:

$ GOTRACEBACK=all skopeo copy \
      containers-storage:registry.fedoraproject.org/fedora:34 \
      docker://localhost:50000/fedora:34
Getting image source signatures
Copying blob 883e787c00d4 done   | 
SIGQUIT: quit
PC=0x562794f7d421 m=0 sigcode=0

goroutine 0 gp=0x5627968a1a80 m=0 mp=0x5627968a2ba0 [idle]:
runtime.futex(0x5627968a2ce0, 0x80, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7fffd495bce0 sp=0x7fffd495bcd8 pc=0x562794f7d421
runtime.futexsleep(0x7fffd495bd58?, 0x94f19150?, 0x562794f55ee7?)
    /usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0x7fffd495bd30 sp=0x7fffd495bce0 pc=0x562794f41f90
runtime.notesleep(0x5627968a2ce0)
    /usr/lib/golang/src/runtime/lock_futex.go:170 +0x87 fp=0x7fffd495bd68 sp=0x7fffd495bd30 pc=0x562794f192e7
runtime.mPark(...)
    /usr/lib/golang/src/runtime/proc.go:1761
runtime.stopm()
    /usr/lib/golang/src/runtime/proc.go:2782 +0x8c fp=0x7fffd495bd98 sp=0x7fffd495bd68 pc=0x562794f4ccac
runtime.findRunnable()
    /usr/lib/golang/src/runtime/proc.go:3512 +0xd5f fp=0x7fffd495bf10 sp=0x7fffd495bd98 pc=0x562794f4e81f
runtime.schedule()
    /usr/lib/golang/src/runtime/proc.go:3868 +0xb1 fp=0x7fffd495bf48 sp=0x7fffd495bf10 pc=0x562794f4f8f1
runtime.goexit0(0xc000583500?)
    /usr/lib/golang/src/runtime/proc.go:4181 +0x18 fp=0x7fffd495bf60 sp=0x7fffd495bf48 pc=0x562794f50798
runtime.mcall()
    /usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0x7fffd495bf78 sp=0x7fffd495bf60 pc=0x562794f795f0

goroutine 1 gp=0xc0000061c0 m=nil [semacquire]:
runtime.gopark(0x562794f1abfb?, 0x0?, 0x20?, 0x9c?, 0xc000494960?)
    /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc00048c588 sp=0xc00048c568 pc=0x562794f485ce
runtime.goparkunlock(...)
    /usr/lib/golang/src/runtime/proc.go:408
runtime.semacquire1(0xc000135778, 0x0, 0x1, 0x0, 0x12)
    /usr/lib/golang/src/runtime/sema.go:160 +0x22c fp=0xc00048c5f0 sp=0xc00048c588 pc=0x562794f5aacc
sync.runtime_Semacquire(0x562794f51ccb?)
    /usr/lib/golang/src/runtime/sema.go:62 +0x25 fp=0xc00048c628 sp=0xc00048c5f0 pc=0x562794f777a5
sync.(*WaitGroup).Wait(0x5627957c6454?)
    /usr/lib/golang/src/sync/waitgroup.go:116 +0x48 fp=0xc00048c650 sp=0xc00048c628 pc=0x562794f9abc8
github.com/containers/image/v5/copy.(*imageCopier).copyLayers.func2.deferwrap2()
    /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/image/v5/copy/single.go:493 +0x25 fp=0xc00048c668 sp=0xc00048c650 pc=0x5627957c6885
github.com/containers/image/v5/copy.(*imageCopier).copyLayers.func2(0xc000320380, 0xc000135770, {0xc000809b00, 0x1, 0x562794f2327a?}, 0xc0006cc910, {0x562796019d90, 0x5627969046a0}, 0xc000809b60, 0xc0006cca18)
    /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/image/v5/copy/single.go:506 +0x403 fp=0xc00048c7f0 sp=0xc00048c668 pc=0x5627957c6703
github.com/containers/image/v5/copy.(*imageCopier).copyLayers(0xc000320380, {0x562796019d90, 0x5627969046a0})
    /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/image/v5/copy/single.go:507 +0x479 fp=0xc00048cbd0 sp=0xc00048c7f0 pc=0x5627957c5c19
github.com/containers/image/v5/copy.(*copier).copySingleImage(0xc000560fa0, {0x562796019d90, 0x5627969046a0}, 0xc000809500, 0x0, {0x4f?, 0x0?, 0x0?})
    /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/image/v5/copy/single.go:210 +0xf05 fp=0xc00048d1f0 sp=0xc00048cbd0 pc=0x5627957c31a5
github.com/containers/image/v5/copy.Image({0x562796019d90, 0x5627969046a0}, 0xc00015ca80, {0x5627960200c0, 0xc00015c018}, {0x5627960205a0, 0xc0000ba180}, 0xc0000061c0?)
    /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/image/v5/copy/copy.go:291 +0x1225 fp=0xc00048d620 sp=0xc00048d1f0 pc=0x5627957b6725
main.(*copyOptions).run.func3()
    /builddir/build/BUILD/skopeo-1.15.0/cmd/skopeo/copy.go:286 +0x3de fp=0xc00048d7a0 sp=0xc00048d620 pc=0x562795a5807e
github.com/containers/common/pkg/retry.IfNecessary({0x562796019d90, 0x5627969046a0}, 0xc00048db30, 0xc00015c7c8)
    /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/common/pkg/retry/retry.go:41 +0x57 fp=0xc00048d888 sp=0xc00048d7a0 pc=0x5627952f0577
main.(*copyOptions).run(0xc00017e280, {0xc0004e8940, 0x2, 0x2}, {0x56279600d848, 0xc000092070})
    /builddir/build/BUILD/skopeo-1.15.0/cmd/skopeo/copy.go:285 +0x12b8 fp=0xc00048dc20 sp=0xc00048d888 pc=0x562795a57af8
main.(*copyOptions).run-fm({0xc0004e8940?, 0x0?, 0x0?}, {0x56279600d848?, 0xc000092070?})
    <autogenerated>:1 +0x45 fp=0xc00048dc60 sp=0xc00048dc20 pc=0x562795a72e65
main.copyCmd.commandAction.func1(0xc00015f208, {0xc0004e8940, 0x2, 0x2})
    /builddir/build/BUILD/skopeo-1.15.0/cmd/skopeo/utils.go:58 +0x57 fp=0xc00048dcb0 sp=0xc00048dc60 pc=0x562795a56697
github.com/spf13/cobra.(*Command).execute(0xc00015f208, {0xc0000400a0, 0x2, 0x2})
    /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/spf13/cobra/command.go:983 +0xaca fp=0xc00048de38 sp=0xc00048dcb0 pc=0x5627950fe2ca
github.com/spf13/cobra.(*Command).ExecuteC(0xc00015ef08)
    /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/spf13/cobra/command.go:1115 +0x3ff fp=0xc00048df10 sp=0xc00048de38 pc=0x5627950feb9f
github.com/spf13/cobra.(*Command).Execute(...)
    /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/spf13/cobra/command.go:1039
main.main()
    /builddir/build/BUILD/skopeo-1.15.0/cmd/skopeo/main.go:131 +0x56 fp=0xc00048df50 sp=0xc00048df10 pc=0x562795a60e96
runtime.main()
    /usr/lib/golang/src/runtime/proc.go:271 +0x29d fp=0xc00048dfe0 sp=0xc00048df50 pc=0x562794f4819d
runtime.goexit({})
    /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00048dfe8 sp=0xc00048dfe0 pc=0x562794f7b641

goroutine 2 gp=0xc000006c40 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc00008efa8 sp=0xc00008ef88 pc=0x562794f485ce
runtime.goparkunlock(...)
    /usr/lib/golang/src/runtime/proc.go:408
runtime.forcegchelper()
    /usr/lib/golang/src/runtime/proc.go:326 +0xb8 fp=0xc00008efe0 sp=0xc00008efa8 pc=0x562794f48458
runtime.goexit({})
    /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00008efe8 sp=0xc00008efe0 pc=0x562794f7b641
created by runtime.init.7 in goroutine 1
    /usr/lib/golang/src/runtime/proc.go:314 +0x1a

goroutine 3 gp=0xc000007180 m=nil [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc00008f780 sp=0xc00008f760 pc=0x562794f485ce
runtime.goparkunlock(...)
    /usr/lib/golang/src/runtime/proc.go:408
runtime.bgsweep(0xc0000b8000)
    /usr/lib/golang/src/runtime/mgcsweep.go:318 +0xdf fp=0xc00008f7c8 sp=0xc00008f780 pc=0x562794f3301f
runtime.gcenable.gowrap1()
    /usr/lib/golang/src/runtime/mgc.go:203 +0x25 fp=0xc00008f7e0 sp=0xc00008f7c8 pc=0x562794f27905
runtime.goexit({})
    /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00008f7e8 sp=0xc00008f7e0 pc=0x562794f7b641
created by runtime.gcenable in goroutine 1
    /usr/lib/golang/src/runtime/mgc.go:203 +0x66

goroutine 4 gp=0xc000007340 m=nil [GC scavenge wait]:
runtime.gopark(0xa1774?, 0x8c2e2a?, 0x0?, 0x0?, 0x0?)
    /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc00008ff78 sp=0xc00008ff58 pc=0x562794f485ce
runtime.goparkunlock(...)
    /usr/lib/golang/src/runtime/proc.go:408
runtime.(*scavengerState).park(0x56279689fda0)
    /usr/lib/golang/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00008ffa8 sp=0xc00008ff78 pc=0x562794f309c9
runtime.bgscavenge(0xc0000b8000)
    /usr/lib/golang/src/runtime/mgcscavenge.go:658 +0x59 fp=0xc00008ffc8 sp=0xc00008ffa8 pc=0x562794f30f79
runtime.gcenable.gowrap2()
    /usr/lib/golang/src/runtime/mgc.go:204 +0x25 fp=0xc00008ffe0 sp=0xc00008ffc8 pc=0x562794f278a5
runtime.goexit({})
    /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00008ffe8 sp=0xc00008ffe0 pc=0x562794f7b641
created by runtime.gcenable in goroutine 1
    /usr/lib/golang/src/runtime/mgc.go:204 +0xa5

goroutine 5 gp=0xc000007c00 m=nil [finalizer wait]:
runtime.gopark(0x0?, 0xc000012288?, 0xe0?, 0x0?, 0x1000000010?)
    /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc0000a1e20 sp=0xc0000a1e00 pc=0x562794f485ce
runtime.runfinq()
    /usr/lib/golang/src/runtime/mfinal.go:194 +0x107 fp=0xc0000a1fe0 sp=0xc0000a1e20 pc=0x562794f26947
runtime.goexit({})
    /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000a1fe8 sp=0xc0000a1fe0 pc=0x562794f7b641
created by runtime.createfing in goroutine 1
    /usr/lib/golang/src/runtime/mfinal.go:164 +0x3d

goroutine 6 gp=0xc0001dba40 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc000090750 sp=0xc000090730 pc=0x562794f485ce
runtime.gcBgMarkWorker()
    /usr/lib/golang/src/runtime/mgc.go:1310 +0xe5 fp=0xc0000907e0 sp=0xc000090750 pc=0x562794f29a05
runtime.goexit({})
    /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000907e8 sp=0xc0000907e0 pc=0x562794f7b641
created by runtime.gcBgMarkStartWorkers in goroutine 1
    /usr/lib/golang/src/runtime/mgc.go:1234 +0x1c

goroutine 18 gp=0xc000102700 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc00008a750 sp=0xc00008a730 pc=0x562794f485ce
runtime.gcBgMarkWorker()
    /usr/lib/golang/src/runtime/mgc.go:1310 +0xe5 fp=0xc00008a7e0 sp=0xc00008a750 pc=0x562794f29a05
runtime.goexit({})
    /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00008a7e8 sp=0xc00008a7e0 pc=0x562794f7b641
created by runtime.gcBgMarkStartWorkers in goroutine 1
    /usr/lib/golang/src/runtime/mgc.go:1234 +0x1c

goroutine 19 gp=0xc0001028c0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc00008af50 sp=0xc00008af30 pc=0x562794f485ce
runtime.gcBgMarkWorker()
    /usr/lib/golang/src/runtime/mgc.go:1310 +0xe5 fp=0xc00008afe0 sp=0xc00008af50 pc=0x562794f29a05
runtime.goexit({})
    /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00008afe8 sp=0xc00008afe0 pc=0x562794f7b641
created by runtime.gcBgMarkStartWorkers in goroutine 1
    /usr/lib/golang/src/runtime/mgc.go:1234 +0x1c

goroutine 7 gp=0xc0001dbc00 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc000090f50 sp=0xc000090f30 pc=0x562794f485ce
runtime.gcBgMarkWorker()
    /usr/lib/golang/src/runtime/mgc.go:1310 +0xe5 fp=0xc000090fe0 sp=0xc000090f50 pc=0x562794f29a05
runtime.goexit({})
    /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000090fe8 sp=0xc000090fe0 pc=0x562794f7b641
created by runtime.gcBgMarkStartWorkers in goroutine 1
    /usr/lib/golang/src/runtime/mgc.go:1234 +0x1c

goroutine 34 gp=0xc000582000 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc000588750 sp=0xc000588730 pc=0x562794f485ce
runtime.gcBgMarkWorker()
    /usr/lib/golang/src/runtime/mgc.go:1310 +0xe5 fp=0xc0005887e0 sp=0xc000588750 pc=0x562794f29a05
runtime.goexit({})
    /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0005887e8 sp=0xc0005887e0 pc=0x562794f7b641
created by runtime.gcBgMarkStartWorkers in goroutine 1
    /usr/lib/golang/src/runtime/mgc.go:1234 +0x1c

goroutine 8 gp=0xc0001dbdc0 m=nil [GC worker (idle)]:
runtime.gopark(0x132ed0f075a?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc000091750 sp=0xc000091730 pc=0x562794f485ce
runtime.gcBgMarkWorker()
    /usr/lib/golang/src/runtime/mgc.go:1310 +0xe5 fp=0xc0000917e0 sp=0xc000091750 pc=0x562794f29a05
runtime.goexit({})
    /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000917e8 sp=0xc0000917e0 pc=0x562794f7b641
created by runtime.gcBgMarkStartWorkers in goroutine 1
    /usr/lib/golang/src/runtime/mgc.go:1234 +0x1c

goroutine 35 gp=0xc0005821c0 m=nil [GC worker (idle)]:
runtime.gopark(0x132e9481d3b?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc000588f50 sp=0xc000588f30 pc=0x562794f485ce
runtime.gcBgMarkWorker()
    /usr/lib/golang/src/runtime/mgc.go:1310 +0xe5 fp=0xc000588fe0 sp=0xc000588f50 pc=0x562794f29a05
runtime.goexit({})
    /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000588fe8 sp=0xc000588fe0 pc=0x562794f7b641
created by runtime.gcBgMarkStartWorkers in goroutine 1
    /usr/lib/golang/src/runtime/mgc.go:1234 +0x1c

goroutine 20 gp=0xc000102a80 m=nil [GC worker (idle)]:
runtime.gopark(0x132f2e70649?, 0x1?, 0xf7?, 0x18?, 0x0?)
    /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc00008b750 sp=0xc00008b730 pc=0x562794f485ce
runtime.gcBgMarkWorker()
    /usr/lib/golang/src/runtime/mgc.go:1310 +0xe5 fp=0xc00008b7e0 sp=0xc00008b750 pc=0x562794f29a05
runtime.goexit({})
    /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00008b7e8 sp=0xc00008b7e0 pc=0x562794f7b641
created by runtime.gcBgMarkStartWorkers in goroutine 1
    /usr/lib/golang/src/runtime/mgc.go:1234 +0x1c

goroutine 36 gp=0xc000582380 m=nil [GC worker (idle)]:
runtime.gopark(0x132f2e5023a?, 0x1?, 0x80?, 0xb4?, 0x0?)
    /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc000589750 sp=0xc000589730 pc=0x562794f485ce
runtime.gcBgMarkWorker()
    /usr/lib/golang/src/runtime/mgc.go:1310 +0xe5 fp=0xc0005897e0 sp=0xc000589750 pc=0x562794f29a05
runtime.goexit({})
    /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0005897e8 sp=0xc0005897e0 pc=0x562794f7b641
created by runtime.gcBgMarkStartWorkers in goroutine 1
    /usr/lib/golang/src/runtime/mgc.go:1234 +0x1c

goroutine 21 gp=0xc000102c40 m=nil [GC worker (idle)]:
runtime.gopark(0x562796906be0?, 0x1?, 0x14?, 0xc1?, 0x0?)
    /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc00008bf50 sp=0xc00008bf30 pc=0x562794f485ce
runtime.gcBgMarkWorker()
    /usr/lib/golang/src/runtime/mgc.go:1310 +0xe5 fp=0xc00008bfe0 sp=0xc00008bf50 pc=0x562794f29a05
runtime.goexit({})
    /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00008bfe8 sp=0xc00008bfe0 pc=0x562794f7b641
created by runtime.gcBgMarkStartWorkers in goroutine 1
    /usr/lib/golang/src/runtime/mgc.go:1234 +0x1c

goroutine 9 gp=0xc00051a000 m=nil [GC worker (idle)]:
runtime.gopark(0x132f2e6f684?, 0x3?, 0xda?, 0x7f?, 0x0?)
    /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc000091f50 sp=0xc000091f30 pc=0x562794f485ce
runtime.gcBgMarkWorker()
    /usr/lib/golang/src/runtime/mgc.go:1310 +0xe5 fp=0xc000091fe0 sp=0xc000091f50 pc=0x562794f29a05
runtime.goexit({})
    /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000091fe8 sp=0xc000091fe0 pc=0x562794f7b641
created by runtime.gcBgMarkStartWorkers in goroutine 1
    /usr/lib/golang/src/runtime/mgc.go:1234 +0x1c

goroutine 37 gp=0xc000582540 m=nil [GC worker (idle)]:
runtime.gopark(0x132ed0f0529?, 0x1?, 0x99?, 0x53?, 0x0?)
    /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc000589f50 sp=0xc000589f30 pc=0x562794f485ce
runtime.gcBgMarkWorker()
    /usr/lib/golang/src/runtime/mgc.go:1310 +0xe5 fp=0xc000589fe0 sp=0xc000589f50 pc=0x562794f29a05
runtime.goexit({})
    /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000589fe8 sp=0xc000589fe0 pc=0x562794f7b641
created by runtime.gcBgMarkStartWorkers in goroutine 1
    /usr/lib/golang/src/runtime/mgc.go:1234 +0x1c

goroutine 38 gp=0xc000582700 m=nil [GC worker (idle)]:
runtime.gopark(0x132e9485bd0?, 0x1?, 0x81?, 0x2f?, 0x0?)
    /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc00058a750 sp=0xc00058a730 pc=0x562794f485ce
runtime.gcBgMarkWorker()
    /usr/lib/golang/src/runtime/mgc.go:1310 +0xe5 fp=0xc00058a7e0 sp=0xc00058a750 pc=0x562794f29a05
runtime.goexit({})
    /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00058a7e8 sp=0xc00058a7e0 pc=0x562794f7b641
created by runtime.gcBgMarkStartWorkers in goroutine 1
    /usr/lib/golang/src/runtime/mgc.go:1234 +0x1c

goroutine 39 gp=0xc0005828c0 m=nil [GC worker (idle)]:
runtime.gopark(0x132f2e71ef1?, 0x3?, 0x4f?, 0x43?, 0x0?)
    /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc00058af50 sp=0xc00058af30 pc=0x562794f485ce
runtime.gcBgMarkWorker()
    /usr/lib/golang/src/runtime/mgc.go:1310 +0xe5 fp=0xc00058afe0 sp=0xc00058af50 pc=0x562794f29a05
runtime.goexit({})
    /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00058afe8 sp=0xc00058afe0 pc=0x562794f7b641
created by runtime.gcBgMarkStartWorkers in goroutine 1
    /usr/lib/golang/src/runtime/mgc.go:1234 +0x1c

goroutine 10 gp=0xc00051a1c0 m=nil [GC worker (idle)]:
runtime.gopark(0x132ed0f0438?, 0xc0001160a0?, 0x1a?, 0xa?, 0x0?)
    /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc000584750 sp=0xc000584730 pc=0x562794f485ce
runtime.gcBgMarkWorker()
    /usr/lib/golang/src/runtime/mgc.go:1310 +0xe5 fp=0xc0005847e0 sp=0xc000584750 pc=0x562794f29a05
runtime.goexit({})
    /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0005847e8 sp=0xc0005847e0 pc=0x562794f7b641
created by runtime.gcBgMarkStartWorkers in goroutine 1
    /usr/lib/golang/src/runtime/mgc.go:1234 +0x1c

goroutine 22 gp=0xc000103180 m=nil [GC worker (idle)]:
runtime.gopark(0x132f2e6b7ef?, 0x1?, 0xf?, 0x7e?, 0x0?)
    /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc00008c750 sp=0xc00008c730 pc=0x562794f485ce
runtime.gcBgMarkWorker()
    /usr/lib/golang/src/runtime/mgc.go:1310 +0xe5 fp=0xc00008c7e0 sp=0xc00008c750 pc=0x562794f29a05
runtime.goexit({})
    /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00008c7e8 sp=0xc00008c7e0 pc=0x562794f7b641
created by runtime.gcBgMarkStartWorkers in goroutine 1
    /usr/lib/golang/src/runtime/mgc.go:1234 +0x1c

goroutine 32 gp=0xc00051b6c0 m=nil [IO wait]:
runtime.gopark(0xf3fb11572f309bc1?, 0x390fb9f16ff08804?, 0xc1?, 0x9b?, 0xb?)
    /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc0000a46f0 sp=0xc0000a46d0 pc=0x562794f485ce
runtime.netpollblock(0x562794fcfc18?, 0x94f10846?, 0x27?)
    /usr/lib/golang/src/runtime/netpoll.go:573 +0xf7 fp=0xc0000a4728 sp=0xc0000a46f0 pc=0x562794f412f7
internal/poll.runtime_pollWait(0x7f2268e46700, 0x72)
    /usr/lib/golang/src/runtime/netpoll.go:345 +0x85 fp=0xc0000a4748 sp=0xc0000a4728 pc=0x562794f75ce5
internal/poll.(*pollDesc).wait(0xc0002fe100?, 0xc000426800?, 0x0)
    /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0000a4770 sp=0xc0000a4748 pc=0x562794fecb47
internal/poll.(*pollDesc).waitRead(...)
    /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0002fe100, {0xc000426800, 0x800, 0x800})
    /usr/lib/golang/src/internal/poll/fd_unix.go:164 +0x27a fp=0xc0000a4808 sp=0xc0000a4770 pc=0x562794fede3a
net.(*netFD).Read(0xc0002fe100, {0xc000426800?, 0x562796906be0?, 0xc000490048?})
    /usr/lib/golang/src/net/fd_posix.go:55 +0x25 fp=0xc0000a4850 sp=0xc0000a4808 pc=0x562795082725
net.(*conn).Read(0xc00013a038, {0xc000426800?, 0xc0000a4930?, 0x562794f1ad65?})
    /usr/lib/golang/src/net/net.go:179 +0x45 fp=0xc0000a4898 sp=0xc0000a4850 pc=0x562795093885
net.(*TCPConn).Read(0xdeb?, {0xc000426800?, 0x1?, 0x100000001?})
    <autogenerated>:1 +0x25 fp=0xc0000a48c8 sp=0xc0000a4898 pc=0x5627950a61c5
crypto/tls.(*atLeastReader).Read(0xc000490048, {0xc000426800?, 0x0?, 0xc000490048?})
    /usr/lib/golang/src/crypto/tls/conn.go:806 +0x3b fp=0xc0000a4910 sp=0xc0000a48c8 pc=0x5627951f20fb
bytes.(*Buffer).ReadFrom(0xc0004e2630, {0x56279600f4c0, 0xc000490048})
    /usr/lib/golang/src/bytes/buffer.go:211 +0x98 fp=0xc0000a4968 sp=0xc0000a4910 pc=0x56279502fe78
crypto/tls.(*Conn).readFromUntil(0xc0004e2388, {0x56279600e400, 0xc00013a038}, 0xc0000a4978?)
    /usr/lib/golang/src/crypto/tls/conn.go:828 +0xde fp=0xc0000a49a0 sp=0xc0000a4968 pc=0x5627951f22de
crypto/tls.(*Conn).readRecordOrCCS(0xc0004e2388, 0x0)
    /usr/lib/golang/src/crypto/tls/conn.go:626 +0x3cf fp=0xc0000a4c20 sp=0xc0000a49a0 pc=0x5627951ef3ef
crypto/tls.(*Conn).readRecord(...)
    /usr/lib/golang/src/crypto/tls/conn.go:588
crypto/tls.(*Conn).Read(0xc0004e2388, {0xc000338000, 0x1000, 0x2?})
    /usr/lib/golang/src/crypto/tls/conn.go:1370 +0x156 fp=0xc0000a4c90 sp=0xc0000a4c20 pc=0x5627951f5c96
net/http.(*persistConn).Read(0xc0000fe6c0, {0xc000338000?, 0x562794f5a720?, 0xc0000a4f28?})
    /usr/lib/golang/src/net/http/transport.go:1977 +0x4a fp=0xc0000a4cf0 sp=0xc0000a4c90 pc=0x5627952c8aea
bufio.(*Reader).fill(0xc000808660)
    /usr/lib/golang/src/bufio/bufio.go:110 +0x103 fp=0xc0000a4d28 sp=0xc0000a4cf0 pc=0x5627950aaee3
bufio.(*Reader).Peek(0xc000808660, 0x1)
    /usr/lib/golang/src/bufio/bufio.go:148 +0x53 fp=0xc0000a4d48 sp=0xc0000a4d28 pc=0x5627950ab013
net/http.(*persistConn).readLoop(0xc0000fe6c0)
    /usr/lib/golang/src/net/http/transport.go:2141 +0x1b9 fp=0xc0000a4fc8 sp=0xc0000a4d48 pc=0x5627952c9879
net/http.(*Transport).dialConn.gowrap2()
    /usr/lib/golang/src/net/http/transport.go:1799 +0x25 fp=0xc0000a4fe0 sp=0xc0000a4fc8 pc=0x5627952c8005
runtime.goexit({})
    /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0000a4fe8 sp=0xc0000a4fe0 pc=0x562794f7b641
created by net/http.(*Transport).dialConn in goroutine 24
    /usr/lib/golang/src/net/http/transport.go:1799 +0x152f

goroutine 14 gp=0xc00051b880 m=nil [select]:
runtime.gopark(0xc000585f88?, 0x2?, 0x60?, 0x0?, 0xc000585f84?)
    /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc000585e30 sp=0xc000585e10 pc=0x562794f485ce
runtime.selectgo(0xc000585f88, 0xc000585f80, 0x0?, 0x0, 0x0?, 0x1)
    /usr/lib/golang/src/runtime/select.go:327 +0x725 fp=0xc000585f50 sp=0xc000585e30 pc=0x562794f59a85
database/sql.(*DB).connectionOpener(0xc0002f92b0, {0x56279601a068, 0xc0000c4e60})
    /usr/lib/golang/src/database/sql/sql.go:1246 +0x87 fp=0xc000585fb8 sp=0xc000585f50 pc=0x562795417887
database/sql.OpenDB.gowrap1()
    /usr/lib/golang/src/database/sql/sql.go:824 +0x28 fp=0xc000585fe0 sp=0xc000585fb8 pc=0x562795415ba8
runtime.goexit({})
    /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000585fe8 sp=0xc000585fe0 pc=0x562794f7b641
created by database/sql.OpenDB in goroutine 1
    /usr/lib/golang/src/database/sql/sql.go:824 +0x14c

goroutine 15 gp=0xc00051ba40 m=nil [select]:
runtime.gopark(0xc000586778?, 0x2?, 0x0?, 0x66?, 0xc000586734?)
    /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc0005865d8 sp=0xc0005865b8 pc=0x562794f485ce
runtime.selectgo(0xc000586778, 0xc000586730, 0x0?, 0x0, 0x0?, 0x1)
    /usr/lib/golang/src/runtime/select.go:327 +0x725 fp=0xc0005866f8 sp=0xc0005865d8 pc=0x562794f59a85
github.com/vbauerster/mpb/v8.(*pState).autoRefreshListener(0xc000561220, 0xc000494ba0)
    /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/vbauerster/mpb/v8/progress.go:324 +0x13f fp=0xc0005867c0 sp=0xc0005866f8 pc=0x5627957a88bf
github.com/vbauerster/mpb/v8.NewWithContext.gowrap2()
    /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/vbauerster/mpb/v8/progress.go:107 +0x25 fp=0xc0005867e0 sp=0xc0005867c0 pc=0x5627957a6b25
runtime.goexit({})
    /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0005867e8 sp=0xc0005867e0 pc=0x562794f7b641
created by github.com/vbauerster/mpb/v8.NewWithContext in goroutine 1
    /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/vbauerster/mpb/v8/progress.go:107 +0x395

goroutine 16 gp=0xc00051bc00 m=nil [select]:
runtime.gopark(0xc00027cf08?, 0x5?, 0x60?, 0x49?, 0xc00027ceb6?)
    /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc00027cd58 sp=0xc00027cd38 pc=0x562794f485ce
runtime.selectgo(0xc00027cf08, 0xc00027ceac, 0x0?, 0x0, 0x0?, 0x1)
    /usr/lib/golang/src/runtime/select.go:327 +0x725 fp=0xc00027ce78 sp=0xc00027cd58 pc=0x562794f59a85
github.com/vbauerster/mpb/v8.(*Progress).serve(0xc0003c5cc0, 0xc000561220, 0x0?)
    /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/vbauerster/mpb/v8/progress.go:274 +0x1dc fp=0xc00027cfb8 sp=0xc00027ce78 pc=0x5627957a821c
github.com/vbauerster/mpb/v8.NewWithContext.gowrap3()
    /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/vbauerster/mpb/v8/progress.go:114 +0x28 fp=0xc00027cfe0 sp=0xc00027cfb8 pc=0x5627957a6ac8
runtime.goexit({})
    /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00027cfe8 sp=0xc00027cfe0 pc=0x562794f7b641
created by github.com/vbauerster/mpb/v8.NewWithContext in goroutine 1
    /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/vbauerster/mpb/v8/progress.go:114 +0x4fa

goroutine 50 gp=0xc00051bdc0 m=nil [chan receive]:
runtime.gopark(0xc0008803c0?, 0xc00041c0c0?, 0xc8?, 0x2d?, 0x562794f19150?)
    /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc0004a2d90 sp=0xc0004a2d70 pc=0x562794f485ce
runtime.chanrecv(0xc000494960, 0xc0004a2f70, 0x1)
    /usr/lib/golang/src/runtime/chan.go:583 +0x3bf fp=0xc0004a2e08 sp=0xc0004a2d90 pc=0x562794f136ff
runtime.chanrecv2(0x56279601bb00?, 0xc00015c030?)
    /usr/lib/golang/src/runtime/chan.go:447 +0x12 fp=0xc0004a2e30 sp=0xc0004a2e08 pc=0x562794f13332
github.com/vbauerster/mpb/v8.heapManager.run(0xc000494960)
    /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/vbauerster/mpb/v8/heap_manager.go:47 +0x96 fp=0xc0004a2fc8 sp=0xc0004a2e30 pc=0x5627957a52f6
github.com/vbauerster/mpb/v8.NewWithContext.gowrap4()
    /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/vbauerster/mpb/v8/progress.go:115 +0x25 fp=0xc0004a2fe0 sp=0xc0004a2fc8 pc=0x5627957a6a65
runtime.goexit({})
    /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0004a2fe8 sp=0xc0004a2fe0 pc=0x562794f7b641
created by github.com/vbauerster/mpb/v8.NewWithContext in goroutine 1
    /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/vbauerster/mpb/v8/progress.go:115 +0x545

goroutine 51 gp=0xc000642e00 m=nil [select]:
runtime.gopark(0xc000057b20?, 0x6?, 0x10?, 0x78?, 0xc00005794c?)
    /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc0000577e8 sp=0xc0000577c8 pc=0x562794f485ce
runtime.selectgo(0xc000057b20, 0xc000057940, 0x562795b7eae2?, 0x0, 0xc00037fe30?, 0x1)
    /usr/lib/golang/src/runtime/select.go:327 +0x725 fp=0xc000057908 sp=0xc0000577e8 pc=0x562794f59a85
net/http.(*persistConn).roundTrip(0xc0000fe6c0, 0xc000280000)
    /usr/lib/golang/src/net/http/transport.go:2675 +0x979 fp=0xc000057b90 sp=0xc000057908 pc=0x5627952cc3d9
net/http.(*Transport).roundTrip(0xc000241900, 0xc000160b40)
    /usr/lib/golang/src/net/http/transport.go:608 +0x79a fp=0xc000057db8 sp=0xc000057b90 pc=0x5627952c029a
net/http.(*Transport).RoundTrip(0xf?, 0x56279600e0a0?)
    /usr/lib/golang/src/net/http/roundtrip.go:17 +0x13 fp=0xc000057dd8 sp=0xc000057db8 pc=0x5627952a5473
net/http.send(0xc000160b40, {0x56279600e0a0, 0xc000241900}, {0x101?, 0xc000100808?, 0x0?})
    /usr/lib/golang/src/net/http/client.go:259 +0x5e4 fp=0xc000057fb8 sp=0xc000057dd8 pc=0x562795262a64
net/http.(*Client).send(0xc00037fce0, 0xc000160b40, {0x140?, 0x55?, 0x0?})
    /usr/lib/golang/src/net/http/client.go:180 +0x98 fp=0xc000058030 sp=0xc000057fb8 pc=0x562795262318
net/http.(*Client).do(0xc00037fce0, 0xc000160b40)
    /usr/lib/golang/src/net/http/client.go:724 +0x8dc fp=0xc000058220 sp=0xc000058030 pc=0x56279526463c
net/http.(*Client).Do(...)
    /usr/lib/golang/src/net/http/client.go:590
github.com/containers/image/v5/docker.(*dockerClient).makeRequestToResolvedURLOnce(0xc000241680, {0x562796019d90, 0x5627969046a0}, {0x562795b7353a, 0x5}, 0xc00015a120, 0xc000058690, {0x56279600e040, 0xc0004e5320}, 0xffffffffffffffff, ...)
    /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/image/v5/docker/docker_client.go:632 +0x405 fp=0xc000058378 sp=0xc000058220 pc=0x562795881785
github.com/containers/image/v5/docker.(*dockerClient).makeRequestToResolvedURL(0xc000241680, {0x562796019d90, 0x5627969046a0}, {0x562795b7353a, 0x5}, 0xc00015a120, 0xc000058690, {0x56279600e040, 0xc0004e5320}, 0xffffffffffffffff, ...)
    /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/image/v5/docker/docker_client.go:559 +0x10d fp=0xc000058488 sp=0xc000058378 pc=0x562795880f6d
github.com/containers/image/v5/docker.(*dockerImageDestination).PutBlobWithOptions.func1({0x56279600d908, 0xc0000340c0}, 0xc0000588f0, 0xc000058900, 0xc000320100, {0x562796019d90, 0x5627969046a0}, 0xc0000588a8, 0xc000058bd0)
    /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/image/v5/docker/docker_image_dest.go:189 +0x294 fp=0xc000058710 sp=0xc000058488 pc=0x562795889f34
github.com/containers/image/v5/docker.(*dockerImageDestination).PutBlobWithOptions(0xc000320100, {0x562796019d90, 0x5627969046a0}, {0x56279600dce8?, 0xc0006100b0?}, {{0x562795d3d410, 0x0}, 0xffffffffffffffff, {0x0, 0x0, ...}, ...}, ...)
    /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/image/v5/docker/docker_image_dest.go:203 +0x858 fp=0xc000058bd0 sp=0xc000058710 pc=0x562795889398
github.com/containers/image/v5/copy.(*imageCopier).copyBlobFromStream(0xc000320380, {0x562796019d90, 0x5627969046a0}, {0x7f22224c8820, _}, {{0xc0006147d0, 0x47}, 0xaf2a800, {0x0, 0x0, ...}, ...}, ...)
    /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/image/v5/copy/blob.go:107 +0x96e fp=0xc000059138 sp=0xc000058bd0 pc=0x5627957b202e
github.com/containers/image/v5/copy.(*imageCopier).copyLayerFromStream(0xc000320380, {0x562796019d90, 0x5627969046a0}, {0x7f22224c8820, 0xc000418000}, {{0xc0006147d0, 0x47}, 0xaf2a800, {0x0, 0x0, ...}, ...}, ...)
    /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/image/v5/copy/single.go:907 +0x2f7 fp=0xc0000593a8 sp=0xc000059138 pc=0x5627957cbdb7
github.com/containers/image/v5/copy.(*imageCopier).copyLayer.func3(0xc000320380, 0xc0003c5cc0, {{0xc0006147d0, 0x47}, 0xaf2a800, {0x0, 0x0, 0x0}, 0x0, {0x562795bada58, ...}, ...}, ...)
    /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/image/v5/copy/single.go:816 +0x49d fp=0xc0000597a0 sp=0xc0000593a8 pc=0x5627957c9c5d
github.com/containers/image/v5/copy.(*imageCopier).copyLayer(0xc000320380, {0x562796019d90, 0x5627969046a0}, {{0xc0006147d0, 0x47}, 0xaf2a800, {0x0, 0x0, 0x0}, 0x0, ...}, ...)
    /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/image/v5/copy/single.go:848 +0xacf fp=0xc000059ca0 sp=0xc0000597a0 pc=0x5627957c924f
github.com/containers/image/v5/copy.(*imageCopier).copyLayers.func1(0x0, {{0xc0006147d0, 0x47}, 0xaf2a800, {0x0, 0x0, 0x0}, 0x0, {0x562795bada58, 0x2c}, ...}, ...)
    /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/image/v5/copy/single.go:459 +0x3b2 fp=0xc000059f48 sp=0xc000059ca0 pc=0x5627957c6cd2
github.com/containers/image/v5/copy.(*imageCopier).copyLayers.func2.gowrap3()
    /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/image/v5/copy/single.go:502 +0x66 fp=0xc000059fe0 sp=0xc000059f48 pc=0x5627957c6826
runtime.goexit({})
    /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000059fe8 sp=0xc000059fe0 pc=0x562794f7b641
created by github.com/containers/image/v5/copy.(*imageCopier).copyLayers.func2 in goroutine 1
    /builddir/build/BUILD/skopeo-1.15.0/vendor/github.com/containers/image/v5/copy/single.go:502 +0x154

goroutine 33 gp=0xc00051a700 m=nil [select]:
runtime.gopark(0xc0004d3f48?, 0x2?, 0xc0?, 0x3d?, 0xc0004d3ef4?)
    /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc0004d3d98 sp=0xc0004d3d78 pc=0x562794f485ce
runtime.selectgo(0xc0004d3f48, 0xc0004d3ef0, 0xc000118180?, 0x0, 0xc0004e5aa0?, 0x1)
    /usr/lib/golang/src/runtime/select.go:327 +0x725 fp=0xc0004d3eb8 sp=0xc0004d3d98 pc=0x562794f59a85
net/http.(*persistConn).writeLoop(0xc0000fe6c0)
    /usr/lib/golang/src/net/http/transport.go:2444 +0xf0 fp=0xc0004d3fc8 sp=0xc0004d3eb8 pc=0x5627952cb350
net/http.(*Transport).dialConn.gowrap3()
    /usr/lib/golang/src/net/http/transport.go:1800 +0x25 fp=0xc0004d3fe0 sp=0xc0004d3fc8 pc=0x5627952c7fa5
runtime.goexit({})
    /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0004d3fe8 sp=0xc0004d3fe0 pc=0x562794f7b641
created by net/http.(*Transport).dialConn in goroutine 24
    /usr/lib/golang/src/net/http/transport.go:1800 +0x1585

rax    0xca
rbx    0x0
rcx    0x562794f7d423
rdx    0x0
rdi    0x5627968a2ce0
rsi    0x80
rbp    0x7fffd495bd20
rsp    0x7fffd495bcd8
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0xd
r13    0x1
r14    0x5627968a1a80
r15    0x7
rip    0x562794f7d421
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x562794f7ce57 m=2 sigcode=0

goroutine 0 gp=0xc000006a80 m=2 mp=0xc000094808 [idle]:
runtime.usleep(0xa00)
    /usr/lib/golang/src/runtime/sys_linux_amd64.s:135 +0x37 fp=0x7f22223ffc70 sp=0x7f22223ffc50 pc=0x562794f7ce57
runtime.sysmon()
    /usr/lib/golang/src/runtime/proc.go:5965 +0xa5 fp=0x7f22223ffcd8 sp=0x7f22223ffc70 pc=0x562794f54aa5
runtime.mstart1()
    /usr/lib/golang/src/runtime/proc.go:1729 +0x95 fp=0x7f22223ffd00 sp=0x7f22223ffcd8 pc=0x562794f4b3b5
runtime.mstart0()
    /usr/lib/golang/src/runtime/proc.go:1686 +0x76 fp=0x7f22223ffd30 sp=0x7f22223ffd00 pc=0x562794f4b2f6
runtime.mstart()
    /usr/lib/golang/src/runtime/asm_amd64.s:394 +0x5 fp=0x7f22223ffd38 sp=0x7f22223ffd30 pc=0x562794f79565
rax    0xfffffffffffffffc
rbx    0xa00
rcx    0x562794f7ce57
rdx    0x0
rdi    0x7f22223ffc50
rsi    0x0
rbp    0x7f22223ffc60
rsp    0x7f22223ffc50
r8     0xd1ebc
r9     0x2515f125a53
r10    0x0
r11    0x212
r12    0x7f22223ffc50
r13    0xffffffffffffff78
r14    0xc000006a80
r15    0x7fffd495bae0
rip    0x562794f7ce57
rflags 0x212
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x562794f7d421 m=3 sigcode=0

goroutine 0 gp=0xc000006fc0 m=3 mp=0xc000095008 [idle]:
runtime.futex(0xc000095148, 0x80, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7f221bdffa78 sp=0x7f221bdffa70 pc=0x562794f7d421
runtime.futexsleep(0x7f221bdffaf0?, 0x94f19150?, 0x562794f55ee7?)
    /usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0x7f221bdffac8 sp=0x7f221bdffa78 pc=0x562794f41f90
runtime.notesleep(0xc000095148)
    /usr/lib/golang/src/runtime/lock_futex.go:170 +0x87 fp=0x7f221bdffb00 sp=0x7f221bdffac8 pc=0x562794f192e7
runtime.mPark(...)
    /usr/lib/golang/src/runtime/proc.go:1761
runtime.stopm()
    /usr/lib/golang/src/runtime/proc.go:2782 +0x8c fp=0x7f221bdffb30 sp=0x7f221bdffb00 pc=0x562794f4ccac
runtime.findRunnable()
    /usr/lib/golang/src/runtime/proc.go:3512 +0xd5f fp=0x7f221bdffca8 sp=0x7f221bdffb30 pc=0x562794f4e81f
runtime.schedule()
    /usr/lib/golang/src/runtime/proc.go:3868 +0xb1 fp=0x7f221bdffce0 sp=0x7f221bdffca8 pc=0x562794f4f8f1
runtime.goexit0(0xc000582fc0?)
    /usr/lib/golang/src/runtime/proc.go:4181 +0x18 fp=0x7f221bdffcf8 sp=0x7f221bdffce0 pc=0x562794f50798
runtime.mcall()
    /usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0x7f221bdffd10 sp=0x7f221bdffcf8 pc=0x562794f795f0
rax    0xca
rbx    0x0
rcx    0x562794f7d423
rdx    0x0
rdi    0xc000095148
rsi    0x80
rbp    0x7f221bdffab8
rsp    0x7f221bdffa70
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x1a
r13    0x1
r14    0xc000006fc0
r15    0xf
rip    0x562794f7d421
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x562794f7d421 m=4 sigcode=0

goroutine 0 gp=0xc0000076c0 m=4 mp=0xc000095808 [idle]:
runtime.futex(0xc000095948, 0x80, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7f22219ffa78 sp=0x7f22219ffa70 pc=0x562794f7d421
runtime.futexsleep(0x7f22219ffaf0?, 0x94f19150?, 0x562794f55ee7?)
    /usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0x7f22219ffac8 sp=0x7f22219ffa78 pc=0x562794f41f90
runtime.notesleep(0xc000095948)
    /usr/lib/golang/src/runtime/lock_futex.go:170 +0x87 fp=0x7f22219ffb00 sp=0x7f22219ffac8 pc=0x562794f192e7
runtime.mPark(...)
    /usr/lib/golang/src/runtime/proc.go:1761
runtime.stopm()
    /usr/lib/golang/src/runtime/proc.go:2782 +0x8c fp=0x7f22219ffb30 sp=0x7f22219ffb00 pc=0x562794f4ccac
runtime.findRunnable()
    /usr/lib/golang/src/runtime/proc.go:3512 +0xd5f fp=0x7f22219ffca8 sp=0x7f22219ffb30 pc=0x562794f4e81f
runtime.schedule()
    /usr/lib/golang/src/runtime/proc.go:3868 +0xb1 fp=0x7f22219ffce0 sp=0x7f22219ffca8 pc=0x562794f4f8f1
runtime.goexit0(0xc000f8a000?)
    /usr/lib/golang/src/runtime/proc.go:4181 +0x18 fp=0x7f22219ffcf8 sp=0x7f22219ffce0 pc=0x562794f50798
runtime.mcall()
    /usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0x7f22219ffd10 sp=0x7f22219ffcf8 pc=0x562794f795f0
rax    0xca
rbx    0x0
rcx    0x562794f7d423
rdx    0x0
rdi    0xc000095948
rsi    0x80
rbp    0x7f22219ffab8
rsp    0x7f22219ffa70
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x1a
r13    0x1
r14    0xc0000076c0
r15    0xf
rip    0x562794f7d421
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x562794f7d421 m=5 sigcode=0

goroutine 0 gp=0xc0001021c0 m=5 mp=0xc000100008 [idle]:
runtime.futex(0xc000100148, 0x80, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7f2220fffa10 sp=0x7f2220fffa08 pc=0x562794f7d421
runtime.futexsleep(0x7f2220fffa88?, 0x94f19150?, 0x0?)
    /usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0x7f2220fffa60 sp=0x7f2220fffa10 pc=0x562794f41f90
runtime.notesleep(0xc000100148)
    /usr/lib/golang/src/runtime/lock_futex.go:170 +0x87 fp=0x7f2220fffa98 sp=0x7f2220fffa60 pc=0x562794f192e7
runtime.mPark(...)
    /usr/lib/golang/src/runtime/proc.go:1761
runtime.stopm()
    /usr/lib/golang/src/runtime/proc.go:2782 +0x8c fp=0x7f2220fffac8 sp=0x7f2220fffa98 pc=0x562794f4ccac
runtime.gcstopm()
    /usr/lib/golang/src/runtime/proc.go:3081 +0xaa fp=0x7f2220fffaf0 sp=0x7f2220fffac8 pc=0x562794f4d8aa
runtime.findRunnable()
    /usr/lib/golang/src/runtime/proc.go:3149 +0x296 fp=0x7f2220fffc68 sp=0x7f2220fffaf0 pc=0x562794f4dd56
runtime.schedule()
    /usr/lib/golang/src/runtime/proc.go:3868 +0xb1 fp=0x7f2220fffca0 sp=0x7f2220fffc68 pc=0x562794f4f8f1
runtime.park_m(0xc00051bdc0)
    /usr/lib/golang/src/runtime/proc.go:4036 +0x1ec fp=0x7f2220fffcf8 sp=0x7f2220fffca0 pc=0x562794f4fecc
runtime.mcall()
    /usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0x7f2220fffd10 sp=0x7f2220fffcf8 pc=0x562794f795f0
rax    0xca
rbx    0x0
rcx    0x562794f7d423
rdx    0x0
rdi    0xc000100148
rsi    0x80
rbp    0x7f2220fffa50
rsp    0x7f2220fffa08
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x7f2220fffad0
r13    0x1
r14    0xc0001021c0
r15    0x1
rip    0x562794f7d421
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x562794f7d421 m=6 sigcode=0

goroutine 0 gp=0xc000007a40 m=6 mp=0xc0000c6008 [idle]:
runtime.futex(0x5627969059f8, 0x80, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7f221b3ffc30 sp=0x7f221b3ffc28 pc=0x562794f7d421
runtime.futexsleep(0x7f221b3ffca8?, 0x94f19150?, 0x8?)
    /usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0x7f221b3ffc80 sp=0x7f221b3ffc30 pc=0x562794f41f90
runtime.notesleep(0x5627969059f8)
    /usr/lib/golang/src/runtime/lock_futex.go:170 +0x87 fp=0x7f221b3ffcb8 sp=0x7f221b3ffc80 pc=0x562794f192e7
runtime.templateThread()
    /usr/lib/golang/src/runtime/proc.go:2760 +0x71 fp=0x7f221b3ffcd8 sp=0x7f221b3ffcb8 pc=0x562794f4cb91
runtime.mstart1()
    /usr/lib/golang/src/runtime/proc.go:1729 +0x95 fp=0x7f221b3ffd00 sp=0x7f221b3ffcd8 pc=0x562794f4b3b5
runtime.mstart0()
    /usr/lib/golang/src/runtime/proc.go:1686 +0x76 fp=0x7f221b3ffd30 sp=0x7f221b3ffd00 pc=0x562794f4b2f6
runtime.mstart()
    /usr/lib/golang/src/runtime/asm_amd64.s:394 +0x5 fp=0x7f221b3ffd38 sp=0x7f221b3ffd30 pc=0x562794f79565
rax    0xca
rbx    0x0
rcx    0x562794f7d423
rdx    0x0
rdi    0x5627969059f8
rsi    0x80
rbp    0x7f221b3ffc70
rsp    0x7f221b3ffc28
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0xc000007a40
r13    0xffffffffffffff78
r14    0xc000007a40
r15    0x7fffd495bb80
rip    0x562794f7d421
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x562794f7d421 m=7 sigcode=0

goroutine 0 gp=0xc000102540 m=7 mp=0xc000100808 [idle]:
runtime.futex(0xc000100948, 0x80, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7f221a9ffa38 sp=0x7f221a9ffa30 pc=0x562794f7d421
runtime.futexsleep(0x7f221a9ffab0?, 0x94f19150?, 0x562794f55ee7?)
    /usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0x7f221a9ffa88 sp=0x7f221a9ffa38 pc=0x562794f41f90
runtime.notesleep(0xc000100948)
    /usr/lib/golang/src/runtime/lock_futex.go:170 +0x87 fp=0x7f221a9ffac0 sp=0x7f221a9ffa88 pc=0x562794f192e7
runtime.mPark(...)
    /usr/lib/golang/src/runtime/proc.go:1761
runtime.stopm()
    /usr/lib/golang/src/runtime/proc.go:2782 +0x8c fp=0x7f221a9ffaf0 sp=0x7f221a9ffac0 pc=0x562794f4ccac
runtime.findRunnable()
    /usr/lib/golang/src/runtime/proc.go:3512 +0xd5f fp=0x7f221a9ffc68 sp=0x7f221a9ffaf0 pc=0x562794f4e81f
runtime.schedule()
    /usr/lib/golang/src/runtime/proc.go:3868 +0xb1 fp=0x7f221a9ffca0 sp=0x7f221a9ffc68 pc=0x562794f4f8f1
runtime.park_m(0xc00051bdc0)
    /usr/lib/golang/src/runtime/proc.go:4036 +0x1ec fp=0x7f221a9ffcf8 sp=0x7f221a9ffca0 pc=0x562794f4fecc
runtime.mcall()
    /usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0x7f221a9ffd10 sp=0x7f221a9ffcf8 pc=0x562794f795f0
rax    0xca
rbx    0x0
rcx    0x562794f7d423
rdx    0x0
rdi    0xc000100948
rsi    0x80
rbp    0x7f221a9ffa78
rsp    0x7f221a9ffa30
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0xd
r13    0x1
r14    0xc000102540
r15    0x7
rip    0x562794f7d421
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x562794f7d421 m=8 sigcode=0

goroutine 0 gp=0xc000102fc0 m=8 mp=0xc000101008 [idle]:
runtime.futex(0xc000101148, 0x80, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7f2219fffa38 sp=0x7f2219fffa30 pc=0x562794f7d421
runtime.futexsleep(0x7f2219fffab0?, 0x94f19150?, 0x562794f55ee7?)
    /usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0x7f2219fffa88 sp=0x7f2219fffa38 pc=0x562794f41f90
runtime.notesleep(0xc000101148)
    /usr/lib/golang/src/runtime/lock_futex.go:170 +0x87 fp=0x7f2219fffac0 sp=0x7f2219fffa88 pc=0x562794f192e7
runtime.mPark(...)
    /usr/lib/golang/src/runtime/proc.go:1761
runtime.stopm()
    /usr/lib/golang/src/runtime/proc.go:2782 +0x8c fp=0x7f2219fffaf0 sp=0x7f2219fffac0 pc=0x562794f4ccac
runtime.findRunnable()
    /usr/lib/golang/src/runtime/proc.go:3512 +0xd5f fp=0x7f2219fffc68 sp=0x7f2219fffaf0 pc=0x562794f4e81f
runtime.schedule()
    /usr/lib/golang/src/runtime/proc.go:3868 +0xb1 fp=0x7f2219fffca0 sp=0x7f2219fffc68 pc=0x562794f4f8f1
runtime.park_m(0xc000643a40)
    /usr/lib/golang/src/runtime/proc.go:4036 +0x1ec fp=0x7f2219fffcf8 sp=0x7f2219fffca0 pc=0x562794f4fecc
runtime.mcall()
    /usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0x7f2219fffd10 sp=0x7f2219fffcf8 pc=0x562794f795f0
rax    0xca
rbx    0x0
rcx    0x562794f7d423
rdx    0x0
rdi    0xc000101148
rsi    0x80
rbp    0x7f2219fffa78
rsp    0x7f2219fffa30
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0xd
r13    0x1
r14    0xc000102fc0
r15    0x3
rip    0x562794f7d421
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x562794f7d421 m=9 sigcode=0

goroutine 0 gp=0xc0006421c0 m=9 mp=0xc000640008 [idle]:
runtime.futex(0xc000640148, 0x80, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7f22195ffa38 sp=0x7f22195ffa30 pc=0x562794f7d421
runtime.futexsleep(0x7f22195ffab0?, 0x94f19150?, 0x562794f55ee7?)
    /usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0x7f22195ffa88 sp=0x7f22195ffa38 pc=0x562794f41f90
runtime.notesleep(0xc000640148)
    /usr/lib/golang/src/runtime/lock_futex.go:170 +0x87 fp=0x7f22195ffac0 sp=0x7f22195ffa88 pc=0x562794f192e7
runtime.mPark(...)
    /usr/lib/golang/src/runtime/proc.go:1761
runtime.stopm()
    /usr/lib/golang/src/runtime/proc.go:2782 +0x8c fp=0x7f22195ffaf0 sp=0x7f22195ffac0 pc=0x562794f4ccac
runtime.findRunnable()
    /usr/lib/golang/src/runtime/proc.go:3512 +0xd5f fp=0x7f22195ffc68 sp=0x7f22195ffaf0 pc=0x562794f4e81f
runtime.schedule()
    /usr/lib/golang/src/runtime/proc.go:3868 +0xb1 fp=0x7f22195ffca0 sp=0x7f22195ffc68 pc=0x562794f4f8f1
runtime.park_m(0xc00051b340)
    /usr/lib/golang/src/runtime/proc.go:4036 +0x1ec fp=0x7f22195ffcf8 sp=0x7f22195ffca0 pc=0x562794f4fecc
runtime.mcall()
    /usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0x7f22195ffd10 sp=0x7f22195ffcf8 pc=0x562794f795f0
rax    0xca
rbx    0x0
rcx    0x562794f7d423
rdx    0x0
rdi    0xc000640148
rsi    0x80
rbp    0x7f22195ffa78
rsp    0x7f22195ffa30
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x1a
r13    0x1
r14    0xc0006421c0
r15    0x7
rip    0x562794f7d421
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x562794f7d421 m=10 sigcode=0

goroutine 0 gp=0xc0006821c0 m=10 mp=0xc000680008 [idle]:
runtime.futex(0xc000680148, 0x80, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7f2218bffa78 sp=0x7f2218bffa70 pc=0x562794f7d421
runtime.futexsleep(0x7f2218bffaf0?, 0x94f19150?, 0x562794f55ee7?)
    /usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0x7f2218bffac8 sp=0x7f2218bffa78 pc=0x562794f41f90
runtime.notesleep(0xc000680148)
    /usr/lib/golang/src/runtime/lock_futex.go:170 +0x87 fp=0x7f2218bffb00 sp=0x7f2218bffac8 pc=0x562794f192e7
runtime.mPark(...)
    /usr/lib/golang/src/runtime/proc.go:1761
runtime.stopm()
    /usr/lib/golang/src/runtime/proc.go:2782 +0x8c fp=0x7f2218bffb30 sp=0x7f2218bffb00 pc=0x562794f4ccac
runtime.findRunnable()
    /usr/lib/golang/src/runtime/proc.go:3512 +0xd5f fp=0x7f2218bffca8 sp=0x7f2218bffb30 pc=0x562794f4e81f
runtime.schedule()
    /usr/lib/golang/src/runtime/proc.go:3868 +0xb1 fp=0x7f2218bffce0 sp=0x7f2218bffca8 pc=0x562794f4f8f1
runtime.goexit0(0xc000448700?)
    /usr/lib/golang/src/runtime/proc.go:4181 +0x18 fp=0x7f2218bffcf8 sp=0x7f2218bffce0 pc=0x562794f50798
runtime.mcall()
    /usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0x7f2218bffd10 sp=0x7f2218bffcf8 pc=0x562794f795f0
rax    0xca
rbx    0x0
rcx    0x562794f7d423
rdx    0x0
rdi    0xc000680148
rsi    0x80
rbp    0x7f2218bffab8
rsp    0x7f2218bffa70
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x1a
r13    0x1
r14    0xc0006821c0
r15    0x9
rip    0x562794f7d421
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x562794f7d421 m=11 sigcode=0

goroutine 0 gp=0xc0001036c0 m=11 mp=0xc000680808 [idle]:
runtime.futex(0xc000680948, 0x80, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7f2213dffa78 sp=0x7f2213dffa70 pc=0x562794f7d421
runtime.futexsleep(0x7f2213dffaf0?, 0x94f19150?, 0x562794f55ee7?)
    /usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0x7f2213dffac8 sp=0x7f2213dffa78 pc=0x562794f41f90
runtime.notesleep(0xc000680948)
    /usr/lib/golang/src/runtime/lock_futex.go:170 +0x87 fp=0x7f2213dffb00 sp=0x7f2213dffac8 pc=0x562794f192e7
runtime.mPark(...)
    /usr/lib/golang/src/runtime/proc.go:1761
runtime.stopm()
    /usr/lib/golang/src/runtime/proc.go:2782 +0x8c fp=0x7f2213dffb30 sp=0x7f2213dffb00 pc=0x562794f4ccac
runtime.findRunnable()
    /usr/lib/golang/src/runtime/proc.go:3512 +0xd5f fp=0x7f2213dffca8 sp=0x7f2213dffb30 pc=0x562794f4e81f
runtime.schedule()
    /usr/lib/golang/src/runtime/proc.go:3868 +0xb1 fp=0x7f2213dffce0 sp=0x7f2213dffca8 pc=0x562794f4f8f1
runtime.goexit0(0xc00078a8c0?)
    /usr/lib/golang/src/runtime/proc.go:4181 +0x18 fp=0x7f2213dffcf8 sp=0x7f2213dffce0 pc=0x562794f50798
runtime.mcall()
    /usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0x7f2213dffd10 sp=0x7f2213dffcf8 pc=0x562794f795f0
rax    0xca
rbx    0x0
rcx    0x562794f7d423
rdx    0x0
rdi    0xc000680948
rsi    0x80
rbp    0x7f2213dffab8
rsp    0x7f2213dffa70
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x1a
r13    0x1
r14    0xc0001036c0
r15    0x7
rip    0x562794f7d421
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x562794f7d421 m=12 sigcode=0

goroutine 0 gp=0xc00051a540 m=12 mp=0xc0000c6808 [idle]:
runtime.futex(0xc0000c6948, 0x80, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7f22133ffa38 sp=0x7f22133ffa30 pc=0x562794f7d421
runtime.futexsleep(0x7f22133ffab0?, 0x94f19150?, 0x562794f55ee7?)
    /usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0x7f22133ffa88 sp=0x7f22133ffa38 pc=0x562794f41f90
runtime.notesleep(0xc0000c6948)
    /usr/lib/golang/src/runtime/lock_futex.go:170 +0x87 fp=0x7f22133ffac0 sp=0x7f22133ffa88 pc=0x562794f192e7
runtime.mPark(...)
    /usr/lib/golang/src/runtime/proc.go:1761
runtime.stopm()
    /usr/lib/golang/src/runtime/proc.go:2782 +0x8c fp=0x7f22133ffaf0 sp=0x7f22133ffac0 pc=0x562794f4ccac
runtime.findRunnable()
    /usr/lib/golang/src/runtime/proc.go:3512 +0xd5f fp=0x7f22133ffc68 sp=0x7f22133ffaf0 pc=0x562794f4e81f
runtime.schedule()
    /usr/lib/golang/src/runtime/proc.go:3868 +0xb1 fp=0x7f22133ffca0 sp=0x7f22133ffc68 pc=0x562794f4f8f1
runtime.park_m(0xc00051b340)
    /usr/lib/golang/src/runtime/proc.go:4036 +0x1ec fp=0x7f22133ffcf8 sp=0x7f22133ffca0 pc=0x562794f4fecc
runtime.mcall()
    /usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0x7f22133ffd10 sp=0x7f22133ffcf8 pc=0x562794f795f0
rax    0xca
rbx    0x0
rcx    0x562794f7d423
rdx    0x0
rdi    0xc0000c6948
rsi    0x80
rbp    0x7f22133ffa78
rsp    0x7f22133ffa30
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x1a
r13    0x1
r14    0xc00051a540
r15    0x3
rip    0x562794f7d421
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x562794f7d421 m=13 sigcode=0

goroutine 0 gp=0xc000642c40 m=13 mp=0xc000640808 [idle]:
runtime.futex(0xc000640948, 0x80, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7f22129ffa38 sp=0x7f22129ffa30 pc=0x562794f7d421
runtime.futexsleep(0x7f22129ffab0?, 0x94f19150?, 0x562794f55ee7?)
    /usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0x7f22129ffa88 sp=0x7f22129ffa38 pc=0x562794f41f90
runtime.notesleep(0xc000640948)
    /usr/lib/golang/src/runtime/lock_futex.go:170 +0x87 fp=0x7f22129ffac0 sp=0x7f22129ffa88 pc=0x562794f192e7
runtime.mPark(...)
    /usr/lib/golang/src/runtime/proc.go:1761
runtime.stopm()
    /usr/lib/golang/src/runtime/proc.go:2782 +0x8c fp=0x7f22129ffaf0 sp=0x7f22129ffac0 pc=0x562794f4ccac
runtime.findRunnable()
    /usr/lib/golang/src/runtime/proc.go:3512 +0xd5f fp=0x7f22129ffc68 sp=0x7f22129ffaf0 pc=0x562794f4e81f
runtime.schedule()
    /usr/lib/golang/src/runtime/proc.go:3868 +0xb1 fp=0x7f22129ffca0 sp=0x7f22129ffc68 pc=0x562794f4f8f1
runtime.park_m(0xc00051bdc0)
    /usr/lib/golang/src/runtime/proc.go:4036 +0x1ec fp=0x7f22129ffcf8 sp=0x7f22129ffca0 pc=0x562794f4fecc
runtime.mcall()
    /usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0x7f22129ffd10 sp=0x7f22129ffcf8 pc=0x562794f795f0
rax    0xca
rbx    0x0
rcx    0x562794f7d423
rdx    0x0
rdi    0xc000640948
rsi    0x80
rbp    0x7f22129ffa78
rsp    0x7f22129ffa30
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0xc000079908
r13    0x1
r14    0xc000642c40
r15    0x1
rip    0x562794f7d421
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x562794f7d421 m=14 sigcode=0

goroutine 0 gp=0xc000103880 m=14 mp=0xc000681008 [idle]:
runtime.futex(0xc000681148, 0x80, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7f2211fffa78 sp=0x7f2211fffa70 pc=0x562794f7d421
runtime.futexsleep(0x7f2211fffaf0?, 0x94f19150?, 0x111fffb20?)
    /usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0x7f2211fffac8 sp=0x7f2211fffa78 pc=0x562794f41f90
runtime.notesleep(0xc000681148)
    /usr/lib/golang/src/runtime/lock_futex.go:170 +0x87 fp=0x7f2211fffb00 sp=0x7f2211fffac8 pc=0x562794f192e7
runtime.mPark(...)
    /usr/lib/golang/src/runtime/proc.go:1761
runtime.stopm()
    /usr/lib/golang/src/runtime/proc.go:2782 +0x8c fp=0x7f2211fffb30 sp=0x7f2211fffb00 pc=0x562794f4ccac
runtime.findRunnable()
    /usr/lib/golang/src/runtime/proc.go:3512 +0xd5f fp=0x7f2211fffca8 sp=0x7f2211fffb30 pc=0x562794f4e81f
runtime.schedule()
    /usr/lib/golang/src/runtime/proc.go:3868 +0xb1 fp=0x7f2211fffce0 sp=0x7f2211fffca8 pc=0x562794f4f8f1
runtime.goexit0(0xc0005b4000?)
    /usr/lib/golang/src/runtime/proc.go:4181 +0x18 fp=0x7f2211fffcf8 sp=0x7f2211fffce0 pc=0x562794f50798
runtime.mcall()
    /usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0x7f2211fffd10 sp=0x7f2211fffcf8 pc=0x562794f795f0
rax    0xca
rbx    0x0
rcx    0x562794f7d423
rdx    0x0
rdi    0xc000681148
rsi    0x80
rbp    0x7f2211fffab8
rsp    0x7f2211fffa70
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0xd
r13    0x1
r14    0xc000103880
r15    0x9
rip    0x562794f7d421
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x562794f7d421 m=15 sigcode=0

goroutine 0 gp=0xc000682540 m=15 mp=0xc000681808 [idle]:
runtime.futex(0xc000681948, 0x80, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7f22115ffa38 sp=0x7f22115ffa30 pc=0x562794f7d421
runtime.futexsleep(0x7f22115ffab0?, 0x94f19150?, 0x1115ffae0?)
    /usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0x7f22115ffa88 sp=0x7f22115ffa38 pc=0x562794f41f90
runtime.notesleep(0xc000681948)
    /usr/lib/golang/src/runtime/lock_futex.go:170 +0x87 fp=0x7f22115ffac0 sp=0x7f22115ffa88 pc=0x562794f192e7
runtime.mPark(...)
    /usr/lib/golang/src/runtime/proc.go:1761
runtime.stopm()
    /usr/lib/golang/src/runtime/proc.go:2782 +0x8c fp=0x7f22115ffaf0 sp=0x7f22115ffac0 pc=0x562794f4ccac
runtime.findRunnable()
    /usr/lib/golang/src/runtime/proc.go:3512 +0xd5f fp=0x7f22115ffc68 sp=0x7f22115ffaf0 pc=0x562794f4e81f
runtime.schedule()
    /usr/lib/golang/src/runtime/proc.go:3868 +0xb1 fp=0x7f22115ffca0 sp=0x7f22115ffc68 pc=0x562794f4f8f1
runtime.park_m(0xc00051bc00)
    /usr/lib/golang/src/runtime/proc.go:4036 +0x1ec fp=0x7f22115ffcf8 sp=0x7f22115ffca0 pc=0x562794f4fecc
runtime.mcall()
    /usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0x7f22115ffd10 sp=0x7f22115ffcf8 pc=0x562794f795f0
rax    0xca
rbx    0x0
rcx    0x562794f7d423
rdx    0x0
rdi    0xc000681948
rsi    0x80
rbp    0x7f22115ffa78
rsp    0x7f22115ffa30
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0xa5
r13    0x1
r14    0xc000682540
r15    0x1
rip    0x562794f7d421
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x562794f7d421 m=16 sigcode=0

goroutine 0 gp=0xc000682700 m=16 mp=0xc000641008 [idle]:
runtime.futex(0xc000641148, 0x80, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7f2210bffa78 sp=0x7f2210bffa70 pc=0x562794f7d421
runtime.futexsleep(0x7f2210bffaf0?, 0x94f19150?, 0x562794f55ee7?)
    /usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0x7f2210bffac8 sp=0x7f2210bffa78 pc=0x562794f41f90
runtime.notesleep(0xc000641148)
    /usr/lib/golang/src/runtime/lock_futex.go:170 +0x87 fp=0x7f2210bffb00 sp=0x7f2210bffac8 pc=0x562794f192e7
runtime.mPark(...)
    /usr/lib/golang/src/runtime/proc.go:1761
runtime.stopm()
    /usr/lib/golang/src/runtime/proc.go:2782 +0x8c fp=0x7f2210bffb30 sp=0x7f2210bffb00 pc=0x562794f4ccac
runtime.findRunnable()
    /usr/lib/golang/src/runtime/proc.go:3512 +0xd5f fp=0x7f2210bffca8 sp=0x7f2210bffb30 pc=0x562794f4e81f
runtime.schedule()
    /usr/lib/golang/src/runtime/proc.go:3868 +0xb1 fp=0x7f2210bffce0 sp=0x7f2210bffca8 pc=0x562794f4f8f1
runtime.goexit0(0xc0005b4380?)
    /usr/lib/golang/src/runtime/proc.go:4181 +0x18 fp=0x7f2210bffcf8 sp=0x7f2210bffce0 pc=0x562794f50798
runtime.mcall()
    /usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0x7f2210bffd10 sp=0x7f2210bffcf8 pc=0x562794f795f0
rax    0xca
rbx    0x0
rcx    0x562794f7d423
rdx    0x0
rdi    0xc000641148
rsi    0x80
rbp    0x7f2210bffab8
rsp    0x7f2210bffa70
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x17
r13    0x1
r14    0xc000682700
r15    0xd
rip    0x562794f7d421
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x562794f7d421 m=17 sigcode=0

goroutine 0 gp=0xc000583880 m=17 mp=0xc000641808 [idle]:
runtime.futex(0xc000641948, 0x80, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7f2207dffa78 sp=0x7f2207dffa70 pc=0x562794f7d421
runtime.futexsleep(0x7f2207dffaf0?, 0x94f19150?, 0x562794f55ee7?)
    /usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0x7f2207dffac8 sp=0x7f2207dffa78 pc=0x562794f41f90
runtime.notesleep(0xc000641948)
    /usr/lib/golang/src/runtime/lock_futex.go:170 +0x87 fp=0x7f2207dffb00 sp=0x7f2207dffac8 pc=0x562794f192e7
runtime.mPark(...)
    /usr/lib/golang/src/runtime/proc.go:1761
runtime.stopm()
    /usr/lib/golang/src/runtime/proc.go:2782 +0x8c fp=0x7f2207dffb30 sp=0x7f2207dffb00 pc=0x562794f4ccac
runtime.findRunnable()
    /usr/lib/golang/src/runtime/proc.go:3512 +0xd5f fp=0x7f2207dffca8 sp=0x7f2207dffb30 pc=0x562794f4e81f
runtime.schedule()
    /usr/lib/golang/src/runtime/proc.go:3868 +0xb1 fp=0x7f2207dffce0 sp=0x7f2207dffca8 pc=0x562794f4f8f1
runtime.goexit0(0xc000f8a380?)
    /usr/lib/golang/src/runtime/proc.go:4181 +0x18 fp=0x7f2207dffcf8 sp=0x7f2207dffce0 pc=0x562794f50798
runtime.mcall()
    /usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0x7f2207dffd10 sp=0x7f2207dffcf8 pc=0x562794f795f0
rax    0xca
rbx    0x0
rcx    0x562794f7d423
rdx    0x0
rdi    0xc000641948
rsi    0x80
rbp    0x7f2207dffab8
rsp    0x7f2207dffa70
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x17
r13    0x1
r14    0xc000583880
r15    0x7
rip    0x562794f7d421
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x562794f7d421 m=18 sigcode=0

goroutine 0 gp=0xc0001db880 m=18 mp=0xc000424008 [idle]:
runtime.futex(0xc000424148, 0x80, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7f22073ffa38 sp=0x7f22073ffa30 pc=0x562794f7d421
runtime.futexsleep(0x7f22073ffab0?, 0x94f19150?, 0x562794f55ee7?)
    /usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0x7f22073ffa88 sp=0x7f22073ffa38 pc=0x562794f41f90
runtime.notesleep(0xc000424148)
    /usr/lib/golang/src/runtime/lock_futex.go:170 +0x87 fp=0x7f22073ffac0 sp=0x7f22073ffa88 pc=0x562794f192e7
runtime.mPark(...)
    /usr/lib/golang/src/runtime/proc.go:1761
runtime.stopm()
    /usr/lib/golang/src/runtime/proc.go:2782 +0x8c fp=0x7f22073ffaf0 sp=0x7f22073ffac0 pc=0x562794f4ccac
runtime.findRunnable()
    /usr/lib/golang/src/runtime/proc.go:3512 +0xd5f fp=0x7f22073ffc68 sp=0x7f22073ffaf0 pc=0x562794f4e81f
runtime.schedule()
    /usr/lib/golang/src/runtime/proc.go:3868 +0xb1 fp=0x7f22073ffca0 sp=0x7f22073ffc68 pc=0x562794f4f8f1
runtime.park_m(0xc00051bdc0)
    /usr/lib/golang/src/runtime/proc.go:4036 +0x1ec fp=0x7f22073ffcf8 sp=0x7f22073ffca0 pc=0x562794f4fecc
runtime.mcall()
    /usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0x7f22073ffd10 sp=0x7f22073ffcf8 pc=0x562794f795f0
rax    0xca
rbx    0x0
rcx    0x562794f7d423
rdx    0x0
rdi    0xc000424148
rsi    0x80
rbp    0x7f22073ffa78
rsp    0x7f22073ffa30
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x7f22073ffaa8
r13    0xc00027ceb6
r14    0xc0001db880
r15    0x0
rip    0x562794f7d421
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

-----

SIGQUIT: quit
PC=0x562794f7d421 m=19 sigcode=0

goroutine 0 gp=0xc0006a01c0 m=19 mp=0xc00069e008 [idle]:
runtime.futex(0xc00069e148, 0x80, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/runtime/sys_linux_amd64.s:557 +0x21 fp=0x7f22069ffa78 sp=0x7f22069ffa70 pc=0x562794f7d421
runtime.futexsleep(0x7f22069ffaf0?, 0x94f19150?, 0x562794f55ee7?)
    /usr/lib/golang/src/runtime/os_linux.go:69 +0x30 fp=0x7f22069ffac8 sp=0x7f22069ffa78 pc=0x562794f41f90
runtime.notesleep(0xc00069e148)
    /usr/lib/golang/src/runtime/lock_futex.go:170 +0x87 fp=0x7f22069ffb00 sp=0x7f22069ffac8 pc=0x562794f192e7
runtime.mPark(...)
    /usr/lib/golang/src/runtime/proc.go:1761
runtime.stopm()
    /usr/lib/golang/src/runtime/proc.go:2782 +0x8c fp=0x7f22069ffb30 sp=0x7f22069ffb00 pc=0x562794f4ccac
runtime.findRunnable()
    /usr/lib/golang/src/runtime/proc.go:3512 +0xd5f fp=0x7f22069ffca8 sp=0x7f22069ffb30 pc=0x562794f4e81f
runtime.schedule()
    /usr/lib/golang/src/runtime/proc.go:3868 +0xb1 fp=0x7f22069ffce0 sp=0x7f22069ffca8 pc=0x562794f4f8f1
runtime.goexit0(0xc000f8a540?)
    /usr/lib/golang/src/runtime/proc.go:4181 +0x18 fp=0x7f22069ffcf8 sp=0x7f22069ffce0 pc=0x562794f50798
runtime.mcall()
    /usr/lib/golang/src/runtime/asm_amd64.s:458 +0x50 fp=0x7f22069ffd10 sp=0x7f22069ffcf8 pc=0x562794f795f0
rax    0xca
rbx    0x0
rcx    0x562794f7d423
rdx    0x0
rdi    0xc00069e148
rsi    0x80
rbp    0x7f22069ffab8
rsp    0x7f22069ffa70
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x1a
r13    0x1
r14    0xc0006a01c0
r15    0x5
rip    0x562794f7d421
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
ERRO[0024] signal: aborted (core dumped)                
ERRO[0024] (Unable to determine exit status)
mtrmac commented 6 months ago

AFAICT, the layer has been uploaded to the server; all of that blob has been consumed, the request was sent, and now it is waiting for the server to respond.

Now, if the destination is localhost inside a container, presumably the registry is not doing anything super expensive (unless, I guess, the backing storage or the CPU were super slow?!). So I don’t know why this would be hanging.

With localhost, we can rule out nondeterministic third-party network equipment.

debarshiray commented 6 months ago

Do TCP connections hanging when connecting to a localhost remapped port, possibly with toolbox involved, ring any bells, by chance?

Note that I am using a Fedora 40 Workstation virtual machine that I created just for this, and typed in the individual commands after lifting them from Toolbx's CI. I don't think I used toolbox(1) on this VM at all.

debarshiray commented 6 months ago

Does the Podman version running the registry container matter? I.e. is it really the distribution version, or the Podman version? (And/or one of the relevant Podman utility packages)

Good question. For some reason, I assumed that all current Fedoras have the same Podman version, but that's not the case. Fedora 40 onwards has 5.0.x and the older Fedoras have 4.9.4.

I will try to downgrade the Podman stack and see if that helps.

Luap99 commented 6 months ago

Assuming you podman as rootless pasta will forward the ports and has to handle many parts of the tcp stack so it can easily be broken in some way which causes this. The easy way to test if it is pasta's fault is to run podman with --network slirp4netns to get the old 4.X default.

Luap99 commented 6 months ago

Also if slirp4netns is working try to get the latest pasta (passt rpm) and see if it changes anything, if it still doies not work feel free to report this against podman I can ping the pasta maintainers there.

debarshiray commented 6 months ago

In the meantime, here are the logs from the docker-registry container.

These are produced at start-up:

time="2024-04-30T19:48:35.128982005Z" level=info msg="Starting upload purge in 32m0s" go.version=go1.20.8 instance.id=95c867a5-220b-4343-b779-bb83d2aa6f4a service=registry version=2.8.3 
time="2024-04-30T19:48:35.128940185Z" level=warning msg="No HTTP secret provided - generated random secret. This may cause problems with uploads if multiple registries are behind a load-balancer. To provide a shared secret, fill in http.secret in the configuration file or set the REGISTRY_HTTP_SECRET environment variable." go.version=go1.20.8 instance.id=95c867a5-220b-4343-b779-bb83d2aa6f4a service=registry version=2.8.3 
time="2024-04-30T19:48:35.129330926Z" level=info msg="redis not configured" go.version=go1.20.8 instance.id=95c867a5-220b-4343-b779-bb83d2aa6f4a service=registry version=2.8.3 
time="2024-04-30T19:48:35.129424844Z" level=info msg="using inmemory blob descriptor cache" go.version=go1.20.8 instance.id=95c867a5-220b-4343-b779-bb83d2aa6f4a service=registry version=2.8.3 
time="2024-04-30T19:48:35.130309522Z" level=info msg="restricting TLS version to tls1.2 or higher" go.version=go1.20.8 instance.id=95c867a5-220b-4343-b779-bb83d2aa6f4a service=registry version=2.8.3 
time="2024-04-30T19:48:35.130331533Z" level=info msg="restricting TLS cipher suites to: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_AES_128_GCM_SHA256,TLS_CHACHA20_POLY1305_SHA256,TLS_AES_256_GCM_SHA384" go.version=go1.20.8 instance.id=95c867a5-220b-4343-b779-bb83d2aa6f4a service=registry version=2.8.3 
time="2024-04-30T19:48:35.131439514Z" level=info msg="listening on [::]:443, tls" go.version=go1.20.8 instance.id=95c867a5-220b-4343-b779-bb83d2aa6f4a service=registry version=2.8.3

These are produced by a stuck attempt to copy a fedora:34 image:

127.0.0.1 - - [30/Apr/2024:19:48:46 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "skopeo/1.15.0"
time="2024-04-30T19:48:46.924498309Z" level=info msg="response completed" go.version=go1.20.8 http.request.host="localhost:50000" http.request.id=2ae4a4e8-074b-430c-b7ca-0d666ae475b2 http.request.method=GET http.request.remoteaddr="127.0.0.1:51044" http.request.uri="/v2/" http.request.useragent="skopeo/1.15.0" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.157789ms http.response.status=200 http.response.written=2 
127.0.0.1 - - [30/Apr/2024:19:48:46 +0000] "HEAD /v2/fedora/blobs/sha256:883e787c00d4208d75fc3e85d100ce64b517e49a2468f0e7f084cf05d16f3e46 HTTP/1.1" 404 157 "" "skopeo/1.15.0"
time="2024-04-30T19:48:46.989676246Z" level=error msg="response completed with error" err.code="blob unknown" err.detail=sha256:883e787c00d4208d75fc3e85d100ce64b517e49a2468f0e7f084cf05d16f3e46 err.message="blob unknown to registry" go.version=go1.20.8 http.request.host="localhost:50000" http.request.id=bc5fe4bf-562d-4349-8b6b-7fee37910e16 http.request.method=HEAD http.request.remoteaddr="127.0.0.1:51050" http.request.uri="/v2/fedora/blobs/sha256:883e787c00d4208d75fc3e85d100ce64b517e49a2468f0e7f084cf05d16f3e46" http.request.useragent="skopeo/1.15.0" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.354122ms http.response.status=404 http.response.written=157 vars.digest="sha256:883e787c00d4208d75fc3e85d100ce64b517e49a2468f0e7f084cf05d16f3e46" vars.name=fedora 
127.0.0.1 - - [30/Apr/2024:19:48:46 +0000] "HEAD /v2/fedora/blobs/sha256:badee9d31a5c4af72e7c2b6f98df496536452df0e6194cd5c70ba72b89a20a4d HTTP/1.1" 404 157 "" "skopeo/1.15.0"
time="2024-04-30T19:48:46.994730355Z" level=error msg="response completed with error" err.code="blob unknown" err.detail=sha256:badee9d31a5c4af72e7c2b6f98df496536452df0e6194cd5c70ba72b89a20a4d err.message="blob unknown to registry" go.version=go1.20.8 http.request.host="localhost:50000" http.request.id=dae26c86-501c-4d6d-86eb-12e2f6563cf6 http.request.method=HEAD http.request.remoteaddr="127.0.0.1:51050" http.request.uri="/v2/fedora/blobs/sha256:badee9d31a5c4af72e7c2b6f98df496536452df0e6194cd5c70ba72b89a20a4d" http.request.useragent="skopeo/1.15.0" http.response.contenttype="application/json; charset=utf-8" http.response.duration=2.108715ms http.response.status=404 http.response.written=157 vars.digest="sha256:badee9d31a5c4af72e7c2b6f98df496536452df0e6194cd5c70ba72b89a20a4d" vars.name=fedora 
127.0.0.1 - - [30/Apr/2024:19:48:46 +0000] "HEAD /v2/fedora-toolbox/blobs/sha256:badee9d31a5c4af72e7c2b6f98df496536452df0e6194cd5c70ba72b89a20a4d HTTP/1.1" 404 157 "" "skopeo/1.15.0"
time="2024-04-30T19:48:46.996124393Z" level=error msg="response completed with error" err.code="blob unknown" err.detail=sha256:badee9d31a5c4af72e7c2b6f98df496536452df0e6194cd5c70ba72b89a20a4d err.message="blob unknown to registry" go.version=go1.20.8 http.request.host="localhost:50000" http.request.id=9b47f8ed-8f27-4d59-b8ce-ce543eadfe08 http.request.method=HEAD http.request.remoteaddr="127.0.0.1:51050" http.request.uri="/v2/fedora-toolbox/blobs/sha256:badee9d31a5c4af72e7c2b6f98df496536452df0e6194cd5c70ba72b89a20a4d" http.request.useragent="skopeo/1.15.0" http.response.contenttype="application/json; charset=utf-8" http.response.duration="916.6µs" http.response.status=404 http.response.written=157 vars.digest="sha256:badee9d31a5c4af72e7c2b6f98df496536452df0e6194cd5c70ba72b89a20a4d" vars.name=fedora-toolbox 
127.0.0.1 - - [30/Apr/2024:19:48:46 +0000] "HEAD /v2/fedora/blobs/sha256:736fb53cfb115c56d5d662b657cfc0f86ce6fdc9ca2c4a9dc04b31f075b21257 HTTP/1.1" 404 157 "" "skopeo/1.15.0"
time="2024-04-30T19:48:46.9973282Z" level=error msg="response completed with error" err.code="blob unknown" err.detail=sha256:736fb53cfb115c56d5d662b657cfc0f86ce6fdc9ca2c4a9dc04b31f075b21257 err.message="blob unknown to registry" go.version=go1.20.8 http.request.host="localhost:50000" http.request.id=5828954a-ec1b-4374-a4bc-1aa909754c25 http.request.method=HEAD http.request.remoteaddr="127.0.0.1:51050" http.request.uri="/v2/fedora/blobs/sha256:736fb53cfb115c56d5d662b657cfc0f86ce6fdc9ca2c4a9dc04b31f075b21257" http.request.useragent="skopeo/1.15.0" http.response.contenttype="application/json; charset=utf-8" http.response.duration="777.106µs" http.response.status=404 http.response.written=157 vars.digest="sha256:736fb53cfb115c56d5d662b657cfc0f86ce6fdc9ca2c4a9dc04b31f075b21257" vars.name=fedora 
127.0.0.1 - - [30/Apr/2024:19:48:48 +0000] "POST /v2/fedora/blobs/uploads/ HTTP/1.1" 202 0 "" "skopeo/1.15.0"
time="2024-04-30T19:48:49.004362226Z" level=info msg="response completed" go.version=go1.20.8 http.request.host="localhost:50000" http.request.id=737e4c3f-778f-4096-9412-cf3015cf237e http.request.method=POST http.request.remoteaddr="127.0.0.1:51050" http.request.uri="/v2/fedora/blobs/uploads/" http.request.useragent="skopeo/1.15.0" http.response.duration=30.324106ms http.response.status=202 http.response.written=0

It's interesting that it refers to fedora-toolbox:34, just because I have it downloaded locally with podman pull.

mtrmac commented 6 months ago

It's interesting that it refers to fedora-toolbox:34, just because I have it downloaded locally with podman pull.

The “blob info cache” maintains local records about which layers exist on which registries, so HEAD requests attempting to reuse known or likely locations before actually starting an upload are expected.

debarshiray commented 6 months ago

Assuming you podman as rootless pasta will forward the ports and has to handle many parts of the tcp stack so it can easily be broken in some way which causes this. The easy way to test if it is pasta's fault is to run podman with --network slirp4netns to get the old 4.X default.

Thanks for that hint. Using podman run --network slirp4netns ... does seem to make it work. The test suite in https://github.com/containers/toolbox/pull/1468 no longer gets stuck on Fedora 40 and Rawhide in the same way as before.

debarshiray commented 6 months ago

Also if slirp4netns is working try to get the latest pasta (passt rpm) and see if it changes anything, if it still doies not work feel free to report this against podman I can ping the pasta maintainers there.

I think I already have the latest Pasta build, and it reproduces the problem:

$ rpm -q passt
passt-0^20240426.gd03c4e2-1.fc40.x86_64
debarshiray commented 6 months ago

I can't move issues across repositories in the Containers organization. So, will you take care of moving the issue?

Luap99 commented 6 months ago

Did you try to push with podman push, does this hang as well? Is the https setup required or does it also hang for unencrypted traffic? Juts looking to make the reproducer a bit simpler.

In any case you should run podman with something like --network pasta:--pcap,/tmp/pasta.pcap which will record all packages send then you can just upload the file here and the pasta folks can have a look based on that.

Luap99 commented 6 months ago

cc @sbrivio-rh @dgibson

debarshiray commented 6 months ago

Did you try to push with podman push, does this hang as well?

No, I didn't. I will try it now.

Is the https setup required or does it also hang for unencrypted traffic?

I don't think HTTPS is necessary, but I haven't actually tried plain text traffic.

Juts looking to make the reproducer a bit simpler.

This part of the test suite wasn't written by me. So I am also trying to remove the unnecessary bits as I go along. eg., the original code created a Docker registry that requires authentication.

I will try to get rid of the HTTPS part.

In any case you should run podman with something like --network pasta:--pcap,/tmp/pasta.pcap which will record all packages send then you can just upload the file here and the pasta folks can have a look based on that.

Okay!

sbrivio-rh commented 6 months ago

Could this have something to do with the host.containers.internal issue, and the container here would need an additional address?

I'm not exactly sure how skopeo uses pasta here, and how local ports are remapped (is it something going into pasta's configuration as well?).

Luap99 commented 6 months ago

Could this have something to do with the host.containers.internal https://github.com/containers/podman/issues/19213#issuecomment-2088022381, and the container here would need an additional address?

I don't think so

Skopeo doesn't use pasta, the setup contains a container registry run inside podman with a forwarded tcp port into the container. The container registry is really just a normal HTTP(S) server with a special API. skopeo just connects to it via the forwarded port and uploads the image data which seems to hang at the end of the upload.

mtrmac commented 6 months ago

Also, the TCP connection worked well enough to set up HTTPS, i.e. the packet flow was bidirectional and working, at least for a while. That should rule out at least the long-term-persistent aspects of IP addressing.

sbrivio-rh commented 6 months ago

Ah, okay, thanks for clarifying. Let's wait for captures then.

debarshiray commented 6 months ago

With plain text HTTP, the reproducer becomes:

Run the Docker registry container with:

$ podman run \
      --detach \
      --name docker-registry \
      --publish 5000:5000 \
      --rm \
      docker.io/library/registry:2
...

Then try to upload an image to it:

$ skopeo copy \
      --dest-tls-verify=false \
      containers-storage:registry.fedoraproject.org/fedora:34 \
      docker://localhost:5000/fedora:34
Getting image source signatures
Copying blob 883e787c00d4 done   | 
<stuck>

Note the extra --dest-tls-verify=false option and that the port has changed from 50000 to 5000.

debarshiray commented 6 months ago

you should run podman with something like --network pasta:--pcap,/tmp/pasta.pcap which will record all packages send then you can just upload the file here and the pasta folks can have a look based on that

Here's the file: https://rishi.fedorapeople.org/podman-22575-pasta.pcap

sbrivio-rh commented 6 months ago

Here's the file: https://rishi.fedorapeople.org/podman-22575-pasta.pcap

Weird, there are some packets there, but absolutely no TCP connections or data. It looks like it comes from an idle container or suchlike...?

sbrivio-rh commented 6 months ago

By the way, I tried the reproducer, but I'm not familiar with skopeo at all, and on the second command I get:

$ skopeo copy       --dest-tls-verify=false       containers-storage:registry.fedoraproject.org/fedora:39       docker://localhost:5000/fedora:39
FATA[0000] initializing source containers-storage:[overlay@/home/sbrivio/.local/share/containers/storage+/run/user/1000/containers]registry.fedoraproject.org/fedora:39: reference "[overlay@/home/sbrivio/.local/share/containers/storage+/run/user/1000/containers]registry.fedoraproject.org/fedora:39" does not resolve to an image ID: identifier is not an image
Luap99 commented 6 months ago

you have to pull the image first if it does not exists locally podman pull registry.fedoraproject.org/fedora:39

sbrivio-rh commented 6 months ago

Works for me :/

$ skopeo copy       --dest-tls-verify=false       containers-storage:registry.fedoraproject.org/fedora:39       docker://localhost:5000/fedora:39
Getting image source signatures
Copying blob 404a04c0d585 done   | 
Copying config f4f05d0fcc done   | 
Writing manifest to image destination

...with Podman and pasta at current HEAD, skopeo a bit older (1.13.3). I guess I should try matching the versions used here for all the components.

debarshiray commented 6 months ago

I have been using these for my reproducer:

$ rpm -q passt podman skopeo
passt-0^20240426.gd03c4e2-1.fc40.x86_64
podman-5.0.2-1.fc40.x86_64
skopeo-1.15.0-1.fc40.x86_64

There's definitely an element of non-determinism here. Skopeo didn't always get stuck on my Fedora 40 Workstation virtual machine, but it often did, while on the Zuul CI instance it seemed to get stuck almost always.

debarshiray commented 6 months ago

I have been doing some printf() debugging within the Skopeo sources to understand the flow, because Skopeo seemed like an easier starting point.

In vendor/github.com/containers/image/v5/copy/single.go, copyLayers(), one instance of a copyLayerHelper() go subrouting is launched. This invokes copyLayer(), where diffIDIsNeeded evaluates to false, and we end up in the final fallback copy layer logic.

Here, vendor/github.com/containers/image/v5/storage/storage_src.go, GetBlob() gets called, but not copyLayerFromStream().

mtrmac commented 6 months ago

That’s different from the earlier traceback, then. There, goroutine 51 is inside copyBlobFromStream, well after GetBlob has returned.

sbrivio-rh commented 6 months ago

That’s different from the earlier traceback, then. There, goroutine 51 is inside copyBlobFromStream, well after GetBlob has returned.

By the way, two notable differences between slirp4netns and pasta that might cause different behaviours here are:

All in all, typically, you'll get larger batches of data with pasta, and faster.

dgibson commented 6 months ago

Here's the file: https://rishi.fedorapeople.org/podman-22575-pasta.pcap

Weird, there are some packets there, but absolutely no TCP connections or data. It looks like it comes from an idle container or suchlike...?

Actually, that makes sense. From the original reproducer:

$ skopeo copy containers-storage:registry.fedoraproject.org/fedora:34 docker://localhost:50000/fedora:34

So we're connecting to the container via a localhost connection, which means it will go via pasta's "splice" path. That bypasses the normal tap interface, so won't show up in the packet trace. This also means that from the container's point of view the connection will also come from localhost, which might not be what the registry server as containerized expects, so that's one possible cause of the issue. Otherwise, I'd suggest this info as next steps:

Luap99 commented 6 months ago

A packet dump from the container's lo interface. That might be a bit fiddly to obtain if the container image doesn't include tcpdump, or dumpcap. It should be possible by using nsenter to enter just the container's netns and do that, though. There might be an easier podman specific way of doing so, @Luap99 ?

I really need to start writing some scripts around this given how common this is. The command would be somthing like podman unshare nsenter --net=$(podman inspect --format {{.NetworkSettings.SandboxKey}} <container id/name>) <command inside netns...> So if we only care about lo traffic then tcpdump -i lo -w lo.pcap as command in the container should do it.

Checking the exact command line of pasta with ps would be helpful. Do this after creating the container, but before running the stalling command, because...

starting podman with --log-level debug will print the pasta arguments

...it's worth checking if the pasta still exists after attempting the reproducer - it's possible pasta is simply crashing.

Just logically speaking if it crashes the sockets will be closed and skopeo shouldn't hang and instead fail with something like connection reset

debarshiray commented 6 months ago

There's clearly a race. I tried inserting a fmt.Printf() each in the deferred function calls in GetBlob() and one after GetBlob() had returned, and suddenly skopeo copy wasn't stuck anymore. Instead it, it ended with an exit code of 1 with this FATAL message:

FATA[0000] reading blob sha256:883e787c00d4208d75fc3e85d100ce64b517e49a2468f0e7f084cf05d16f3e46: read /var/tmp/container_images_2486861299: file already closed
mtrmac commented 6 months ago

What’s the change? I’m rather skeptical.

  1. I can’t recall seeing a failure like that. It would be a curious coincidence if you happened to hit one, after years of Skopeo existence, within 2 days or so, and while changing the code.
  2. By the time this stream gets down to copyBlobFromStream (where that error is almost certainly reported, in blobPipelineDetectCompressionStep), the stream is an io.Reader without a Close method even easily accessible. Now, granted, Go is dynamic enough that something could cast the stream to an io.Closer and call Close, but why would it?
  3. If this causes an error reading the input, how does it explain a hang?

It’s far too attractive for me to suspect that “adding a Printf to a deferred function call” happened to turn one of the deferred calls into a non-deferred one.

debarshiray commented 6 months ago

What’s the change? I’m rather skeptical.

Here's a scratch build with the changes I am playing with: https://koji.fedoraproject.org/koji/taskinfo?taskID=117212076

You can grab the source RPM and look at the changes I made. Note that I commented out the fmt.Printf() invocations in the deferred calls and it still gives me the FATAL above.

Leaving aside my nosing around, there were occasions when an unmodified skopeo(1) still worked, and didn't get stuck, with the Docker registry container running with pasta(1). It even made me believe for a while that only copying some images triggered the bug.

debarshiray commented 6 months ago

Here's the file: https://rishi.fedorapeople.org/podman-22575-pasta.pcap

Weird, there are some packets there, but absolutely no TCP connections or data. It looks like it comes from an idle container or suchlike...?

Actually, that makes sense. From the original reproducer:

$ skopeo copy containers-storage:registry.fedoraproject.org/fedora:34 docker://localhost:50000/fedora:34

So we're connecting to the container via a localhost connection, which means it will go via pasta's "splice" path. That bypasses the normal tap interface, so won't show up in the packet trace. This also means that from the container's point of view the connection will also come from localhost,

Thanks for that explanation.

which might not be what the registry server as containerized expects, so that's one possible cause of the issue. Otherwise, I'd suggest this info as next steps:

The documentation for the Docker registry does talk about running it locally inside a container. Although, they talk about Docker containers, not Podman containers. So, maybe the difference due to pasta(1) matters?

ip a output from within the registry container (just to make sure there aren't somehow some additional network interfaces confusing the picture.

After reproducing a stuck skopeo copy, and with the Docker registry container running:

$ podman exec docker-registry ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc fq_codel state UNKNOWN qlen 1000
    link/ether ae:98:25:49:26:ea brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global noprefixroute enp1s0
       valid_lft forever preferred_lft forever
    inet6 fe80::ac98:25ff:fe49:26ea/64 scope link 
       valid_lft forever preferred_lft forever
mtrmac commented 6 months ago

Thanks. That does all look reasonable AFAICT.

I can see one bug in there already, although that one should not make a difference. And the file already closed behavior is consistent with that RPM… except I can’t explain it. Looking.

debarshiray commented 6 months ago

Thanks. That does all look reasonable AFAICT.

I can see one bug in there already, although that one should not make a difference. And the file already closed behavior is consistent with that RPM… except I can’t explain it. Looking.

It gets stuck again if I move around the fmt.Printf() invocations a bit. See this scratch build: https://koji.fedoraproject.org/koji/taskinfo?taskID=117213243

Except, this time it's stuck inside copyLayerFromStream() within blobPipelineDecryptionStep(). This is puzzling because I was certain that copyLayerFromStream() wasn't being called before.

debarshiray commented 6 months ago

Checking the exact command line of pasta with ps would be helpful. Do this after creating the container, but before running the stalling command, because... ...it's worth checking if the pasta still exists after attempting the reproducer - it's possible pasta is simply crashing.

The pasta(1) process doesn't crash and runs as:

/usr/bin/pasta \
    --config-net \
    -t 5000-5000:5000-5000 \
    --dns-forward 169.254.0.1 \
    -u none \
    -T none \
    -U none \
    --no-map-gw \
    --quiet \
    --netns /run/user/1000/netns/netns-52b2cf2f-a904-7570-d531-3d0d0f243879

I also don't see anything in coredumpctl list.

mtrmac commented 6 months ago

It’s far too attractive for me to suspect that “adding a Printf to a deferred function call” happened to turn one of the deferred calls into a non-deferred one.

So this is true after all, but oh how hidden that was.

diff --git a/vendor/github.com/containers/image/v5/storage/storage_src.go b/vendor/github.com/containers/image/v5/storage/storage_src.go
index 967334c..6304743 100644
--- a/vendor/github.com/containers/image/v5/storage/storage_src.go
+++ b/vendor/github.com/containers/image/v5/storage/storage_src.go
@@ -161,9 +161,10 @@ func (s *storageImageSource) GetBlob(ctx context.Context, info types.BlobInfo, c
        if err != nil {
                return nil, 0, err
        }
+       rc2 := rc
        defer func() {
                fmt.Printf(">>> GetBlob defer #0, rc %#p\n", rc)
-               rc.Close()
+               rc2.Close()
        }()

        tmpFile, err := tmpdir.CreateBigFileTemp(s.systemContext, "")

rc is a named return value in this function, so return ioutils.NewReadCloserWrapper sets rc, and then that defer closes the temporary file instead of the input stream as intended.

defer rc.Close() evaluates rc immediately, and works; defer func() { rc.Close() }() evaluates rc with the final return value, and doesn’t.

debarshiray commented 6 months ago

Except, this time it's stuck inside copyLayerFromStream() within blobPipelineDecryptionStep(). This is puzzling because I was certain that copyLayerFromStream() wasn't being called before.

Ok, I was just misled by the progress bar overwriting the last line.

mtrmac commented 6 months ago

skopeo … |& cat to get rid of the progress bars.

mtrmac commented 6 months ago

I can see one bug in there already, although that one should not make a difference.

https://github.com/containers/image/pull/2394 for the record. Also removes the named return value gotcha.

debarshiray commented 6 months ago

It’s far too attractive for me to suspect that “adding a Printf to a deferred function call” happened to turn one of the deferred calls into a non-deferred one.

So this is true after all, but oh how hidden that was.

diff --git a/vendor/github.com/containers/image/v5/storage/storage_src.go b/vendor/github.com/containers/image/v5/storage/storage_src.go
index 967334c..6304743 100644
--- a/vendor/github.com/containers/image/v5/storage/storage_src.go
+++ b/vendor/github.com/containers/image/v5/storage/storage_src.go
@@ -161,9 +161,10 @@ func (s *storageImageSource) GetBlob(ctx context.Context, info types.BlobInfo, c
        if err != nil {
                return nil, 0, err
        }
+       rc2 := rc
        defer func() {
                fmt.Printf(">>> GetBlob defer #0, rc %#p\n", rc)
-               rc.Close()
+               rc2.Close()
        }()

        tmpFile, err := tmpdir.CreateBigFileTemp(s.systemContext, "")

rc is a named return value in this function, so return ioutils.NewReadCloserWrapper sets rc, and then that defer closes the temporary file instead of the input stream as intended.

defer rc.Close() evaluates rc immediately, and works; defer func() { rc.Close() }() evaluates rc with the final return value, and doesn’t.

Wow! I did suspect that it might be related to the different evaluation of rc due to the change in the deferred call, but I didn't spot the named return value. :)

Are those named return values really needed? The way the function is written, they don't seem necessary at all. At least we should use different names for them.

mtrmac commented 6 months ago

Also, I see

$ for i in $(seq 1 100); do (podman run --rm --detach       --name docker-registry       --publish 5000:5000       --rm       docker.io/library/registry:2 &&  skopeo copy --dest-creds=x       --dest-tls-verify=false       containers-storage:registry.fedoraproject.org/fedora:34       docker://localhost:5000/fedora:34 |& cat && podman stop -a && sleep 1) || break; done

passing, as an unprivileged user (with su - from a root’s session), with

# rpm -q passt podman skopeo
passt-0^20240426.gd03c4e2-1.fc40.x86_64
podman-5.0.2-1.fc40.x86_64
skopeo-1.15.0-1.fc40.x86_64

There must be some other variable involved if this is hanging frequently in other environments.

debarshiray commented 6 months ago

I can see one bug in there already, although that one should not make a difference.

containers/image#2394 for the record. Also removes the named return value gotcha.

Cool, I will take a look. My GNOME Shell suddenly stopped accepting keyboard input, which seems to happen on Fedora 38. So I had to reboot and missed your comment due to a stale page.