search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.07k stars 2.35k forks source link

fatal error: concurrent map writes when running podman container runlabel #2420

Closed frobware closed 5 years ago

frobware commented 5 years ago

/kind bug

Description

Running ppodman container runlabel -p pod quay.io/baude/demodb:latest generates a fatal error fatal error: concurrent map writes and then dumps its goroutines.

Steps to reproduce the issue:

$ podman container runlabel -p pod quay.io/baude/demodb:latest
Trying to pull quay.io/baude/demodb:latest...Getting image source signatures
Copying blob 38e2e6cd5626: 0 B / ? [------------------------------------------=]
Copying blob 705054bc3f5b: 0 B / ? [------------------------------------------=]
Copying blob c7051e069564: 0 B / ? [------------------------------------------=]
Copying blob 7308e914506c: 0 B / ? [------------------------------------------=]
Copying blob a3ed95caeb02: 0 B / ? [------------------------------------------=]
Copying blob 35e6984cb587: 0 B / ? [------------------------------------------=]
Copying blob 3a173c4702b4: 0 B / ? [------------------------------------------=]
Copying blob a3ed95caeb02: 0 B / ? [------------------------------------------=]
Copying blob efd003ff8e24: 0 B / ? [------------------------------------------=]
Copying blob ba5d30791443: 0 B / ? [------------------------------------------=]
Copying blob f3e943c9e01d: 0 B / ? [------------------------------------------=]
Copying blob a3ed95caeb02: 0 B / ? [------------------------------------------=]
Copying blob e5243a434e4f: 0 B / ? [------------------------------------------=]
Copying blob a3ed95caeb02: 0 B / ? [------------------------------------------=]
Copying blob a3ed95caeb02: 0 B / ? [------------------------------------------=]
Copying blob 910d8b012ee8: 0 B / ? [------------------------------------------=]
Copying blob 1fb787f18e3d: 0 B / ? [------------------------------------------=]
Copying blob a3ed95caeb02: 0 B / ? [------------------------------------------=]
Copying blob 7a0cfbee5299: 0 B / ? [------------------------------------------=]
Copying blob 6fa7c8911619: 0 B / ? [------------------------------------------=]
Copying blob a3ed95caeb02: 0 B / ? [------------------------------------------=]
Copying blob a3ed95caeb02: 0 B / ? [------------------------------------------=]
Copying blob a3ed95caeb02: 0 B / ? [------------------------------------------=]
Copying blob a3ed95caeb02: 0 B / ? [------------------------------------------=]
Copying blob a3ed95caeb02: 0 B / ? [------------------------------------------=]
Copying blob 31044f7a38a9: 0 B / ? [------------------------------------------=]
fatal error: concurrent map writes

goroutine 61 [running]:
runtime.throw(0x13bf36b, 0x15)
    /usr/lib/golang/src/runtime/panic.go:608 +0x72 fp=0xc0006f7708 sp=0xc0006f76d8 pc=0x436272
runtime.mapassign(0x120c120, 0xc0001f5080, 0xc0006f7838, 0x211cfa0)
    /usr/lib/golang/src/runtime/map.go:651 +0x545 fp=0xc0006f7790 sp=0xc0006f7708 pc=0x4194b5
github.com/containers/libpod/vendor/github.com/containers/image/pkg/blobinfocache.(*memoryCache).RecordKnownLocation(0xc0004ceee0, 0x1587c20, 0x211b5f8, 0xc0002d2ce0, 0x7, 0xc0003327d0, 0x47, 0xc00056f360, 0x14)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/pkg/blobinfocache/memory.go:79 +0x20e fp=0xc0006f7878 sp=0xc0006f7790 pc=0xbeadde
github.com/containers/libpod/vendor/github.com/containers/image/docker.(*dockerImageSource).GetBlob(0xc0002c6e80, 0x1589760, 0xc000040150, 0xc0003327d0, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/docker/docker_image_src.go:187 +0x49b fp=0xc0006f79b0 sp=0xc0006f7878 pc=0xccf36b
github.com/containers/libpod/vendor/github.com/containers/image/copy.(*imageCopier).copyLayer(0xc00047c2d0, 0x1589760, 0xc000040150, 0xc0003327d0, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:613 +0x16e fp=0xc0006f7d58 sp=0xc0006f79b0 pc=0xca81de
github.com/containers/libpod/vendor/github.com/containers/image/copy.(*imageCopier).copyLayers.func1(0x4, 0xc0003327d0, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:465 +0x632 fp=0xc0006f7f88 sp=0xc0006f7d58 pc=0xcacc02
runtime.goexit()
    /usr/lib/golang/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0006f7f90 sp=0xc0006f7f88 pc=0x4641d1
created by github.com/containers/libpod/vendor/github.com/containers/image/copy.(*imageCopier).copyLayers
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:484 +0x66e

goroutine 1 [select]:
github.com/containers/libpod/vendor/golang.org/x/sync/semaphore.(*Weighted).Acquire(0xc00043eb90, 0x1589760, 0xc000040150, 0x1, 0x47, 0xffffffffffffffff)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/golang.org/x/sync/semaphore/semaphore.go:60 +0x1aa
github.com/containers/libpod/vendor/github.com/containers/image/copy.(*imageCopier).copyLayers(0xc00047c2d0, 0x1589760, 0xc000040150, 0x0, 0x0)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:483 +0x5e5
github.com/containers/libpod/vendor/github.com/containers/image/copy.(*copier).copyOneImage(0xc0004c6b40, 0x1589760, 0xc000040150, 0xc0002f1f00, 0xc0004c6360, 0xc0004c6ba0, 0x0, 0x0, 0x30, 0xc0004119b0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:278 +0x4fb
github.com/containers/libpod/vendor/github.com/containers/image/copy.Image(0x1589760, 0xc000040150, 0xc0002f1f00, 0x1595ac0, 0xc00021df80, 0x1595dc0, 0xc0004ccc00, 0xc0004c6360, 0x0, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:177 +0x8ab
github.com/containers/libpod/libpod/image.(*Runtime).doPullImage(0xc0002f0f40, 0x1589760, 0xc000040150, 0xc000133760, 0xc0001f4120, 0x1, 0x1, 0x0, 0x0, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/libpod/image/pull.go:241 +0x2ca
github.com/containers/libpod/libpod/image.(*Runtime).pullImageFromHeuristicSource(0xc0002f0f40, 0x1589760, 0xc000040150, 0x7ffc24866b5c, 0x1b, 0x15776c0, 0xc00000e018, 0x0, 0x0, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/libpod/image/pull.go:210 +0x1c4
github.com/containers/libpod/libpod/image.(*Runtime).New(0xc0002f0f40, 0x1589760, 0xc000040150, 0x7ffc24866b5c, 0x1b, 0x0, 0x0, 0x0, 0x0, 0x15776c0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/libpod/image/image.go:148 +0x18f
github.com/containers/libpod/cmd/podman/shared.GetRunlabel(0x7ffc24866b58, 0x3, 0x7ffc24866b5c, 0x1b, 0x1589760, 0xc000040150, 0xc0000e4300, 0xc00018e001, 0x0, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/cmd/podman/shared/container.go:624 +0x163
main.runlabelCmd(0xc000132c60, 0x0, 0x0)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/cmd/podman/runlabel.go:164 +0x600
github.com/containers/libpod/vendor/github.com/urfave/cli.HandleAction(0x11e3160, 0x14474c0, 0xc000132c60, 0x0, 0xc00021c660)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/urfave/cli/app.go:501 +0xc8
github.com/containers/libpod/vendor/github.com/urfave/cli.Command.Run(0x13ad6b9, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x13edc47, 0x2f, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/urfave/cli/command.go:165 +0x459
github.com/containers/libpod/vendor/github.com/urfave/cli.(*App).RunAsSubcommand(0xc00018ce00, 0xc0001326e0, 0x0, 0x0)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/urfave/cli/app.go:383 +0x827
github.com/containers/libpod/vendor/github.com/urfave/cli.Command.startApp(0x13ae6a1, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x13b9522, 0x11, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/urfave/cli/command.go:377 +0x808
github.com/containers/libpod/vendor/github.com/urfave/cli.Command.Run(0x13ae6a1, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x13b9522, 0x11, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/urfave/cli/command.go:103 +0x80f
github.com/containers/libpod/vendor/github.com/urfave/cli.(*App).Run(0xc00018cc40, 0xc0000301e0, 0x6, 0x6, 0x0, 0x0)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/urfave/cli/app.go:259 +0x6bb
main.main()
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/cmd/podman/main.go:273 +0x15a6

goroutine 5 [chan receive]:
github.com/containers/libpod/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x20fc320)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/golang/glog/glog.go:882 +0x8b
created by github.com/containers/libpod/vendor/github.com/golang/glog.init.0
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/golang/glog/glog.go:410 +0x203

goroutine 6 [syscall]:
os/signal.signal_recv(0x0)
    /usr/lib/golang/src/runtime/sigqueue.go:139 +0x9c
os/signal.loop()
    /usr/lib/golang/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
    /usr/lib/golang/src/os/signal/signal_unix.go:29 +0x41

goroutine 58 [runnable]:
syscall.openat(0xffffffffffffff9c, 0xc0005555c0, 0x1b, 0x802c1, 0xc000000180, 0x1b, 0xc0006e7228, 0x53fb0f)
    /usr/lib/golang/src/syscall/zsyscall_linux_amd64.go:62 +0x184
syscall.Open(0xc0005555c0, 0x1b, 0x802c1, 0xc000000180, 0xc7b14f, 0xc0006e7280, 0x2)
    /usr/lib/golang/src/syscall/syscall_linux.go:138 +0x50
os.openFileNolog(0xc0005555c0, 0x1b, 0x2c1, 0xc000000180, 0x19, 0x1f78bf1, 0x1)
    /usr/lib/golang/src/os/file_unix.go:190 +0x85
os.OpenFile(0xc0005555c0, 0x1b, 0x2c1, 0xc000000180, 0xc0000b6568, 0x1, 0x20fd0c0)
    /usr/lib/golang/src/os/file.go:284 +0x5f
github.com/containers/libpod/vendor/github.com/containers/image/storage.(*storageImageDestination).PutBlob(0xc00019a620, 0x1589760, 0xc000040150, 0x1577080, 0xc00056a200, 0xc0003328c0, 0x47, 0x351, 0x0, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/storage/storage_image.go:362 +0x18c
github.com/containers/libpod/vendor/github.com/containers/image/copy.(*copier).copyBlobFromStream(0xc0004c6b40, 0x1589760, 0xc000040150, 0x15774e0, 0xc0002c6c00, 0xc0003328c0, 0x47, 0x351, 0x0, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:786 +0x354
github.com/containers/libpod/vendor/github.com/containers/image/copy.(*imageCopier).copyLayerFromStream(0xc00047c2d0, 0x1589760, 0xc000040150, 0x15774e0, 0xc0002c6c00, 0xc0003328c0, 0x47, 0x351, 0x0, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:672 +0x19e
github.com/containers/libpod/vendor/github.com/containers/image/copy.(*imageCopier).copyLayer(0xc00047c2d0, 0x1589760, 0xc000040150, 0xc0003328c0, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:619 +0x30f
github.com/containers/libpod/vendor/github.com/containers/image/copy.(*imageCopier).copyLayers.func1(0x1, 0xc0003328c0, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:465 +0x632
created by github.com/containers/libpod/vendor/github.com/containers/image/copy.(*imageCopier).copyLayers
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:484 +0x66e

goroutine 109 [runnable]:
internal/poll.runtime_pollWait(0x7efeac279e30, 0x72, 0xc0006a1858)
    /usr/lib/golang/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc0000eaa98, 0x72, 0xffffffffffffff00, 0x15790a0, 0x1f71a40)
    /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc0000eaa98, 0xc000025000, 0x1000, 0x1000)
    /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc0000eaa80, 0xc000025000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/internal/poll/fd_unix.go:169 +0x179
net.(*netFD).Read(0xc0000eaa80, 0xc000025000, 0x1000, 0x1000, 0x413e7b, 0xc000532000, 0x124c9c0)
    /usr/lib/golang/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00000e050, 0xc000025000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/net.go:177 +0x68
crypto/tls.(*block).readFromUntil(0xc000345080, 0x1577400, 0xc00000e050, 0x5, 0xc00000e050, 0xdb470d01b74b8161)
    /usr/lib/golang/src/crypto/tls/conn.go:492 +0x89
crypto/tls.(*Conn).readRecord(0xc00009d500, 0x1447d17, 0xc00009d620, 0x20fc140)
    /usr/lib/golang/src/crypto/tls/conn.go:593 +0xdd
crypto/tls.(*Conn).Read(0xc00009d500, 0xc00076c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/crypto/tls/conn.go:1145 +0xf1
net/http.(*persistConn).Read(0xc0000e05a0, 0xc00076c000, 0x1000, 0x1000, 0x40fda3, 0xc000384470, 0x434fd2)
    /usr/lib/golang/src/net/http/transport.go:1497 +0x75
bufio.(*Reader).fill(0xc0000303c0)
    /usr/lib/golang/src/bufio/bufio.go:100 +0x10f
bufio.(*Reader).Peek(0xc0000303c0, 0x1, 0xc00004a4e0, 0xc000384558, 0x0, 0x0, 0xc00009c700)
    /usr/lib/golang/src/bufio/bufio.go:132 +0x3f
net/http.(*persistConn).readLoop(0xc0000e05a0)
    /usr/lib/golang/src/net/http/transport.go:1645 +0x1a2
created by net/http.(*Transport).dialConn
    /usr/lib/golang/src/net/http/transport.go:1338 +0x941

goroutine 55 [select]:
github.com/containers/libpod/vendor/gopkg.in/cheggaaa/pb%2ev1.catchTerminate(0xc0002a7020)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/gopkg.in/cheggaaa/pb.v1/pb_x.go:112 +0x181
created by github.com/containers/libpod/vendor/gopkg.in/cheggaaa/pb%2ev1.lockEcho
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/gopkg.in/cheggaaa/pb.v1/pb_x.go:83 +0x315

goroutine 56 [select]:
github.com/containers/libpod/vendor/gopkg.in/cheggaaa/pb%2ev1.(*Pool).writer(0xc000030a80)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/gopkg.in/cheggaaa/pb.v1/pool.go:77 +0xf9
created by github.com/containers/libpod/vendor/gopkg.in/cheggaaa/pb%2ev1.(*Pool).Start
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/gopkg.in/cheggaaa/pb.v1/pool.go:60 +0xba

goroutine 57 [select]:
net/http.(*persistConn).roundTrip(0xc0000e06c0, 0xc00047cbd0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/http/transport.go:2101 +0x56a
net/http.(*Transport).roundTrip(0xc00035ed80, 0xc0000ee200, 0x1575a20, 0xc000080040, 0x4ed)
    /usr/lib/golang/src/net/http/transport.go:465 +0x9b1
net/http.(*Transport).RoundTrip(0xc00035ed80, 0xc0000ee200, 0xc00035ed80, 0x0, 0x0)
    /usr/lib/golang/src/net/http/roundtrip.go:17 +0x35
net/http.send(0xc0000ee200, 0x15774a0, 0xc00035ed80, 0x0, 0x0, 0x0, 0xc0006ae2a0, 0x8537fb, 0xc000093350, 0x2)
    /usr/lib/golang/src/net/http/client.go:250 +0x14b
net/http.(*Client).send(0xc0001f4f30, 0xc0000ee200, 0x0, 0x0, 0x0, 0xc0006ae2a0, 0x1, 0x2, 0x0)
    /usr/lib/golang/src/net/http/client.go:174 +0xfa
net/http.(*Client).do(0xc0001f4f30, 0xc0004f5000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/http/client.go:641 +0x2a8
net/http.(*Client).Do(0xc0001f4f30, 0xc0004f5000, 0xc0006fb720, 0x2, 0x2)
    /usr/lib/golang/src/net/http/client.go:509 +0x35
github.com/containers/libpod/vendor/github.com/containers/image/docker.(*dockerClient).makeRequestToResolvedURL(0xc0003a01e0, 0x1589760, 0xc000040150, 0x13a685f, 0x3, 0xc00053c460, 0x6d, 0x0, 0x0, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/docker/docker_client.go:449 +0x3dc
github.com/containers/libpod/vendor/github.com/containers/image/docker.(*dockerClient).makeRequest(0xc0003a01e0, 0x1589760, 0xc000040150, 0x13a685f, 0x3, 0xc0002a75c0, 0x5e, 0x0, 0x0, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/docker/docker_client.go:418 +0x247
github.com/containers/libpod/vendor/github.com/containers/image/docker.(*dockerImageSource).GetBlob(0xc0002c6e80, 0x1589760, 0xc000040150, 0xc000332910, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/docker/docker_image_src.go:179 +0x267
github.com/containers/libpod/vendor/github.com/containers/image/copy.(*imageCopier).copyLayer(0xc00047c2d0, 0x1589760, 0xc000040150, 0xc000332910, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:613 +0x16e
github.com/containers/libpod/vendor/github.com/containers/image/copy.(*imageCopier).copyLayers.func1(0x0, 0xc000332910, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:465 +0x632
created by github.com/containers/libpod/vendor/github.com/containers/image/copy.(*imageCopier).copyLayers
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:484 +0x66e

goroutine 94 [select]:
net/http.(*persistConn).readLoop(0xc0001f8000)
    /usr/lib/golang/src/net/http/transport.go:1761 +0x6b9
created by net/http.(*Transport).dialConn
    /usr/lib/golang/src/net/http/transport.go:1338 +0x941

goroutine 59 [select]:
net/http.(*persistConn).roundTrip(0xc00023e360, 0xc0002bd560, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/http/transport.go:2101 +0x56a
net/http.(*Transport).roundTrip(0xc00035ed80, 0xc0004f4000, 0x1575a20, 0xc000080040, 0x4ed)
    /usr/lib/golang/src/net/http/transport.go:465 +0x9b1
net/http.(*Transport).RoundTrip(0xc00035ed80, 0xc0004f4000, 0xc00035ed80, 0x0, 0x0)
    /usr/lib/golang/src/net/http/roundtrip.go:17 +0x35
net/http.send(0xc0004f4000, 0x15774a0, 0xc00035ed80, 0x0, 0x0, 0x0, 0xc000305660, 0x8537fb, 0xc0004d2310, 0x2)
    /usr/lib/golang/src/net/http/client.go:250 +0x14b
net/http.(*Client).send(0xc0001f4f30, 0xc0004f4000, 0x0, 0x0, 0x0, 0xc000305660, 0x1, 0x2, 0x0)
    /usr/lib/golang/src/net/http/client.go:174 +0xfa
net/http.(*Client).do(0xc0001f4f30, 0xc0004f4e00, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/http/client.go:641 +0x2a8
net/http.(*Client).Do(0xc0001f4f30, 0xc0004f4e00, 0xc000625720, 0x2, 0x2)
    /usr/lib/golang/src/net/http/client.go:509 +0x35
github.com/containers/libpod/vendor/github.com/containers/image/docker.(*dockerClient).makeRequestToResolvedURL(0xc0003a01e0, 0x1589760, 0xc000040150, 0x13a685f, 0x3, 0xc00053c3f0, 0x6d, 0x0, 0x0, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/docker/docker_client.go:449 +0x3dc
github.com/containers/libpod/vendor/github.com/containers/image/docker.(*dockerClient).makeRequest(0xc0003a01e0, 0x1589760, 0xc000040150, 0x13a685f, 0x3, 0xc0002a73e0, 0x5e, 0x0, 0x0, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/docker/docker_client.go:418 +0x247
github.com/containers/libpod/vendor/github.com/containers/image/docker.(*dockerImageSource).GetBlob(0xc0002c6e80, 0x1589760, 0xc000040150, 0xc000332870, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/docker/docker_image_src.go:179 +0x267
github.com/containers/libpod/vendor/github.com/containers/image/copy.(*imageCopier).copyLayer(0xc00047c2d0, 0x1589760, 0xc000040150, 0xc000332870, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:613 +0x16e
github.com/containers/libpod/vendor/github.com/containers/image/copy.(*imageCopier).copyLayers.func1(0x2, 0xc000332870, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:465 +0x632
created by github.com/containers/libpod/vendor/github.com/containers/image/copy.(*imageCopier).copyLayers
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:484 +0x66e

goroutine 60 [select]:
net/http.(*persistConn).roundTrip(0xc0000e05a0, 0xc00047c570, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/http/transport.go:2101 +0x56a
net/http.(*Transport).roundTrip(0xc00035ed80, 0xc0000ee000, 0x1575a20, 0xc000080040, 0x4ed)
    /usr/lib/golang/src/net/http/transport.go:465 +0x9b1
net/http.(*Transport).RoundTrip(0xc00035ed80, 0xc0000ee000, 0xc00035ed80, 0x0, 0x0)
    /usr/lib/golang/src/net/http/roundtrip.go:17 +0x35
net/http.send(0xc0000ee000, 0x15774a0, 0xc00035ed80, 0x0, 0x0, 0x0, 0xc0006ae1d0, 0x8537fb, 0xc000093050, 0x2)
    /usr/lib/golang/src/net/http/client.go:250 +0x14b
net/http.(*Client).send(0xc0001f4f30, 0xc0000ee000, 0x0, 0x0, 0x0, 0xc0006ae1d0, 0x1, 0x2, 0x0)
    /usr/lib/golang/src/net/http/client.go:174 +0xfa
net/http.(*Client).do(0xc0001f4f30, 0xc0000ee600, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/http/client.go:641 +0x2a8
net/http.(*Client).Do(0xc0001f4f30, 0xc0000ee600, 0xc000627720, 0x2, 0x2)
    /usr/lib/golang/src/net/http/client.go:509 +0x35
github.com/containers/libpod/vendor/github.com/containers/image/docker.(*dockerClient).makeRequestToResolvedURL(0xc0003a01e0, 0x1589760, 0xc000040150, 0x13a685f, 0x3, 0xc0002e4310, 0x6d, 0x0, 0x0, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/docker/docker_client.go:449 +0x3dc
github.com/containers/libpod/vendor/github.com/containers/image/docker.(*dockerClient).makeRequest(0xc0003a01e0, 0x1589760, 0xc000040150, 0x13a685f, 0x3, 0xc00004afc0, 0x5e, 0x0, 0x0, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/docker/docker_client.go:418 +0x247
github.com/containers/libpod/vendor/github.com/containers/image/docker.(*dockerImageSource).GetBlob(0xc0002c6e80, 0x1589760, 0xc000040150, 0xc000332820, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/docker/docker_image_src.go:179 +0x267
github.com/containers/libpod/vendor/github.com/containers/image/copy.(*imageCopier).copyLayer(0xc00047c2d0, 0x1589760, 0xc000040150, 0xc000332820, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:613 +0x16e
github.com/containers/libpod/vendor/github.com/containers/image/copy.(*imageCopier).copyLayers.func1(0x3, 0xc000332820, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:465 +0x632
created by github.com/containers/libpod/vendor/github.com/containers/image/copy.(*imageCopier).copyLayers
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:484 +0x66e

goroutine 62 [select]:
net/http.(*persistConn).roundTrip(0xc00023ec60, 0xc0002bdda0, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/http/transport.go:2101 +0x56a
net/http.(*Transport).roundTrip(0xc00035ed80, 0xc0004f4100, 0x1575a20, 0xc000080040, 0x4ed)
    /usr/lib/golang/src/net/http/transport.go:465 +0x9b1
net/http.(*Transport).RoundTrip(0xc00035ed80, 0xc0004f4100, 0xc00035ed80, 0x0, 0x0)
    /usr/lib/golang/src/net/http/roundtrip.go:17 +0x35
net/http.send(0xc0004f4100, 0x15774a0, 0xc00035ed80, 0x0, 0x0, 0x0, 0xc000305730, 0x8537fb, 0xc0004d2490, 0x2)
    /usr/lib/golang/src/net/http/client.go:250 +0x14b
net/http.(*Client).send(0xc0001f4f30, 0xc0004f4100, 0x0, 0x0, 0x0, 0xc000305730, 0x1, 0x2, 0x0)
    /usr/lib/golang/src/net/http/client.go:174 +0xfa
net/http.(*Client).do(0xc0001f4f30, 0xc0004f4c00, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/http/client.go:641 +0x2a8
net/http.(*Client).Do(0xc0001f4f30, 0xc0004f4c00, 0xc000393720, 0x2, 0x2)
    /usr/lib/golang/src/net/http/client.go:509 +0x35
github.com/containers/libpod/vendor/github.com/containers/image/docker.(*dockerClient).makeRequestToResolvedURL(0xc0003a01e0, 0x1589760, 0xc000040150, 0x13a685f, 0x3, 0xc00053c380, 0x6d, 0x0, 0x0, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/docker/docker_client.go:449 +0x3dc
github.com/containers/libpod/vendor/github.com/containers/image/docker.(*dockerClient).makeRequest(0xc0003a01e0, 0x1589760, 0xc000040150, 0x13a685f, 0x3, 0xc0002a7140, 0x5e, 0x0, 0x0, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/docker/docker_client.go:418 +0x247
github.com/containers/libpod/vendor/github.com/containers/image/docker.(*dockerImageSource).GetBlob(0xc0002c6e80, 0x1589760, 0xc000040150, 0xc000332780, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/docker/docker_image_src.go:179 +0x267
github.com/containers/libpod/vendor/github.com/containers/image/copy.(*imageCopier).copyLayer(0xc00047c2d0, 0x1589760, 0xc000040150, 0xc000332780, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:613 +0x16e
github.com/containers/libpod/vendor/github.com/containers/image/copy.(*imageCopier).copyLayers.func1(0x5, 0xc000332780, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:465 +0x632
created by github.com/containers/libpod/vendor/github.com/containers/image/copy.(*imageCopier).copyLayers
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:484 +0x66e

goroutine 95 [select]:
net/http.(*persistConn).writeLoop(0xc0001f8000)
    /usr/lib/golang/src/net/http/transport.go:1885 +0x113
created by net/http.(*Transport).dialConn
    /usr/lib/golang/src/net/http/transport.go:1339 +0x966

goroutine 110 [select]:
net/http.(*persistConn).writeLoop(0xc0000e05a0)
    /usr/lib/golang/src/net/http/transport.go:1885 +0x113
created by net/http.(*Transport).dialConn
    /usr/lib/golang/src/net/http/transport.go:1339 +0x966

goroutine 146 [select]:
net/http.(*persistConn).writeLoop(0xc00035e7e0)
    /usr/lib/golang/src/net/http/transport.go:1885 +0x113
created by net/http.(*Transport).dialConn
    /usr/lib/golang/src/net/http/transport.go:1339 +0x966

goroutine 97 [select]:
net/http.(*persistConn).writeLoop(0xc00023ec60)
    /usr/lib/golang/src/net/http/transport.go:1885 +0x113
created by net/http.(*Transport).dialConn
    /usr/lib/golang/src/net/http/transport.go:1339 +0x966

goroutine 111 [runnable]:
syscall.Syscall(0x0, 0xf, 0xc0006c3000, 0x1000, 0x1000, 0x1000, 0x0)
    /usr/lib/golang/src/syscall/asm_linux_amd64.s:18 +0x5
syscall.read(0xf, 0xc0006c3000, 0x1000, 0x1000, 0xffffffffffffff00, 0x0, 0x0)
    /usr/lib/golang/src/syscall/zsyscall_linux_amd64.go:732 +0x5a
syscall.Read(0xf, 0xc0006c3000, 0x1000, 0x1000, 0xffffffffffffffff, 0x15790a0, 0x1f71a40)
    /usr/lib/golang/src/syscall/syscall_unix.go:172 +0x49
internal/poll.(*FD).Read(0xc00034c680, 0xc0006c3000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/internal/poll/fd_unix.go:165 +0x119
net.(*netFD).Read(0xc00034c680, 0xc0006c3000, 0x1000, 0x1000, 0x413e7b, 0xc000532000, 0x124c9c0)
    /usr/lib/golang/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00000e060, 0xc0006c3000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/net.go:177 +0x68
crypto/tls.(*block).readFromUntil(0xc000345aa0, 0x1577400, 0xc00000e060, 0x5, 0xc00000e060, 0x11c2940)
    /usr/lib/golang/src/crypto/tls/conn.go:492 +0x89
crypto/tls.(*Conn).readRecord(0xc000288000, 0x1447d17, 0xc000288120, 0x20fc140)
    /usr/lib/golang/src/crypto/tls/conn.go:593 +0xdd
crypto/tls.(*Conn).Read(0xc000288000, 0xc000600000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/crypto/tls/conn.go:1145 +0xf1
net/http.(*persistConn).Read(0xc0000e06c0, 0xc000600000, 0x1000, 0x1000, 0x40fda3, 0xc00038a470, 0x434fd2)
    /usr/lib/golang/src/net/http/transport.go:1497 +0x75
bufio.(*Reader).fill(0xc0000307e0)
    /usr/lib/golang/src/bufio/bufio.go:100 +0x10f
bufio.(*Reader).Peek(0xc0000307e0, 0x1, 0xc0000aa5a0, 0xc00038a558, 0x0, 0x3a, 0xffffffffffffffff)
    /usr/lib/golang/src/bufio/bufio.go:132 +0x3f
net/http.(*persistConn).readLoop(0xc0000e06c0)
    /usr/lib/golang/src/net/http/transport.go:1645 +0x1a2
created by net/http.(*Transport).dialConn
    /usr/lib/golang/src/net/http/transport.go:1338 +0x941

goroutine 96 [runnable]:
internal/poll.runtime_pollWait(0x7efeac279950, 0x72, 0xc0006a3858)
    /usr/lib/golang/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc0005f6618, 0x72, 0xffffffffffffff00, 0x15790a0, 0x1f71a40)
    /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc0005f6618, 0xc000642000, 0x1000, 0x1000)
    /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc0005f6600, 0xc000642000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/internal/poll/fd_unix.go:169 +0x179
net.(*netFD).Read(0xc0005f6600, 0xc000642000, 0x1000, 0x1000, 0x413e7b, 0xc000532000, 0x124c9c0)
    /usr/lib/golang/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0002ea078, 0xc000642000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/net.go:177 +0x68
crypto/tls.(*block).readFromUntil(0xc0001f53e0, 0x1577400, 0xc0002ea078, 0x5, 0xc0002ea078, 0x21)
    /usr/lib/golang/src/crypto/tls/conn.go:492 +0x89
crypto/tls.(*Conn).readRecord(0xc000179500, 0x1447d17, 0xc000179620, 0x7efeac2835d0)
    /usr/lib/golang/src/crypto/tls/conn.go:593 +0xdd
crypto/tls.(*Conn).Read(0xc000179500, 0xc000773000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/crypto/tls/conn.go:1145 +0xf1
net/http.(*persistConn).Read(0xc00023ec60, 0xc000773000, 0x1000, 0x1000, 0x40fda3, 0xc0006a3c70, 0x434fd2)
    /usr/lib/golang/src/net/http/transport.go:1497 +0x75
bufio.(*Reader).fill(0xc00021cf00)
    /usr/lib/golang/src/bufio/bufio.go:100 +0x10f
bufio.(*Reader).Peek(0xc00021cf00, 0x1, 0xc0002a6000, 0xc000384558, 0x0, 0x0, 0xc00009c700)
    /usr/lib/golang/src/bufio/bufio.go:132 +0x3f
net/http.(*persistConn).readLoop(0xc00023ec60)
    /usr/lib/golang/src/net/http/transport.go:1645 +0x1a2
created by net/http.(*Transport).dialConn
    /usr/lib/golang/src/net/http/transport.go:1338 +0x941

goroutine 112 [select]:
net/http.(*persistConn).writeLoop(0xc0000e06c0)
    /usr/lib/golang/src/net/http/transport.go:1885 +0x113
created by net/http.(*Transport).dialConn
    /usr/lib/golang/src/net/http/transport.go:1339 +0x966

goroutine 131 [select]:
net/http.(*persistConn).writeLoop(0xc00023e360)
    /usr/lib/golang/src/net/http/transport.go:1885 +0x113
created by net/http.(*Transport).dialConn
    /usr/lib/golang/src/net/http/transport.go:1339 +0x966

goroutine 113 [select]:
net/http.(*persistConn).readLoop(0xc00035e7e0)
    /usr/lib/golang/src/net/http/transport.go:1761 +0x6b9
created by net/http.(*Transport).dialConn
    /usr/lib/golang/src/net/http/transport.go:1338 +0x941

goroutine 130 [runnable]:
internal/poll.runtime_pollWait(0x7efeac279af0, 0x72, 0xc000660858)
    /usr/lib/golang/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc00034c298, 0x72, 0xffffffffffffff00, 0x15790a0, 0x1f71a40)
    /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc00034c298, 0xc0005ce000, 0x1000, 0x1000)
    /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc00034c280, 0xc0005ce000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/internal/poll/fd_unix.go:169 +0x179
net.(*netFD).Read(0xc00034c280, 0xc0005ce000, 0x1000, 0x1000, 0x413e7b, 0xc000532000, 0x124c9c0)
    /usr/lib/golang/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00000e058, 0xc0005ce000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/net/net.go:177 +0x68
crypto/tls.(*block).readFromUntil(0xc0003d6810, 0x1577400, 0xc00000e058, 0x5, 0xc00000e058, 0xc0005796e0)
    /usr/lib/golang/src/crypto/tls/conn.go:492 +0x89
crypto/tls.(*Conn).readRecord(0xc00009dc00, 0x1447d17, 0xc00009dd20, 0x7efeac1f2ee8)
    /usr/lib/golang/src/crypto/tls/conn.go:593 +0xdd
crypto/tls.(*Conn).Read(0xc00009dc00, 0xc0005f9000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/lib/golang/src/crypto/tls/conn.go:1145 +0xf1
net/http.(*persistConn).Read(0xc00023e360, 0xc0005f9000, 0x1000, 0x1000, 0x40fda3, 0xc000660c70, 0x434fd2)
    /usr/lib/golang/src/net/http/transport.go:1497 +0x75
bufio.(*Reader).fill(0xc00021d5c0)
    /usr/lib/golang/src/bufio/bufio.go:100 +0x10f
bufio.(*Reader).Peek(0xc00021d5c0, 0x1, 0xc0005aa060, 0xc000388558, 0x0, 0x3a, 0xffffffffffffffff)
    /usr/lib/golang/src/bufio/bufio.go:132 +0x3f
net/http.(*persistConn).readLoop(0xc00023e360)
    /usr/lib/golang/src/net/http/transport.go:1645 +0x1a2
created by net/http.(*Transport).dialConn
    /usr/lib/golang/src/net/http/transport.go:1338 +0x941

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

$ podman version
Version:       1.0.0
Go Version:    go1.11.4
Git Commit:    "49780a1cf10d572edc4e1ea3b8a8429ce391d47d"
Built:         Mon Jan 14 20:38:17 2019
OS/Arch:       linux/amd64

Output of podman info:

$ podman info
host:
  BuildahVersion: 1.6-dev
  Conmon:
    package: podman-1.0.0-1.git82e8011.fc29.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.12.0-dev, commit: 49780a1cf10d572edc4e1ea3b8a8429ce391d47d'
  Distribution:
    distribution: fedora
    version: "29"
  MemFree: 11358875648
  MemTotal: 20422283264
  OCIRuntime:
    package: runc-1.0.0-68.dev.git6635b4f.fc29.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc6+dev
      commit: ef9132178ccc3d2775d4fb51f1e431f30cac1398-dirty
      spec: 1.0.1-dev
  SwapFree: 10280235008
  SwapTotal: 10280235008
  arch: amd64
  cpus: 4
  hostname: t460s
  kernel: 4.20.10-200.fc29.x86_64
  os: linux
  rootless: true
  uptime: 13h 38m 29.63s (Approximately 0.54 days)
insecure registries:
  registries: []
registries:
  registries:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /home/aim/.config/containers/storage.conf
  ContainerStore:
    number: 1
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mount_program=/usr/bin/fuse-overlayfs
  GraphRoot: /home/aim/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
  ImageStore:
    number: 5
  RunRoot: /run/user/1000

Additional environment details (AWS, VirtualBox, physical, etc.):

Fedora 29 - and up to date as of Sun Feb 24 07:29:39 GMT 2019.

rhatdan commented 5 years ago

@vrothberg This looks to be in a area of code you know best.

mheon commented 5 years ago

C/Image progress bars, looks like. Good news is we already replaced them upstream. Need to confirm this is fixed there, but my initial thinking is it probably is.

On Sun, Feb 24, 2019, 02:34 Andrew McDermott notifications@github.com wrote:

/kind bug

Description

Running ppodman container runlabel -p pod quay.io/baude/demodb:latest generates a fatal error fatal error: concurrent map writes and then dumps its goroutines.

Steps to reproduce the issue:

$ podman container runlabel -p pod quay.io/baude/demodb:latestTrying to pull quay.io/baude/demodb:latest...Getting image source signaturesCopying blob 38e2e6cd5626: 0 B / ? [------------------------------------------=]Copying blob 705054bc3f5b: 0 B / ? [------------------------------------------=]Copying blob c7051e069564: 0 B / ? [------------------------------------------=]Copying blob 7308e914506c: 0 B / ? [------------------------------------------=]Copying blob a3ed95caeb02: 0 B / ? [------------------------------------------=]Copying blob 35e6984cb587: 0 B / ? [------------------------------------------=]Copying blob 3a173c4702b4: 0 B / ? [------------------------------------------=]Copying blob a3ed95caeb02: 0 B / ? [------------------------------------------=]Copying blob efd003ff8e24: 0 B / ? [------------------------------------------=]Copying blob ba5d30791443: 0 B / ? [------------------------------------------=]Copying blob f3e943c9e01d: 0 B / ? [------------------------------------------=]Copying blob a3ed95caeb02: 0 B / ? [------------------------------------------=]Copying blob e5243a434e4f: 0 B / ? [------------------------------------------=]Copying blob a3ed95caeb02: 0 B / ? [------------------------------------------=]Copying blob a3ed95caeb02: 0 B / ? [------------------------------------------=]Copying blob 910d8b012ee8: 0 B / ? [------------------------------------------=]Copying blob 1fb787f18e3d: 0 B / ? [------------------------------------------=]Copying blob a3ed95caeb02: 0 B / ? [------------------------------------------=]Copying blob 7a0cfbee5299: 0 B / ? [------------------------------------------=]Copying blob 6fa7c8911619: 0 B / ? [------------------------------------------=]Copying blob a3ed95caeb02: 0 B / ? [------------------------------------------=]Copying blob a3ed95caeb02: 0 B / ? [------------------------------------------=]Copying blob a3ed95caeb02: 0 B / ? [------------------------------------------=]Copying blob a3ed95caeb02: 0 B / ? [------------------------------------------=]Copying blob a3ed95caeb02: 0 B / ? [------------------------------------------=]Copying blob 31044f7a38a9: 0 B / ? [------------------------------------------=]fatal error: concurrent map writes goroutine 61 [running]:runtime.throw(0x13bf36b, 0x15) /usr/lib/golang/src/runtime/panic.go:608 +0x72 fp=0xc0006f7708 sp=0xc0006f76d8 pc=0x436272runtime.mapassign(0x120c120, 0xc0001f5080, 0xc0006f7838, 0x211cfa0) /usr/lib/golang/src/runtime/map.go:651 +0x545 fp=0xc0006f7790 sp=0xc0006f7708 pc=0x4194b5github.com/containers/libpod/vendor/github.com/containers/image/pkg/blobinfocache.(memoryCache).RecordKnownLocation(0xc0004ceee0, 0x1587c20, 0x211b5f8, 0xc0002d2ce0, 0x7, 0xc0003327d0, 0x47, 0xc00056f360, 0x14) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/pkg/blobinfocache/memory.go:79 +0x20e fp=0xc0006f7878 sp=0xc0006f7790 pc=0xbeaddegithub.com/containers/libpod/vendor/github.com/containers/image/docker.(dockerImageSource).GetBlob(0xc0002c6e80, 0x1589760, 0xc000040150, 0xc0003327d0, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/docker/docker_image_src.go:187 +0x49b fp=0xc0006f79b0 sp=0xc0006f7878 pc=0xccf36bgithub.com/containers/libpod/vendor/github.com/containers/image/copy.(imageCopier).copyLayer(0xc00047c2d0, 0x1589760, 0xc000040150, 0xc0003327d0, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:613 +0x16e fp=0xc0006f7d58 sp=0xc0006f79b0 pc=0xca81degithub.com/containers/libpod/vendor/github.com/containers/image/copy.(imageCopier).copyLayers.func1(0x4, 0xc0003327d0, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:465 +0x632 fp=0xc0006f7f88 sp=0xc0006f7d58 pc=0xcacc02runtime.goexit() /usr/lib/golang/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0006f7f90 sp=0xc0006f7f88 pc=0x4641d1created by github.com/containers/libpod/vendor/github.com/containers/image/copy.(imageCopier).copyLayers /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:484 +0x66e goroutine 1 [select]:github.com/containers/libpod/vendor/golang.org/x/sync/semaphore.(Weighted).Acquire(0xc00043eb90, 0x1589760, 0xc000040150, 0x1, 0x47, 0xffffffffffffffff) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/golang.org/x/sync/semaphore/semaphore.go:60 +0x1aagithub.com/containers/libpod/vendor/github.com/containers/image/copy.(imageCopier).copyLayers(0xc00047c2d0, 0x1589760, 0xc000040150, 0x0, 0x0) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:483 +0x5e5github.com/containers/libpod/vendor/github.com/containers/image/copy.(copier).copyOneImage(0xc0004c6b40, 0x1589760, 0xc000040150, 0xc0002f1f00, 0xc0004c6360, 0xc0004c6ba0, 0x0, 0x0, 0x30, 0xc0004119b0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:278 +0x4fbgithub.com/containers/libpod/vendor/github.com/containers/image/copy.Image(0x1589760, 0xc000040150, 0xc0002f1f00, 0x1595ac0, 0xc00021df80, 0x1595dc0, 0xc0004ccc00, 0xc0004c6360, 0x0, 0x0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:177 +0x8abgithub.com/containers/libpod/libpod/image.(Runtime).doPullImage(0xc0002f0f40, 0x1589760, 0xc000040150, 0xc000133760, 0xc0001f4120, 0x1, 0x1, 0x0, 0x0, 0x0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/libpod/image/pull.go:241 +0x2cagithub.com/containers/libpod/libpod/image.(Runtime).pullImageFromHeuristicSource(0xc0002f0f40, 0x1589760, 0xc000040150, 0x7ffc24866b5c, 0x1b, 0x15776c0, 0xc00000e018, 0x0, 0x0, 0x0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/libpod/image/pull.go:210 +0x1c4github.com/containers/libpod/libpod/image.(Runtime).New(0xc0002f0f40, 0x1589760, 0xc000040150, 0x7ffc24866b5c, 0x1b, 0x0, 0x0, 0x0, 0x0, 0x15776c0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/libpod/image/image.go:148 +0x18fgithub.com/containers/libpod/cmd/podman/shared.GetRunlabel(0x7ffc24866b58, 0x3, 0x7ffc24866b5c, 0x1b, 0x1589760, 0xc000040150, 0xc0000e4300, 0xc00018e001, 0x0, 0x0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/cmd/podman/shared/container.go:624 +0x163main.runlabelCmd(0xc000132c60, 0x0, 0x0) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/cmd/podman/runlabel.go:164 +0x600github.com/containers/libpod/vendor/github.com/urfave/cli.HandleAction(0x11e3160, 0x14474c0, 0xc000132c60, 0x0, 0xc00021c660) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/urfave/cli/app.go:501 +0xc8github.com/containers/libpod/vendor/github.com/urfave/cli.Command.Run(0x13ad6b9, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x13edc47, 0x2f, 0x0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/urfave/cli/command.go:165 +0x459github.com/containers/libpod/vendor/github.com/urfave/cli.(App).RunAsSubcommand(0xc00018ce00, 0xc0001326e0, 0x0, 0x0) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/urfave/cli/app.go:383 +0x827github.com/containers/libpod/vendor/github.com/urfave/cli.Command.startApp(0x13ae6a1, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x13b9522, 0x11, 0x0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/urfave/cli/command.go:377 +0x808github.com/containers/libpod/vendor/github.com/urfave/cli.Command.Run(0x13ae6a1, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x13b9522, 0x11, 0x0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/urfave/cli/command.go:103 +0x80fgithub.com/containers/libpod/vendor/github.com/urfave/cli.(App).Run(0xc00018cc40, 0xc0000301e0, 0x6, 0x6, 0x0, 0x0) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/urfave/cli/app.go:259 +0x6bbmain.main() /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/cmd/podman/main.go:273 +0x15a6 goroutine 5 [chan receive]:github.com/containers/libpod/vendor/github.com/golang/glog.(loggingT).flushDaemon(0x20fc320) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/golang/glog/glog.go:882 +0x8bcreated by github.com/containers/libpod/vendor/github.com/golang/glog.init.0 /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/golang/glog/glog.go:410 +0x203 goroutine 6 [syscall]:os/signal.signal_recv(0x0) /usr/lib/golang/src/runtime/sigqueue.go:139 +0x9cos/signal.loop() /usr/lib/golang/src/os/signal/signal_unix.go:23 +0x22created by os/signal.init.0 /usr/lib/golang/src/os/signal/signal_unix.go:29 +0x41 goroutine 58 [runnable]:syscall.openat(0xffffffffffffff9c, 0xc0005555c0, 0x1b, 0x802c1, 0xc000000180, 0x1b, 0xc0006e7228, 0x53fb0f) /usr/lib/golang/src/syscall/zsyscall_linux_amd64.go:62 +0x184syscall.Open(0xc0005555c0, 0x1b, 0x802c1, 0xc000000180, 0xc7b14f, 0xc0006e7280, 0x2) /usr/lib/golang/src/syscall/syscall_linux.go:138 +0x50os.openFileNolog(0xc0005555c0, 0x1b, 0x2c1, 0xc000000180, 0x19, 0x1f78bf1, 0x1) /usr/lib/golang/src/os/file_unix.go:190 +0x85os.OpenFile(0xc0005555c0, 0x1b, 0x2c1, 0xc000000180, 0xc0000b6568, 0x1, 0x20fd0c0) /usr/lib/golang/src/os/file.go:284 +0x5fgithub.com/containers/libpod/vendor/github.com/containers/image/storage.(storageImageDestination).PutBlob(0xc00019a620, 0x1589760, 0xc000040150, 0x1577080, 0xc00056a200, 0xc0003328c0, 0x47, 0x351, 0x0, 0x0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/storage/storage_image.go:362 +0x18cgithub.com/containers/libpod/vendor/github.com/containers/image/copy.(copier).copyBlobFromStream(0xc0004c6b40, 0x1589760, 0xc000040150, 0x15774e0, 0xc0002c6c00, 0xc0003328c0, 0x47, 0x351, 0x0, 0x0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:786 +0x354github.com/containers/libpod/vendor/github.com/containers/image/copy.(imageCopier).copyLayerFromStream(0xc00047c2d0, 0x1589760, 0xc000040150, 0x15774e0, 0xc0002c6c00, 0xc0003328c0, 0x47, 0x351, 0x0, 0x0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:672 +0x19egithub.com/containers/libpod/vendor/github.com/containers/image/copy.(imageCopier).copyLayer(0xc00047c2d0, 0x1589760, 0xc000040150, 0xc0003328c0, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:619 +0x30fgithub.com/containers/libpod/vendor/github.com/containers/image/copy.(imageCopier).copyLayers.func1(0x1, 0xc0003328c0, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:465 +0x632created by github.com/containers/libpod/vendor/github.com/containers/image/copy.(imageCopier).copyLayers /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:484 +0x66e goroutine 109 [runnable]:internal/poll.runtime_pollWait(0x7efeac279e30, 0x72, 0xc0006a1858) /usr/lib/golang/src/runtime/netpoll.go:173 +0x66internal/poll.(pollDesc).wait(0xc0000eaa98, 0x72, 0xffffffffffffff00, 0x15790a0, 0x1f71a40) /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:85 +0x9ainternal/poll.(pollDesc).waitRead(0xc0000eaa98, 0xc000025000, 0x1000, 0x1000) /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:90 +0x3dinternal/poll.(FD).Read(0xc0000eaa80, 0xc000025000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/lib/golang/src/internal/poll/fd_unix.go:169 +0x179net.(netFD).Read(0xc0000eaa80, 0xc000025000, 0x1000, 0x1000, 0x413e7b, 0xc000532000, 0x124c9c0) /usr/lib/golang/src/net/fd_unix.go:202 +0x4fnet.(conn).Read(0xc00000e050, 0xc000025000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/lib/golang/src/net/net.go:177 +0x68crypto/tls.(block).readFromUntil(0xc000345080, 0x1577400, 0xc00000e050, 0x5, 0xc00000e050, 0xdb470d01b74b8161) /usr/lib/golang/src/crypto/tls/conn.go:492 +0x89crypto/tls.(Conn).readRecord(0xc00009d500, 0x1447d17, 0xc00009d620, 0x20fc140) /usr/lib/golang/src/crypto/tls/conn.go:593 +0xddcrypto/tls.(Conn).Read(0xc00009d500, 0xc00076c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/lib/golang/src/crypto/tls/conn.go:1145 +0xf1net/http.(persistConn).Read(0xc0000e05a0, 0xc00076c000, 0x1000, 0x1000, 0x40fda3, 0xc000384470, 0x434fd2) /usr/lib/golang/src/net/http/transport.go:1497 +0x75bufio.(Reader).fill(0xc0000303c0) /usr/lib/golang/src/bufio/bufio.go:100 +0x10fbufio.(Reader).Peek(0xc0000303c0, 0x1, 0xc00004a4e0, 0xc000384558, 0x0, 0x0, 0xc00009c700) /usr/lib/golang/src/bufio/bufio.go:132 +0x3fnet/http.(persistConn).readLoop(0xc0000e05a0) /usr/lib/golang/src/net/http/transport.go:1645 +0x1a2created by net/http.(Transport).dialConn /usr/lib/golang/src/net/http/transport.go:1338 +0x941 goroutine 55 [select]:github.com/containers/libpod/vendor/gopkg.in/cheggaaa/pb%2ev1.catchTerminate(0xc0002a7020) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/gopkg.in/cheggaaa/pb.v1/pb_x.go:112 +0x181created by github.com/containers/libpod/vendor/gopkg.in/cheggaaa/pb%2ev1.lockEcho /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/gopkg.in/cheggaaa/pb.v1/pb_x.go:83 +0x315 goroutine 56 [select]:github.com/containers/libpod/vendor/gopkg.in/cheggaaa/pb%2ev1.(Pool).writer(0xc000030a80) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/gopkg.in/cheggaaa/pb.v1/pool.go:77 +0xf9created by github.com/containers/libpod/vendor/gopkg.in/cheggaaa/pb%2ev1.(Pool).Start /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/gopkg.in/cheggaaa/pb.v1/pool.go:60 +0xba goroutine 57 [select]:net/http.(persistConn).roundTrip(0xc0000e06c0, 0xc00047cbd0, 0x0, 0x0, 0x0) /usr/lib/golang/src/net/http/transport.go:2101 +0x56anet/http.(Transport).roundTrip(0xc00035ed80, 0xc0000ee200, 0x1575a20, 0xc000080040, 0x4ed) /usr/lib/golang/src/net/http/transport.go:465 +0x9b1net/http.(Transport).RoundTrip(0xc00035ed80, 0xc0000ee200, 0xc00035ed80, 0x0, 0x0) /usr/lib/golang/src/net/http/roundtrip.go:17 +0x35net/http.send(0xc0000ee200, 0x15774a0, 0xc00035ed80, 0x0, 0x0, 0x0, 0xc0006ae2a0, 0x8537fb, 0xc000093350, 0x2) /usr/lib/golang/src/net/http/client.go:250 +0x14bnet/http.(Client).send(0xc0001f4f30, 0xc0000ee200, 0x0, 0x0, 0x0, 0xc0006ae2a0, 0x1, 0x2, 0x0) /usr/lib/golang/src/net/http/client.go:174 +0xfanet/http.(Client).do(0xc0001f4f30, 0xc0004f5000, 0x0, 0x0, 0x0) /usr/lib/golang/src/net/http/client.go:641 +0x2a8net/http.(Client).Do(0xc0001f4f30, 0xc0004f5000, 0xc0006fb720, 0x2, 0x2) /usr/lib/golang/src/net/http/client.go:509 +0x35github.com/containers/libpod/vendor/github.com/containers/image/docker.(dockerClient).makeRequestToResolvedURL(0xc0003a01e0, 0x1589760, 0xc000040150, 0x13a685f, 0x3, 0xc00053c460, 0x6d, 0x0, 0x0, 0x0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/docker/docker_client.go:449 +0x3dcgithub.com/containers/libpod/vendor/github.com/containers/image/docker.(dockerClient).makeRequest(0xc0003a01e0, 0x1589760, 0xc000040150, 0x13a685f, 0x3, 0xc0002a75c0, 0x5e, 0x0, 0x0, 0x0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/docker/docker_client.go:418 +0x247github.com/containers/libpod/vendor/github.com/containers/image/docker.(dockerImageSource).GetBlob(0xc0002c6e80, 0x1589760, 0xc000040150, 0xc000332910, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/docker/docker_image_src.go:179 +0x267github.com/containers/libpod/vendor/github.com/containers/image/copy.(imageCopier).copyLayer(0xc00047c2d0, 0x1589760, 0xc000040150, 0xc000332910, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:613 +0x16egithub.com/containers/libpod/vendor/github.com/containers/image/copy.(imageCopier).copyLayers.func1(0x0, 0xc000332910, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:465 +0x632created by github.com/containers/libpod/vendor/github.com/containers/image/copy.(imageCopier).copyLayers /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:484 +0x66e goroutine 94 [select]:net/http.(persistConn).readLoop(0xc0001f8000) /usr/lib/golang/src/net/http/transport.go:1761 +0x6b9created by net/http.(Transport).dialConn /usr/lib/golang/src/net/http/transport.go:1338 +0x941 goroutine 59 [select]:net/http.(persistConn).roundTrip(0xc00023e360, 0xc0002bd560, 0x0, 0x0, 0x0) /usr/lib/golang/src/net/http/transport.go:2101 +0x56anet/http.(Transport).roundTrip(0xc00035ed80, 0xc0004f4000, 0x1575a20, 0xc000080040, 0x4ed) /usr/lib/golang/src/net/http/transport.go:465 +0x9b1net/http.(Transport).RoundTrip(0xc00035ed80, 0xc0004f4000, 0xc00035ed80, 0x0, 0x0) /usr/lib/golang/src/net/http/roundtrip.go:17 +0x35net/http.send(0xc0004f4000, 0x15774a0, 0xc00035ed80, 0x0, 0x0, 0x0, 0xc000305660, 0x8537fb, 0xc0004d2310, 0x2) /usr/lib/golang/src/net/http/client.go:250 +0x14bnet/http.(Client).send(0xc0001f4f30, 0xc0004f4000, 0x0, 0x0, 0x0, 0xc000305660, 0x1, 0x2, 0x0) /usr/lib/golang/src/net/http/client.go:174 +0xfanet/http.(Client).do(0xc0001f4f30, 0xc0004f4e00, 0x0, 0x0, 0x0) /usr/lib/golang/src/net/http/client.go:641 +0x2a8net/http.(Client).Do(0xc0001f4f30, 0xc0004f4e00, 0xc000625720, 0x2, 0x2) /usr/lib/golang/src/net/http/client.go:509 +0x35github.com/containers/libpod/vendor/github.com/containers/image/docker.(dockerClient).makeRequestToResolvedURL(0xc0003a01e0, 0x1589760, 0xc000040150, 0x13a685f, 0x3, 0xc00053c3f0, 0x6d, 0x0, 0x0, 0x0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/docker/docker_client.go:449 +0x3dcgithub.com/containers/libpod/vendor/github.com/containers/image/docker.(dockerClient).makeRequest(0xc0003a01e0, 0x1589760, 0xc000040150, 0x13a685f, 0x3, 0xc0002a73e0, 0x5e, 0x0, 0x0, 0x0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/docker/docker_client.go:418 +0x247github.com/containers/libpod/vendor/github.com/containers/image/docker.(dockerImageSource).GetBlob(0xc0002c6e80, 0x1589760, 0xc000040150, 0xc000332870, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/docker/docker_image_src.go:179 +0x267github.com/containers/libpod/vendor/github.com/containers/image/copy.(imageCopier).copyLayer(0xc00047c2d0, 0x1589760, 0xc000040150, 0xc000332870, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:613 +0x16egithub.com/containers/libpod/vendor/github.com/containers/image/copy.(imageCopier).copyLayers.func1(0x2, 0xc000332870, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:465 +0x632created by github.com/containers/libpod/vendor/github.com/containers/image/copy.(imageCopier).copyLayers /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:484 +0x66e goroutine 60 [select]:net/http.(persistConn).roundTrip(0xc0000e05a0, 0xc00047c570, 0x0, 0x0, 0x0) /usr/lib/golang/src/net/http/transport.go:2101 +0x56anet/http.(Transport).roundTrip(0xc00035ed80, 0xc0000ee000, 0x1575a20, 0xc000080040, 0x4ed) /usr/lib/golang/src/net/http/transport.go:465 +0x9b1net/http.(Transport).RoundTrip(0xc00035ed80, 0xc0000ee000, 0xc00035ed80, 0x0, 0x0) /usr/lib/golang/src/net/http/roundtrip.go:17 +0x35net/http.send(0xc0000ee000, 0x15774a0, 0xc00035ed80, 0x0, 0x0, 0x0, 0xc0006ae1d0, 0x8537fb, 0xc000093050, 0x2) /usr/lib/golang/src/net/http/client.go:250 +0x14bnet/http.(Client).send(0xc0001f4f30, 0xc0000ee000, 0x0, 0x0, 0x0, 0xc0006ae1d0, 0x1, 0x2, 0x0) /usr/lib/golang/src/net/http/client.go:174 +0xfanet/http.(Client).do(0xc0001f4f30, 0xc0000ee600, 0x0, 0x0, 0x0) /usr/lib/golang/src/net/http/client.go:641 +0x2a8net/http.(Client).Do(0xc0001f4f30, 0xc0000ee600, 0xc000627720, 0x2, 0x2) /usr/lib/golang/src/net/http/client.go:509 +0x35github.com/containers/libpod/vendor/github.com/containers/image/docker.(dockerClient).makeRequestToResolvedURL(0xc0003a01e0, 0x1589760, 0xc000040150, 0x13a685f, 0x3, 0xc0002e4310, 0x6d, 0x0, 0x0, 0x0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/docker/docker_client.go:449 +0x3dcgithub.com/containers/libpod/vendor/github.com/containers/image/docker.(dockerClient).makeRequest(0xc0003a01e0, 0x1589760, 0xc000040150, 0x13a685f, 0x3, 0xc00004afc0, 0x5e, 0x0, 0x0, 0x0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/docker/docker_client.go:418 +0x247github.com/containers/libpod/vendor/github.com/containers/image/docker.(dockerImageSource).GetBlob(0xc0002c6e80, 0x1589760, 0xc000040150, 0xc000332820, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/docker/docker_image_src.go:179 +0x267github.com/containers/libpod/vendor/github.com/containers/image/copy.(imageCopier).copyLayer(0xc00047c2d0, 0x1589760, 0xc000040150, 0xc000332820, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:613 +0x16egithub.com/containers/libpod/vendor/github.com/containers/image/copy.(imageCopier).copyLayers.func1(0x3, 0xc000332820, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:465 +0x632created by github.com/containers/libpod/vendor/github.com/containers/image/copy.(imageCopier).copyLayers /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:484 +0x66e goroutine 62 [select]:net/http.(persistConn).roundTrip(0xc00023ec60, 0xc0002bdda0, 0x0, 0x0, 0x0) /usr/lib/golang/src/net/http/transport.go:2101 +0x56anet/http.(Transport).roundTrip(0xc00035ed80, 0xc0004f4100, 0x1575a20, 0xc000080040, 0x4ed) /usr/lib/golang/src/net/http/transport.go:465 +0x9b1net/http.(Transport).RoundTrip(0xc00035ed80, 0xc0004f4100, 0xc00035ed80, 0x0, 0x0) /usr/lib/golang/src/net/http/roundtrip.go:17 +0x35net/http.send(0xc0004f4100, 0x15774a0, 0xc00035ed80, 0x0, 0x0, 0x0, 0xc000305730, 0x8537fb, 0xc0004d2490, 0x2) /usr/lib/golang/src/net/http/client.go:250 +0x14bnet/http.(Client).send(0xc0001f4f30, 0xc0004f4100, 0x0, 0x0, 0x0, 0xc000305730, 0x1, 0x2, 0x0) /usr/lib/golang/src/net/http/client.go:174 +0xfanet/http.(Client).do(0xc0001f4f30, 0xc0004f4c00, 0x0, 0x0, 0x0) /usr/lib/golang/src/net/http/client.go:641 +0x2a8net/http.(Client).Do(0xc0001f4f30, 0xc0004f4c00, 0xc000393720, 0x2, 0x2) /usr/lib/golang/src/net/http/client.go:509 +0x35github.com/containers/libpod/vendor/github.com/containers/image/docker.(dockerClient).makeRequestToResolvedURL(0xc0003a01e0, 0x1589760, 0xc000040150, 0x13a685f, 0x3, 0xc00053c380, 0x6d, 0x0, 0x0, 0x0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/docker/docker_client.go:449 +0x3dcgithub.com/containers/libpod/vendor/github.com/containers/image/docker.(dockerClient).makeRequest(0xc0003a01e0, 0x1589760, 0xc000040150, 0x13a685f, 0x3, 0xc0002a7140, 0x5e, 0x0, 0x0, 0x0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/docker/docker_client.go:418 +0x247github.com/containers/libpod/vendor/github.com/containers/image/docker.(dockerImageSource).GetBlob(0xc0002c6e80, 0x1589760, 0xc000040150, 0xc000332780, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/docker/docker_image_src.go:179 +0x267github.com/containers/libpod/vendor/github.com/containers/image/copy.(imageCopier).copyLayer(0xc00047c2d0, 0x1589760, 0xc000040150, 0xc000332780, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:613 +0x16egithub.com/containers/libpod/vendor/github.com/containers/image/copy.(imageCopier).copyLayers.func1(0x5, 0xc000332780, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:465 +0x632created by github.com/containers/libpod/vendor/github.com/containers/image/copy.(imageCopier).copyLayers /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:484 +0x66e goroutine 95 [select]:net/http.(persistConn).writeLoop(0xc0001f8000) /usr/lib/golang/src/net/http/transport.go:1885 +0x113created by net/http.(Transport).dialConn /usr/lib/golang/src/net/http/transport.go:1339 +0x966 goroutine 110 [select]:net/http.(persistConn).writeLoop(0xc0000e05a0) /usr/lib/golang/src/net/http/transport.go:1885 +0x113created by net/http.(Transport).dialConn /usr/lib/golang/src/net/http/transport.go:1339 +0x966 goroutine 146 [select]:net/http.(persistConn).writeLoop(0xc00035e7e0) /usr/lib/golang/src/net/http/transport.go:1885 +0x113created by net/http.(Transport).dialConn /usr/lib/golang/src/net/http/transport.go:1339 +0x966 goroutine 97 [select]:net/http.(persistConn).writeLoop(0xc00023ec60) /usr/lib/golang/src/net/http/transport.go:1885 +0x113created by net/http.(Transport).dialConn /usr/lib/golang/src/net/http/transport.go:1339 +0x966 goroutine 111 [runnable]:syscall.Syscall(0x0, 0xf, 0xc0006c3000, 0x1000, 0x1000, 0x1000, 0x0) /usr/lib/golang/src/syscall/asm_linux_amd64.s:18 +0x5syscall.read(0xf, 0xc0006c3000, 0x1000, 0x1000, 0xffffffffffffff00, 0x0, 0x0) /usr/lib/golang/src/syscall/zsyscall_linux_amd64.go:732 +0x5asyscall.Read(0xf, 0xc0006c3000, 0x1000, 0x1000, 0xffffffffffffffff, 0x15790a0, 0x1f71a40) /usr/lib/golang/src/syscall/syscall_unix.go:172 +0x49internal/poll.(FD).Read(0xc00034c680, 0xc0006c3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/lib/golang/src/internal/poll/fd_unix.go:165 +0x119net.(netFD).Read(0xc00034c680, 0xc0006c3000, 0x1000, 0x1000, 0x413e7b, 0xc000532000, 0x124c9c0) /usr/lib/golang/src/net/fd_unix.go:202 +0x4fnet.(conn).Read(0xc00000e060, 0xc0006c3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/lib/golang/src/net/net.go:177 +0x68crypto/tls.(block).readFromUntil(0xc000345aa0, 0x1577400, 0xc00000e060, 0x5, 0xc00000e060, 0x11c2940) /usr/lib/golang/src/crypto/tls/conn.go:492 +0x89crypto/tls.(Conn).readRecord(0xc000288000, 0x1447d17, 0xc000288120, 0x20fc140) /usr/lib/golang/src/crypto/tls/conn.go:593 +0xddcrypto/tls.(Conn).Read(0xc000288000, 0xc000600000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/lib/golang/src/crypto/tls/conn.go:1145 +0xf1net/http.(persistConn).Read(0xc0000e06c0, 0xc000600000, 0x1000, 0x1000, 0x40fda3, 0xc00038a470, 0x434fd2) /usr/lib/golang/src/net/http/transport.go:1497 +0x75bufio.(Reader).fill(0xc0000307e0) /usr/lib/golang/src/bufio/bufio.go:100 +0x10fbufio.(Reader).Peek(0xc0000307e0, 0x1, 0xc0000aa5a0, 0xc00038a558, 0x0, 0x3a, 0xffffffffffffffff) /usr/lib/golang/src/bufio/bufio.go:132 +0x3fnet/http.(persistConn).readLoop(0xc0000e06c0) /usr/lib/golang/src/net/http/transport.go:1645 +0x1a2created by net/http.(Transport).dialConn /usr/lib/golang/src/net/http/transport.go:1338 +0x941 goroutine 96 [runnable]:internal/poll.runtime_pollWait(0x7efeac279950, 0x72, 0xc0006a3858) /usr/lib/golang/src/runtime/netpoll.go:173 +0x66internal/poll.(pollDesc).wait(0xc0005f6618, 0x72, 0xffffffffffffff00, 0x15790a0, 0x1f71a40) /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:85 +0x9ainternal/poll.(pollDesc).waitRead(0xc0005f6618, 0xc000642000, 0x1000, 0x1000) /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:90 +0x3dinternal/poll.(FD).Read(0xc0005f6600, 0xc000642000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/lib/golang/src/internal/poll/fd_unix.go:169 +0x179net.(netFD).Read(0xc0005f6600, 0xc000642000, 0x1000, 0x1000, 0x413e7b, 0xc000532000, 0x124c9c0) /usr/lib/golang/src/net/fd_unix.go:202 +0x4fnet.(conn).Read(0xc0002ea078, 0xc000642000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/lib/golang/src/net/net.go:177 +0x68crypto/tls.(block).readFromUntil(0xc0001f53e0, 0x1577400, 0xc0002ea078, 0x5, 0xc0002ea078, 0x21) /usr/lib/golang/src/crypto/tls/conn.go:492 +0x89crypto/tls.(Conn).readRecord(0xc000179500, 0x1447d17, 0xc000179620, 0x7efeac2835d0) /usr/lib/golang/src/crypto/tls/conn.go:593 +0xddcrypto/tls.(Conn).Read(0xc000179500, 0xc000773000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/lib/golang/src/crypto/tls/conn.go:1145 +0xf1net/http.(persistConn).Read(0xc00023ec60, 0xc000773000, 0x1000, 0x1000, 0x40fda3, 0xc0006a3c70, 0x434fd2) /usr/lib/golang/src/net/http/transport.go:1497 +0x75bufio.(Reader).fill(0xc00021cf00) /usr/lib/golang/src/bufio/bufio.go:100 +0x10fbufio.(Reader).Peek(0xc00021cf00, 0x1, 0xc0002a6000, 0xc000384558, 0x0, 0x0, 0xc00009c700) /usr/lib/golang/src/bufio/bufio.go:132 +0x3fnet/http.(persistConn).readLoop(0xc00023ec60) /usr/lib/golang/src/net/http/transport.go:1645 +0x1a2created by net/http.(Transport).dialConn /usr/lib/golang/src/net/http/transport.go:1338 +0x941 goroutine 112 [select]:net/http.(persistConn).writeLoop(0xc0000e06c0) /usr/lib/golang/src/net/http/transport.go:1885 +0x113created by net/http.(Transport).dialConn /usr/lib/golang/src/net/http/transport.go:1339 +0x966 goroutine 131 [select]:net/http.(persistConn).writeLoop(0xc00023e360) /usr/lib/golang/src/net/http/transport.go:1885 +0x113created by net/http.(Transport).dialConn /usr/lib/golang/src/net/http/transport.go:1339 +0x966 goroutine 113 [select]:net/http.(persistConn).readLoop(0xc00035e7e0) /usr/lib/golang/src/net/http/transport.go:1761 +0x6b9created by net/http.(Transport).dialConn /usr/lib/golang/src/net/http/transport.go:1338 +0x941 goroutine 130 [runnable]:internal/poll.runtime_pollWait(0x7efeac279af0, 0x72, 0xc000660858) /usr/lib/golang/src/runtime/netpoll.go:173 +0x66internal/poll.(pollDesc).wait(0xc00034c298, 0x72, 0xffffffffffffff00, 0x15790a0, 0x1f71a40) /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:85 +0x9ainternal/poll.(pollDesc).waitRead(0xc00034c298, 0xc0005ce000, 0x1000, 0x1000) /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:90 +0x3dinternal/poll.(FD).Read(0xc00034c280, 0xc0005ce000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/lib/golang/src/internal/poll/fd_unix.go:169 +0x179net.(netFD).Read(0xc00034c280, 0xc0005ce000, 0x1000, 0x1000, 0x413e7b, 0xc000532000, 0x124c9c0) /usr/lib/golang/src/net/fd_unix.go:202 +0x4fnet.(conn).Read(0xc00000e058, 0xc0005ce000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/lib/golang/src/net/net.go:177 +0x68crypto/tls.(block).readFromUntil(0xc0003d6810, 0x1577400, 0xc00000e058, 0x5, 0xc00000e058, 0xc0005796e0) /usr/lib/golang/src/crypto/tls/conn.go:492 +0x89crypto/tls.(Conn).readRecord(0xc00009dc00, 0x1447d17, 0xc00009dd20, 0x7efeac1f2ee8) /usr/lib/golang/src/crypto/tls/conn.go:593 +0xddcrypto/tls.(Conn).Read(0xc00009dc00, 0xc0005f9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/lib/golang/src/crypto/tls/conn.go:1145 +0xf1net/http.(persistConn).Read(0xc00023e360, 0xc0005f9000, 0x1000, 0x1000, 0x40fda3, 0xc000660c70, 0x434fd2) /usr/lib/golang/src/net/http/transport.go:1497 +0x75bufio.(Reader).fill(0xc00021d5c0) /usr/lib/golang/src/bufio/bufio.go:100 +0x10fbufio.(Reader).Peek(0xc00021d5c0, 0x1, 0xc0005aa060, 0xc000388558, 0x0, 0x3a, 0xffffffffffffffff) /usr/lib/golang/src/bufio/bufio.go:132 +0x3fnet/http.(persistConn).readLoop(0xc00023e360) /usr/lib/golang/src/net/http/transport.go:1645 +0x1a2created by net/http.(Transport).dialConn /usr/lib/golang/src/net/http/transport.go:1338 +0x941

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

$ podman version Version: 1.0.0 Go Version: go1.11.4 Git Commit: "49780a1cf10d572edc4e1ea3b8a8429ce391d47d" Built: Mon Jan 14 20:38:17 2019 OS/Arch: linux/amd64

Output of podman info:

$ podman info host: BuildahVersion: 1.6-dev Conmon: package: podman-1.0.0-1.git82e8011.fc29.x86_64 path: /usr/libexec/podman/conmon version: 'conmon version 1.12.0-dev, commit: 49780a1cf10d572edc4e1ea3b8a8429ce391d47d' Distribution: distribution: fedora version: "29" MemFree: 11358875648 MemTotal: 20422283264 OCIRuntime: package: runc-1.0.0-68.dev.git6635b4f.fc29.x86_64 path: /usr/bin/runc version: |- runc version 1.0.0-rc6+dev commit: ef9132178ccc3d2775d4fb51f1e431f30cac1398-dirty spec: 1.0.1-dev SwapFree: 10280235008 SwapTotal: 10280235008 arch: amd64 cpus: 4 hostname: t460s kernel: 4.20.10-200.fc29.x86_64 os: linux rootless: true uptime: 13h 38m 29.63s (Approximately 0.54 days) insecure registries: registries: [] registries: registries:

  • docker.io
  • registry.fedoraproject.org
  • quay.io
  • registry.access.redhat.com
  • registry.centos.org store: ConfigFile: /home/aim/.config/containers/storage.conf ContainerStore: number: 1 GraphDriverName: overlay GraphOptions:
  • overlay.mount_program=/usr/bin/fuse-overlayfs GraphRoot: /home/aim/.local/share/containers/storage GraphStatus: Backing Filesystem: extfs Native Overlay Diff: "false" Supports d_type: "true" ImageStore: number: 5 RunRoot: /run/user/1000

Additional environment details (AWS, VirtualBox, physical, etc.):

Fedora 29 - and up to date as of Sun Feb 24 07:29:39 GMT 2019.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/containers/libpod/issues/2420, or mute the thread https://github.com/notifications/unsubscribe-auth/AHYHCIMx9lrC_s5JUFkkrEFTK7e0UUxKks5vQkBygaJpZM4bOY_- .

rhatdan commented 5 years ago

@frobware Could you see if this works with master branch?

frobware commented 5 years ago

@frobware Could you see if this works with master branch?

Will take a look later in my day. Given the nature it wasn't always reproducible.

vrothberg commented 5 years ago

I don't think that the progress bars are causing the issue. The panic() seems to be caused in the blobinfocache.

goroutine 61 [running]:
runtime.throw(0x13bf36b, 0x15)
    /usr/lib/golang/src/runtime/panic.go:608 +0x72 fp=0xc0006f7708 sp=0xc0006f76d8 pc=0x436272
runtime.mapassign(0x120c120, 0xc0001f5080, 0xc0006f7838, 0x211cfa0)
    /usr/lib/golang/src/runtime/map.go:651 +0x545 fp=0xc0006f7790 sp=0xc0006f7708 pc=0x4194b5
github.com/containers/libpod/vendor/github.com/containers/image/pkg/blobinfocache.(*memoryCache).RecordKnownLocation(0xc0004ceee0, 0x1587c20, 0x211b5f8, 0xc0002d2ce0, 0x7, 0xc0003327d0, 0x47, 0xc00056f360, 0x14)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/pkg/blobinfocache/memory.go:79 +0x20e fp=0xc0006f7878 sp=0xc0006f7790 pc=0xbeadde
github.com/containers/libpod/vendor/github.com/containers/image/docker.(*dockerImageSource).GetBlob(0xc0002c6e80, 0x1589760, 0xc000040150, 0xc0003327d0, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/docker/docker_image_src.go:187 +0x49b fp=0xc0006f79b0 sp=0xc0006f7878 pc=0xccf36b
github.com/containers/libpod/vendor/github.com/containers/image/copy.(*imageCopier).copyLayer(0xc00047c2d0, 0x1589760, 0xc000040150, 0xc0003327d0, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:613 +0x16e fp=0xc0006f7d58 sp=0xc0006f79b0 pc=0xca81de
github.com/containers/libpod/vendor/github.com/containers/image/copy.(*imageCopier).copyLayers.func1(0x4, 0xc0003327d0, 0x47, 0xffffffffffffffff, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:465 +0x632 fp=0xc0006f7f88 sp=0xc0006f7d58 pc=0xcacc02
runtime.goexit()
    /usr/lib/golang/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0006f7f90 sp=0xc0006f7f88 pc=0x4641d1
created by github.com/containers/libpod/vendor/github.com/containers/image/copy.(*imageCopier).copyLayers
    /builddir/build/BUILD/libpod-82e80110c3f2d8728745c47e340f3bee4d408846/_build/src/github.com/containers/libpod/vendor/github.com/containers/image/copy/copy.go:484 +0x66e
vrothberg commented 5 years ago

Yes, that's a bug in blobinfocache and subject to a race condition which explains that it's hard to reproduce. I'll prepare a fix for containers/image. Thanks for the detailed report, @frobware!

rhatdan commented 5 years ago

Is this something we should hold up 1.1 release for?

vrothberg commented 5 years ago

Yes, let's hurry and get the fix in.

edsantiago commented 5 years ago

Nice timing: I just experienced this as a one-off, about 10 minutes ago, podman-1.0.0-1.git82e8011.fc29 rootless on a pull. First and only time I've seen it. Yes, I'd say this would be a good one to get into 1.1. Thank you for fixing this before I had a chance to file a bug report!