search

moby / buildkit

concurrent, cache-efficient, and Dockerfile-agnostic builder toolkit
https://github.com/moby/moby/issues/34227
Apache License 2.0
8.05k stars 1.13k forks source link

Hang in random places #1915

Open xkszltl opened 3 years ago

xkszltl commented 3 years ago

After a recent update of docker-ce, we're getting hanging issue is buildkit builds. With a high chance, buildkit build will freeze in the middle, sometimes even in the middle of a line. CPU was idle in this case. We try to ps for expected process running in docker from host, but nothing found.

We tend to find this after heavy compilation, with a lot of I/O and logging. Is it related to BUILDKIT_STEP_LOG_MAX_SIZE and BUILDKIT_STEP_LOG_MAX_SPEED recently added (BTW this should be documented). We set them to 1GB and 1MB in docker's systemd env var, because we expect 200+MB of logs.

tonistiigi commented 3 years ago

What makes you think this is caused by the log clipping. Does the hang happen outside of buildkit on some tool you use to process the logs.

More discussion https://github.com/docker/buildx/issues/482

If you get a hang send SIGQUIT to the buildkitd process and report back the stacktrace.

xkszltl commented 3 years ago

There's no tool I used other than shell. It's simply screen log (inside bash+tmux+ssh to be exact, which I never had issue with).

I'll try the SIGQUIT next time. I'm using docker-ce, does this mean sudo killall -s QUIT dockerd and the log will be available through systemctl status docker?

tonistiigi commented 3 years ago

In dockerd you can also send SIGUSR1 signal that will not close down the process when logging the trace.

Make sure to also post the exact version you are using.

xkszltl commented 3 years ago

@tonistiigi

Version

$ rpm -qa docker* *container*
nvidia-container-runtime-3.4.0-1.x86_64
containerd.io-1.4.3-3.1.el7.x86_64
plexus-containers-container-default-1.5.5-14.el7.noarch
docker-ce-20.10.2-3.el7.x86_64
container-selinux-2.119.2-1.911c772.el7_8.noarch
docker-devel-1.3.2-4.el7.centos.x86_64
ghc-containers-0.5.0.0-26.4.el7.x86_64
libnvidia-container1-1.3.1-1.x86_64
containers-common-0.1.40-11.el7_8.x86_64
plexus-containers-component-metadata-1.5.5-14.el7.noarch
container-storage-setup-0.11.0-2.git5eaf76c.el7.noarch
docker-ce-rootless-extras-20.10.2-3.el7.x86_64
docker-ce-cli-20.10.2-3.el7.x86_64
ghc-unordered-containers-0.2.3.0-3.el7.x86_64
nvidia-container-toolkit-1.4.0-2.x86_64
plexus-containers-component-annotations-1.5.5-14.el7.noarch
docker-ce-debuginfo-18.06.3.ce-3.el7.x86_64
libnvidia-container-tools-1.3.1-1.x86_64
docker-unit-test-1.13.1-68.gitdded712.el7.centos.x86_64
docker-distribution-2.6.2-2.git48294d9.el7.x86_64

Stacktrace

goroutine 139 [running]:
github.com/docker/docker/pkg/signal.DumpStacks(0x55d5a5576870, 0xf, 0x0, 0x0, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/pkg/signal/trap.go:83 +0x9d
github.com/docker/docker/daemon.(*Daemon).setupDumpStackTrap.func1(0xc000051560, 0x55d5a5576870, 0xf)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/debugtrap_unix.go:19 +0x74
created by github.com/docker/docker/daemon.(*Daemon).setupDumpStackTrap
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/debugtrap_unix.go:17 +0xc1

goroutine 1 [chan receive, 2108 minutes]:
main.(*DaemonCli).start(0xc000933500, 0xc000050a20, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/cmd/dockerd/daemon.go:249 +0xc04
main.runDaemon(...)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/cmd/dockerd/docker_unix.go:13
main.newDaemonCommand.func1(0xc00020eb00, 0xc0009334a0, 0x0, 0x3, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/cmd/dockerd/docker.go:34 +0x7c
github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).execute(0xc00020eb00, 0xc00004e0d0, 0x3, 0x3, 0xc00020eb00, 0xc00004e0d0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:850 +0x462
github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc00020eb00, 0x0, 0x0, 0x10)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:958 +0x34b
github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).Execute(...)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:895
main.main()
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/cmd/dockerd/docker.go:97 +0x191

goroutine 6 [syscall]:
os/signal.signal_recv(0x55d5a6944340)
    /usr/local/go/src/runtime/sigqueue.go:147 +0x9e
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:23 +0x24
created by os/signal.init.0
    /usr/local/go/src/os/signal/signal_unix.go:29 +0x43

goroutine 116 [select]:
github.com/docker/docker/vendor/go.opencensus.io/stats/view.(*worker).start(0xc0001d05a0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/go.opencensus.io/stats/view/worker.go:154 +0x102
created by github.com/docker/docker/vendor/go.opencensus.io/stats/view.init.0
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/go.opencensus.io/stats/view/worker.go:32 +0x59

goroutine 149 [select, 2108 minutes]:
github.com/docker/docker/libcontainerd/remote.(*client).processEventStream(0xc0008530a0, 0x55d5a696d280, 0xc00004e7c0, 0x55d5a5571adf, 0xc)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/libcontainerd/remote/client.go:782 +0x314
created by github.com/docker/docker/libcontainerd/remote.NewClient
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/libcontainerd/remote/client.go:70 +0x27f

goroutine 137 [chan receive, 877 minutes]:
github.com/docker/docker/pkg/signal.Trap.func1(0xc000051200, 0x55d5a6907ca0, 0xc00012be30, 0xc00000ee80)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/pkg/signal/trap.go:38 +0x5d
created by github.com/docker/docker/pkg/signal.Trap
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/pkg/signal/trap.go:36 +0x11d

goroutine 142 [IO wait, 2108 minutes]:
internal/poll.runtime_pollWait(0x7f3a2220bc60, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc0001c4918, 0x72, 0x0, 0x0, 0x55d5a556692e)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc0001c4900, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1fa
net.(*netFD).accept(0xc0001c4900, 0xc000520700, 0x7f3a22336d98, 0x0)
    /usr/local/go/src/net/fd_unix.go:238 +0x44
net.(*UnixListener).accept(0xc000876690, 0xc0008cdd98, 0x55d5a39a23fa, 0x30)
    /usr/local/go/src/net/unixsock_posix.go:162 +0x34
net.(*UnixListener).Accept(0xc000876690, 0x55d5a66c12a0, 0xc0008bc150, 0x55d5a647d880, 0x55d5a7c9b290)
    /usr/local/go/src/net/unixsock.go:260 +0x49
net/http.(*Server).Serve(0xc0008da000, 0x55d5a695d000, 0xc000876690, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2925 +0x282
net/http.Serve(...)
    /usr/local/go/src/net/http/server.go:2497
github.com/docker/docker/daemon.(*Daemon).listenMetricsSock.func1(0x55d5a695d000, 0xc000876690, 0xc00004e8c0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/metrics_unix.go:33 +0x101
created by github.com/docker/docker/daemon.(*Daemon).listenMetricsSock
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/metrics_unix.go:31 +0x1af

goroutine 144 [select, 2108 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc00004eb80)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:69 +0xc4
created by github.com/docker/docker/vendor/google.golang.org/grpc.newCCBalancerWrapper
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:60 +0x16f

goroutine 145 [chan receive, 2108 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc00045ab00)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:1179 +0x6ec
created by github.com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).connect
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:815 +0x12a

goroutine 163 [IO wait, 19 minutes]:
internal/poll.runtime_pollWait(0x7f3a2220bb90, 0x72, 0xffffffffffffffff)
    /usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc0001c4b18, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0001c4b00, 0xc0008dc000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1d1
net.(*netFD).Read(0xc0001c4b00, 0xc0008dc000, 0x8000, 0x8000, 0x0, 0x800000601, 0x7f3900000000)
    /usr/local/go/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc0006c0810, 0xc0008dc000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:184 +0x6a
bufio.(*Reader).Read(0xc000051d40, 0xc00022c3b8, 0x9, 0x9, 0xc0008cfde0, 0x55d5a399a7ce, 0xc000239920)
    /usr/local/go/src/bufio/bufio.go:226 +0x26c
io.ReadAtLeast(0x55d5a6904ce0, 0xc000051d40, 0xc00022c3b8, 0x9, 0x9, 0x9, 0x55d5a39ef890, 0xc000239920, 0xc0008cfe18)
    /usr/local/go/src/io/io.go:310 +0x89
io.ReadFull(...)
    /usr/local/go/src/io/io.go:329
github.com/docker/docker/vendor/golang.org/x/net/http2.readFrameHeader(0xc00022c3b8, 0x9, 0x9, 0x55d5a6904ce0, 0xc000051d40, 0x0, 0xc000000000, 0x10055d5a39a23fa, 0xc000412d30)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/golang.org/x/net/http2/frame.go:237 +0x89
github.com/docker/docker/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00022c380, 0xc00b63e000, 0xc00b63e000, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/golang.org/x/net/http2/frame.go:492 +0xa3
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc000198380)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1265 +0x185
created by github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:300 +0xd25

goroutine 164 [select, 19 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000412d20, 0x1, 0x0, 0x0, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:395 +0x124
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000051e60, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:513 +0x1e5
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc000198380)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:346 +0x7d
created by github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:344 +0xedd

goroutine 121 [select, 2108 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0004537c0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:69 +0xc4
created by github.com/docker/docker/vendor/google.golang.org/grpc.newCCBalancerWrapper
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:60 +0x16f

goroutine 122 [chan receive, 2108 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc0001e3b80)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:1179 +0x6ec
created by github.com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).connect
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:815 +0x12a

goroutine 124 [IO wait, 68 minutes]:
internal/poll.runtime_pollWait(0x7f3a2220bac0, 0x72, 0xffffffffffffffff)
    /usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc0007e2498, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0007e2480, 0xc000982000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1d1
net.(*netFD).Read(0xc0007e2480, 0xc000982000, 0x8000, 0x8000, 0x0, 0x800000601, 0x7f3a00000000)
    /usr/local/go/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc0006842d8, 0xc000982000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:184 +0x6a
bufio.(*Reader).Read(0xc0001cf0e0, 0xc000972118, 0x9, 0x9, 0xc0008d3de0, 0x55d5a399a7ce, 0xc0000f6e40)
    /usr/local/go/src/bufio/bufio.go:226 +0x26c
io.ReadAtLeast(0x55d5a6904ce0, 0xc0001cf0e0, 0xc000972118, 0x9, 0x9, 0x9, 0x55d5a39ef890, 0xc0000f6e40, 0xc0008d3e18)
    /usr/local/go/src/io/io.go:310 +0x89
io.ReadFull(...)
    /usr/local/go/src/io/io.go:329
github.com/docker/docker/vendor/golang.org/x/net/http2.readFrameHeader(0xc000972118, 0x9, 0x9, 0x55d5a6904ce0, 0xc0001cf0e0, 0x0, 0xc000000000, 0x10055d5a39a23fa, 0xc0001d0ba0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/golang.org/x/net/http2/frame.go:237 +0x89
github.com/docker/docker/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0009720e0, 0xc0002f25c0, 0xc0002f25c0, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/golang.org/x/net/http2/frame.go:492 +0xa3
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00099a000)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1265 +0x185
created by github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:300 +0xd25

goroutine 125 [select, 68 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0001d0b90, 0x1, 0x0, 0x0, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:395 +0x124
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0001cf200, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:513 +0x1e5
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc00099a000)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:346 +0x7d
created by github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:344 +0xedd

goroutine 165 [select, 2108 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc.newClientStream.func5(0xc000465500, 0xc00005de60, 0x55d5a696d340, 0xc000876fc0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:319 +0xd9
created by github.com/docker/docker/vendor/google.golang.org/grpc.newClientStream
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:318 +0x9a7

goroutine 166 [select, 2108 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc0001de800)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/transport.go:318 +0xce
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/transport.go:333
github.com/docker/docker/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc0001c4d80, 0x55d5a67c77a0, 0xc0001cf3e0, 0x0, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:871 +0x755
github.com/docker/docker/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc0001c4d80, 0x203000, 0x203000)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:736 +0x48
github.com/docker/docker/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc00005de60, 0xc0008d4e48, 0xc0008d4e18, 0x0, 0x48)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:594 +0x9e
github.com/docker/docker/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc00005de60, 0x55d5a67c77a0, 0xc0001cf3e0, 0x55d5a39c3f2c, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:735 +0x105
github.com/docker/docker/vendor/github.com/containerd/containerd/api/services/events/v1.(*eventsSubscribeClient).Recv(0xc0003e4db0, 0x0, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/containerd/containerd/api/services/events/v1/events.pb.go:354 +0x64
github.com/docker/docker/vendor/github.com/containerd/containerd.(*eventRemote).Subscribe.func1(0xc000051f80, 0x55d5a698e060, 0xc0003e4db0, 0xc0002399e0, 0x55d5a696d280, 0xc00004ed80)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/containerd/containerd/events.go:99 +0x8f
created by github.com/docker/docker/vendor/github.com/containerd/containerd.(*eventRemote).Subscribe
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/containerd/containerd/events.go:95 +0x1c6

goroutine 178 [sync.Cond.Wait, 2108 minutes]:
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc000f9d190, 0x0)
    /usr/local/go/src/runtime/sema.go:510 +0xfc
sync.(*Cond).Wait(0xc000f9d180)
    /usr/local/go/src/sync/cond.go:56 +0x9f
github.com/docker/docker/daemon/stats.(*Collector).Run(0xc000f9d140)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/stats/collector.go:95 +0x4a
created by github.com/docker/docker/daemon.(*Daemon).newStatsCollector
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/stats_collector.go:24 +0x7f

goroutine 128 [select, 801 minutes]:
github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).watchLoop(0xc0007e0200)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/store.go:430 +0xc8
created by github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).startWatch
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/store.go:447 +0xcf

goroutine 83 [chan receive, 3 minutes]:
github.com/docker/docker/daemon.(*Daemon).execCommandGC(0xc00000c1e0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/exec.go:309 +0x14b
created by github.com/docker/docker/daemon.NewDaemon
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/daemon.go:1136 +0x2aa8

goroutine 84 [select, 2108 minutes]:
github.com/docker/docker/libcontainerd/remote.(*client).processEventStream(0xc0009020e0, 0x55d5a696d280, 0xc00004e7c0, 0x55d5a5562049, 0x4)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/libcontainerd/remote/client.go:782 +0x314
created by github.com/docker/docker/libcontainerd/remote.NewClient
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/libcontainerd/remote/client.go:70 +0x27f

goroutine 194 [select, 2108 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc.newClientStream.func5(0xc000a4dc00, 0xc001042000, 0x55d5a696d340, 0xc001036120)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:319 +0xd9
created by github.com/docker/docker/vendor/google.golang.org/grpc.newClientStream
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:318 +0x9a7

goroutine 195 [select, 2108 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc001048000)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/transport.go:318 +0xce
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/transport.go:333
github.com/docker/docker/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc001044000, 0x55d5a67c77a0, 0xc0010401e0, 0x0, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:871 +0x755
github.com/docker/docker/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc001044000, 0x203000, 0x203000)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:736 +0x48
github.com/docker/docker/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc001042000, 0xc000d44e48, 0xc000d44e18, 0x0, 0x48)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:594 +0x9e
github.com/docker/docker/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc001042000, 0x55d5a67c77a0, 0xc0010401e0, 0x55d5a39c3f2c, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:735 +0x105
github.com/docker/docker/vendor/github.com/containerd/containerd/api/services/events/v1.(*eventsSubscribeClient).Recv(0xc001032070, 0x0, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/containerd/containerd/api/services/events/v1/events.pb.go:354 +0x64
github.com/docker/docker/vendor/github.com/containerd/containerd.(*eventRemote).Subscribe.func1(0xc00102e000, 0x55d5a698e060, 0xc001032070, 0xc00102c000, 0x55d5a696d280, 0xc001026000)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/containerd/containerd/events.go:99 +0x8f
created by github.com/docker/docker/vendor/github.com/containerd/containerd.(*eventRemote).Subscribe
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/containerd/containerd/events.go:95 +0x1c6

goroutine 226 [chan receive, 2108 minutes]:
github.com/docker/docker/vendor/github.com/godbus/dbus/v5.newConn.func1(0xc0008da0e0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/godbus/dbus/v5/conn.go:253 +0x4d
created by github.com/docker/docker/vendor/github.com/godbus/dbus/v5.newConn
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/godbus/dbus/v5/conn.go:252 +0x13a

goroutine 242 [IO wait, 2108 minutes]:
internal/poll.runtime_pollWait(0x7f3a2220b9f0, 0x72, 0x1000)
    /usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc000b1c198, 0x72, 0x0, 0x10, 0xc00086f320)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).ReadMsg(0xc000b1c180, 0xc000ab09c0, 0x10, 0x10, 0xc00086f320, 0x1000, 0x1000, 0x0, 0x0, 0x0, ...)
    /usr/local/go/src/internal/poll/fd_unix.go:243 +0x230
net.(*netFD).readMsg(0xc000b1c180, 0xc000ab09c0, 0x10, 0x10, 0xc00086f320, 0x1000, 0x1000, 0xc0008c9d50, 0x55d5a7ce3bf3, 0x1, ...)
    /usr/local/go/src/net/fd_unix.go:214 +0x92
net.(*UnixConn).readMsg(0xc00105a028, 0xc000ab09c0, 0x10, 0x10, 0xc00086f320, 0x1000, 0x1000, 0xc000ab0890, 0xc000ab088c, 0xc000ae2140, ...)
    /usr/local/go/src/net/unixsock_posix.go:115 +0x93
net.(*UnixConn).ReadMsgUnix(0xc00105a028, 0xc000ab09c0, 0x10, 0x10, 0xc00086f320, 0x1000, 0x1000, 0xc000ae2520, 0x20, 0x18, ...)
    /usr/local/go/src/net/unixsock.go:143 +0x9a
github.com/docker/docker/vendor/github.com/godbus/dbus/v5.(*oobReader).Read(0xc00086f300, 0xc000ab09c0, 0x10, 0x10, 0xc0008c9df0, 0x55d5a39a23fa, 0x20)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/godbus/dbus/v5/transport_unix.go:21 +0x8f
io.ReadAtLeast(0x55d5a6906dc0, 0xc00086f300, 0xc000ab09c0, 0x10, 0x10, 0x10, 0xc001026a90, 0x0, 0x0)
    /usr/local/go/src/io/io.go:310 +0x89
io.ReadFull(...)
    /usr/local/go/src/io/io.go:329
github.com/docker/docker/vendor/github.com/godbus/dbus/v5.(*unixTransport).ReadMessage(0xc001056120, 0xc00000006a, 0x6a, 0xc000a92c01)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/godbus/dbus/v5/transport_unix.go:91 +0x11a
github.com/docker/docker/vendor/github.com/godbus/dbus/v5.(*Conn).inWorker(0xc0008da0e0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/godbus/dbus/v5/conn.go:348 +0x42
created by github.com/docker/docker/vendor/github.com/godbus/dbus/v5.(*Conn).Auth
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/godbus/dbus/v5/auth.go:118 +0x623

goroutine 227 [chan receive, 2108 minutes]:
github.com/docker/docker/vendor/github.com/docker/libnetwork/iptables.signalHandler()
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/iptables/firewalld.go:125 +0x56
created by github.com/docker/docker/vendor/github.com/docker/libnetwork/iptables.FirewalldInit
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/iptables/firewalld.go:79 +0x106

goroutine 346 [chan receive, 2108 minutes]:
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*scheduler).loop.func2(0xc000902d90)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/scheduler.go:76 +0x3e
created by github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*scheduler).loop
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/scheduler.go:75 +0x79

goroutine 273 [IO wait, 859 minutes]:
internal/poll.runtime_pollWait(0x7f3a2220b920, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc000861018, 0x72, 0x0, 0x0, 0x55d5a556692e)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc000861000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1fa
net.(*netFD).accept(0xc000861000, 0xc001624800, 0x0, 0x0)
    /usr/local/go/src/net/fd_unix.go:238 +0x44
net.(*UnixListener).accept(0xc000883f80, 0xc001060f48, 0xc001060f50, 0x18)
    /usr/local/go/src/net/unixsock_posix.go:162 +0x34
net.(*UnixListener).Accept(0xc000883f80, 0x55d5a68b6ea8, 0x55d5a6998020, 0xc001624800, 0xc0007e0200)
    /usr/local/go/src/net/unixsock.go:260 +0x49
github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).acceptClientConnections(0xc0007e0200, 0xc00045eb70, 0x2c, 0x55d5a695d000, 0xc000883f80)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/sandbox_externalkey_unix.go:144 +0x3d
created by github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).startExternalKeyListener
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/sandbox_externalkey_unix.go:138 +0x339

goroutine 272 [select, 2108 minutes]:
github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/overlay.(*driver).peerOpRoutine(0xc001049700, 0x55d5a696d280, 0xc0001e9f80, 0xc00106afc0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/overlay/peerdb.go:279 +0x10c
created by github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/overlay.Init
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/overlay/overlay.go:77 +0x17f

goroutine 150 [chan receive, 2108 minutes]:
main.(*DaemonCli).setupConfigReloadTrap.func1(0xc000ebe0c0, 0xc000933500)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/cmd/dockerd/daemon_unix.go:83 +0x57
created by main.(*DaemonCli).setupConfigReloadTrap
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/cmd/dockerd/daemon_unix.go:82 +0xb7

goroutine 347 [IO wait, 859 minutes]:
internal/poll.runtime_pollWait(0x7f3a2220bd30, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc0001c4798, 0x72, 0x0, 0x0, 0x55d5a556692e)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc0001c4780, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1fa
net.(*netFD).accept(0xc0001c4780, 0x55d5a3df9ca6, 0xc00022c0a0, 0xc00118e090)
    /usr/local/go/src/net/fd_unix.go:238 +0x44
net.(*UnixListener).accept(0xc000876120, 0x5ffbfd61, 0xc000d45da0, 0x55d5a3a5f878)
    /usr/local/go/src/net/unixsock_posix.go:162 +0x34
net.(*UnixListener).Accept(0xc000876120, 0xc000d45df0, 0x18, 0xc0006aa300, 0x55d5a3df91c6)
    /usr/local/go/src/net/unixsock.go:260 +0x49
net/http.(*Server).Serve(0xc00022c000, 0x55d5a695d000, 0xc000876120, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2925 +0x282
github.com/docker/docker/api/server.(*HTTPServer).Serve(...)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/server.go:112
github.com/docker/docker/api/server.(*Server).serveAPI.func1(0xc000fc21e0, 0xc00000ee60)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/server.go:86 +0xd1
created by github.com/docker/docker/api/server.(*Server).serveAPI
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/server.go:83 +0x8d

goroutine 331 [sync.Cond.Wait, 801 minutes]:
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc000bd7cd0, 0xc0000000d5)
    /usr/local/go/src/runtime/sema.go:510 +0xfc
sync.(*Cond).Wait(0xc000bd7cc0)
    /usr/local/go/src/sync/cond.go:56 +0x9f
github.com/docker/docker/vendor/github.com/moby/buildkit/util/cond.(*StatefulCond).Wait(0xc000c35dd0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/cond/cond.go:28 +0xb1
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*scheduler).loop(0xc000902d90)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/scheduler.go:101 +0x18c
created by github.com/docker/docker/vendor/github.com/moby/buildkit/solver.newScheduler
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/scheduler.go:35 +0x1ad

goroutine 332 [select, 2108 minutes]:
github.com/docker/docker/daemon.(*Daemon).ProcessClusterNotifications(0xc00000c1e0, 0x55d5a696d280, 0xc00004e7c0, 0xc001041380)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/events.go:126 +0xe7
created by main.(*DaemonCli).start
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/cmd/dockerd/daemon.go:234 +0xb71

goroutine 151 [chan receive, 2108 minutes]:
github.com/docker/docker/api/server.(*Server).serveAPI(0xc0004128c0, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/server.go:94 +0x14c
github.com/docker/docker/api/server.(*Server).Wait(0xc0004128c0, 0xc00106a060)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/server.go:199 +0x31
created by main.(*DaemonCli).start
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/cmd/dockerd/daemon.go:242 +0xbd6

goroutine 107056 [chan send]:
github.com/docker/docker/builder/builder-next.(*statusProxy).SendMsg(0xc000c0ec00, 0x55d5a67f1480, 0xc002064000, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/builder/builder-next/builder.go:452 +0x78
github.com/docker/docker/vendor/github.com/moby/buildkit/control.(*Controller).Status.func2(0xc0027c1768, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/moby/buildkit/control/control.go:352 +0x757
github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc0013fdce0, 0xc000c0ec20)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x66
created by github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x68

goroutine 1091 [select]:
github.com/docker/docker/vendor/github.com/docker/libnetwork/osl.removeUnusedPaths()
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/osl/namespace_linux.go:105 +0x2dd
created by github.com/docker/docker/vendor/github.com/docker/libnetwork/osl.createBasePath
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/osl/namespace_linux.go:90 +0xbd

goroutine 107097 [semacquire, 859 minutes]:
sync.runtime_Semacquire(0xc0013fdcf0)
    /usr/local/go/src/runtime/sema.go:56 +0x44
sync.(*WaitGroup).Wait(0xc0013fdce8)
    /usr/local/go/src/sync/waitgroup.go:130 +0x66
github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Wait(0xc0013fdce0, 0xc000c0ec20, 0x55d5a696d280)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:40 +0x33
github.com/docker/docker/vendor/github.com/moby/buildkit/control.(*Controller).Status(0xc000f4e510, 0xc0013fdc50, 0x55d5a698d8e0, 0xc000c0ec00, 0xc0044a0eb8, 0xc000b7ef60)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/moby/buildkit/control/control.go:362 +0x1a5
github.com/docker/docker/builder/builder-next.(*Builder).Build.func3(0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/builder/builder-next/builder.go:397 +0x14b
github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc001df5740, 0xc001df5770)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x66
created by github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x68

goroutine 107124 [select]:
github.com/docker/docker/vendor/github.com/moby/buildkit/session.monitorHealth(0x55d5a696d280, 0xc001ee0b00, 0xc0038e0000, 0xc000f5b230)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/moby/buildkit/session/grpc.go:87 +0x1cd
created by github.com/docker/docker/vendor/github.com/moby/buildkit/session.grpcClientConn
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/moby/buildkit/session/grpc.go:73 +0x335

goroutine 102480 [chan receive, 859 minutes]:
github.com/docker/docker/vendor/github.com/moby/buildkit/session.(*Manager).handleConn(0xc000ae2ca0, 0x55d5a696d280, 0xc001ee0b00, 0x55d5a6998020, 0xc001802468, 0xc0012be150, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/moby/buildkit/session/manager.go:144 +0x4c4
github.com/docker/docker/vendor/github.com/moby/buildkit/session.(*Manager).HandleHTTPRequest(0xc000ae2ca0, 0x55d5a696d340, 0xc0012be540, 0x55d5a695d280, 0xc0027235e0, 0xc002ad2e00, 0xc00013d500, 0x55d5a39a8b14)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/moby/buildkit/session/manager.go:91 +0x7cf
github.com/docker/docker/api/server/router/session.(*sessionRouter).startSession(0xc000bf7830, 0x55d5a696d340, 0xc0012be540, 0x55d5a695d280, 0xc0027235e0, 0xc002ad2e00, 0xc0012be270, 0xc00013d501, 0xc0001e6f20)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/router/session/session_routes.go:11 +0x6b
github.com/docker/docker/api/server/middleware.ExperimentalMiddleware.WrapHandler.func1(0x55d5a696d340, 0xc0012be540, 0x55d5a695d280, 0xc0027235e0, 0xc002ad2e00, 0xc0012be270, 0x55d5a696d340, 0xc0012be540)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/middleware/experimental.go:26 +0x177
github.com/docker/docker/api/server/middleware.VersionMiddleware.WrapHandler.func1(0x55d5a696d340, 0xc0012be450, 0x55d5a695d280, 0xc0027235e0, 0xc002ad2e00, 0xc0012be270, 0x55d5a39a1b72, 0xc001155760)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/middleware/version.go:62 +0x5fb
github.com/docker/docker/pkg/authorization.(*Middleware).WrapHandler.func1(0x55d5a696d340, 0xc0012be450, 0x55d5a695d280, 0xc0027235e0, 0xc002ad2e00, 0xc0012be270, 0xc001155760, 0x55d5a6562901)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/pkg/authorization/middleware.go:59 +0x826
github.com/docker/docker/api/server/middleware.DebugRequestMiddleware.func1(0x55d5a696d340, 0xc0012be450, 0x55d5a695d280, 0xc0027235e0, 0xc002ad2e00, 0xc0012be270, 0x55d5a696d340, 0xc0012be450)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/middleware/debug.go:53 +0x4a0
github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1(0x55d5a695d280, 0xc0027235e0, 0xc002ad2d00)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/server.go:141 +0x241
net/http.HandlerFunc.ServeHTTP(0xc000a8b2e0, 0x55d5a695d280, 0xc0027235e0, 0xc002ad2d00)
    /usr/local/go/src/net/http/server.go:2036 +0x46
github.com/docker/docker/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc00084e300, 0x55d5a695d280, 0xc0027235e0, 0xc002ad2a00)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/gorilla/mux/mux.go:210 +0xe4
net/http.serverHandler.ServeHTTP(0xc00022c000, 0x55d5a695d280, 0xc0027235e0, 0xc002ad2a00)
    /usr/local/go/src/net/http/server.go:2831 +0xa6
net/http.(*conn).serve(0xc000d17720, 0x55d5a696d280, 0xc0011c99c0)
    /usr/local/go/src/net/http/server.go:1919 +0x877
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:2957 +0x386

goroutine 107125 [IO wait]:
internal/poll.runtime_pollWait(0x7f3a186b2e30, 0x72, 0xffffffffffffffff)
    /usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc003401998, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc003401980, 0xc0038ee000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1d1
net.(*netFD).Read(0xc003401980, 0xc0038ee000, 0x8000, 0x8000, 0x0, 0x800010601, 0x7f3a00000000)
    /usr/local/go/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc001802468, 0xc0038ee000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:184 +0x6a
bufio.(*Reader).Read(0xc0016e64e0, 0xc0027236f8, 0x9, 0x9, 0xc000465c00, 0x7f3a22337460, 0x0)
    /usr/local/go/src/bufio/bufio.go:226 +0x26c
io.ReadAtLeast(0x55d5a6904ce0, 0xc0016e64e0, 0xc0027236f8, 0x9, 0x9, 0x9, 0x55d5a40d5e77, 0xc0002f232c, 0xc0014f6de8)
    /usr/local/go/src/io/io.go:310 +0x89
io.ReadFull(...)
    /usr/local/go/src/io/io.go:329
github.com/docker/docker/vendor/golang.org/x/net/http2.readFrameHeader(0xc0027236f8, 0x9, 0x9, 0x55d5a6904ce0, 0xc0016e64e0, 0x0, 0xbff74f9600000000, 0x730daf12b888, 0x55d5a7d419e0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/golang.org/x/net/http2/frame.go:237 +0x89
github.com/docker/docker/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0027236c0, 0xc0002f2320, 0xc0002f2320, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/golang.org/x/net/http2/frame.go:492 +0xa3
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00099a380)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1265 +0x185
created by github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:300 +0xd25

goroutine 107095 [IO wait, 859 minutes]:
internal/poll.runtime_pollWait(0x7f3a2220b5e0, 0x72, 0xffffffffffffffff)
    /usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc003401618, 0x72, 0x0, 0x1, 0xffffffffffffffff)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc003401600, 0xc001649c01, 0x1, 0x1, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1d1
net.(*netFD).Read(0xc003401600, 0xc001649c01, 0x1, 0x1, 0x8, 0xc0013ad710, 0x55d5a4e2ae90)
    /usr/local/go/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc001802460, 0xc001649c01, 0x1, 0x1, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:184 +0x6a
net/http.(*connReader).backgroundRead(0xc001649bf0)
    /usr/local/go/src/net/http/server.go:688 +0x5a
created by net/http.(*connReader).startBackgroundRead
    /usr/local/go/src/net/http/server.go:684 +0xd6

goroutine 107016 [select, 859 minutes]:
github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress.(*MultiReader).Reader.func1(0x55d5a696d340, 0xc00321c180, 0xc001df5a10, 0xc0001b7880)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress/multireader.go:37 +0xc3
created by github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress.(*MultiReader).Reader
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress/multireader.go:36 +0x1c4

goroutine 102478 [semacquire, 859 minutes]:
sync.runtime_Semacquire(0xc001df5750)
    /usr/local/go/src/runtime/sema.go:56 +0x44
sync.(*WaitGroup).Wait(0xc001df5748)
    /usr/local/go/src/sync/waitgroup.go:130 +0x66
github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Wait(0xc001df5740, 0xc000c3d5c0, 0xc003676960)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:40 +0x33
github.com/docker/docker/builder/builder-next.(*Builder).Build(0xc000c0f180, 0x55d5a696d280, 0xc00381d880, 0x55d5a6944200, 0x55d5a7d619e8, 0x55d5a6905200, 0xc000c3d440, 0x55d5a6905a60, 0xc000c3d460, 0x55d5a6905a60, ...)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/builder/builder-next/builder.go:413 +0xe6d
github.com/docker/docker/api/server/backend/build.(*Backend).Build(0xc000c35e30, 0x55d5a696d340, 0xc001df5290, 0x55d5a6944200, 0x55d5a7d619e8, 0x55d5a6905200, 0xc000c3d440, 0x55d5a6905a60, 0xc000c3d460, 0x55d5a6905a60, ...)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/backend/build/backend.go:64 +0x141
github.com/docker/docker/api/server/router/build.(*buildRouter).postBuild(0xc000bd7ec0, 0x55d5a696d340, 0xc001df5290, 0x55d5a695d280, 0xc00366ce00, 0xc00148a500, 0xc001df4f90, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/router/build/build_routes.go:282 +0x731
github.com/docker/docker/api/server/middleware.ExperimentalMiddleware.WrapHandler.func1(0x55d5a696d340, 0xc001df5290, 0x55d5a695d280, 0xc00366ce00, 0xc00148a500, 0xc001df4f90, 0x55d5a696d340, 0xc001df5290)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/middleware/experimental.go:26 +0x177
github.com/docker/docker/api/server/middleware.VersionMiddleware.WrapHandler.func1(0x55d5a696d340, 0xc001df5050, 0x55d5a695d280, 0xc00366ce00, 0xc00148a500, 0xc001df4f90, 0xc00003c000, 0x55d5a64e1800)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/middleware/version.go:62 +0x5fb
github.com/docker/docker/pkg/authorization.(*Middleware).WrapHandler.func1(0x55d5a696d340, 0xc001df5050, 0x55d5a695d280, 0xc00366ce00, 0xc00148a500, 0xc001df4f90, 0x2, 0x55d5a6562901)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/pkg/authorization/middleware.go:59 +0x826
github.com/docker/docker/api/server/middleware.DebugRequestMiddleware.func1(0x55d5a696d340, 0xc001df5050, 0x55d5a695d280, 0xc00366ce00, 0xc00148a500, 0xc001df4f90, 0x55d5a696d340, 0xc001df5050)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/middleware/debug.go:25 +0x8ae
github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1(0x55d5a695d280, 0xc00366ce00, 0xc00148a400)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/server.go:141 +0x241
net/http.HandlerFunc.ServeHTTP(0xc000a8ad80, 0x55d5a695d280, 0xc00366ce00, 0xc00148a400)
    /usr/local/go/src/net/http/server.go:2036 +0x46
github.com/docker/docker/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc00084e300, 0x55d5a695d280, 0xc00366ce00, 0xc002ad2900)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/gorilla/mux/mux.go:210 +0xe4
net/http.serverHandler.ServeHTTP(0xc00022c000, 0x55d5a695d280, 0xc00366ce00, 0xc002ad2900)
    /usr/local/go/src/net/http/server.go:2831 +0xa6
net/http.(*conn).serve(0xc000d17540, 0x55d5a696d280, 0xc0011c8580)
    /usr/local/go/src/net/http/server.go:1919 +0x877
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:2957 +0x386

goroutine 107098 [runnable]:
syscall.Syscall(0x1, 0x18, 0xc0038cf000, 0x2d, 0x2d, 0x2d, 0x0)
    /usr/local/go/src/syscall/asm_linux_amd64.s:18 +0x5
syscall.write(0x18, 0xc0038cf000, 0x2d, 0x1000, 0x0, 0x2, 0xc000000078)
    /usr/local/go/src/syscall/zsyscall_linux_amd64.go:1005 +0x5c
syscall.Write(...)
    /usr/local/go/src/syscall/syscall_unix.go:202
internal/poll.(*FD).Write(0xc003401600, 0xc0038cf000, 0x2d, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:268 +0x1a3
net.(*netFD).Write(0xc003401600, 0xc0038cf000, 0x2d, 0x1000, 0x4, 0x0, 0x0)
    /usr/local/go/src/net/fd_unix.go:220 +0x51
net.(*conn).Write(0xc001802460, 0xc0038cf000, 0x2d, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:196 +0x6a
net/http.checkConnErrorWriter.Write(0xc000d17540, 0xc0038cf000, 0x2d, 0x1000, 0xc0014f2d38, 0x55d5a3a945b7, 0xc00366ce40)
    /usr/local/go/src/net/http/server.go:3448 +0x58
bufio.(*Writer).Flush(0xc0011c8780, 0x0, 0x30)
    /usr/local/go/src/bufio/bufio.go:593 +0x77
net/http.(*chunkWriter).flush(0xc00366ce40)
    /usr/local/go/src/net/http/server.go:396 +0x3d
net/http.(*response).Flush(0xc00366ce00)
    /usr/local/go/src/net/http/server.go:1667 +0x49
github.com/docker/docker/pkg/ioutils.(*WriteFlusher).Flush(0xc000d55f40)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/pkg/ioutils/writeflusher.go:49 +0x7c
github.com/docker/docker/pkg/ioutils.(*WriteFlusher).Write(0xc000d55f40, 0xc00b9c0000, 0x27, 0x30, 0x0, 0x30, 0x25)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/pkg/ioutils/writeflusher.go:34 +0xd9
github.com/docker/docker/api/server/router/build.(*syncWriter).Write(0xc000c3d440, 0xc00b9c0000, 0x27, 0x30, 0x30, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/router/build/build_routes.go:321 +0x7a
github.com/docker/docker/pkg/streamformatter.(*AuxFormatter).Emit(0xc000eadd40, 0x55d5a558188a, 0x13, 0x55d5a634b100, 0xc000c1c000, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/pkg/streamformatter/streamformatter.go:154 +0x17b
github.com/docker/docker/builder/builder-next.(*Builder).Build.func4(0xc006a64f68, 0x7f3a222aa101)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/builder/builder-next/builder.go:406 +0xe4
github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc001df5740, 0xc000c3d5c0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x66
created by github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x68

goroutine 107122 [select, 859 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc001ee0940)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:69 +0xc4
created by github.com/docker/docker/vendor/google.golang.org/grpc.newCCBalancerWrapper
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:60 +0x16f

goroutine 107123 [chan receive, 859 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc000116840)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:1179 +0x6ec
created by github.com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).connect
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:815 +0x12a

goroutine 107126 [select]:
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0019a6320, 0x1, 0x0, 0x0, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:395 +0x124
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0016e66c0, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:513 +0x1e5
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc00099a380)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:346 +0x7d
created by github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:344 +0xedd
tonistiigi commented 3 years ago

Hmm, hard to say what is exactly going on in that trace. Build looks complete but handling status is not. But

net/http.(*response).Flush(0xc00366ce00)
    /usr/local/go/src/net/http/server.go:1667 +0x49
github.com/docker/docker/pkg/ioutils.(*WriteFlusher).Flush(0xc000d55f40)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/pkg/ioutils/writeflusher.go:49 +0x7c
github.com/docker/docker/pkg/ioutils.(*WriteFlusher).Write(0xc000d55f40, 0xc00b9c0000, 0x27, 0x30, 0x0, 0x30, 0x25)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/pkg/ioutils/writeflusher.go:34 +0xd9
github.com/docker/docker/api/server/router/build.(*syncWriter).Write(0xc000c3d440, 0xc00b9c0000, 0x27, 0x30, 0x30, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/router/build/build_routes.go:321 +0x7a
github.com/docker/docker/pkg/streamformatter.(*AuxFormatter).Emit(0xc000eadd40, 0x55d5a558188a, 0x13, 0x55d5a634b100, 0xc000c1c000, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/pkg/streamformatter/streamformatter.go:154 +0x17b

seems to show that progress is being written and it is not completely locked. Maybe in some conditions writing status goes into a loop (ignoring the possibility that the network locked and just restored before taking the trace).

Is there any way we can reproduce? Wouldn't hurt to take some other traces for comparison if it happens again.

xkszltl commented 3 years ago

Glad to. I'm not sure about repro because I also seem hanging in different places on different machines, one of them (an ubuntu1804 VM in US) even works totally fine so far.

Is there any network access? Should not be any but since you mentioned. I build in China and connectivity to docker hub is very bad, and no connectivity to google.

xkszltl commented 3 years ago

@tonistiigi Another incident:

$ rpm -qa docker* *container*
nvidia-container-runtime-3.4.0-1.x86_64
containerd.io-1.4.3-3.1.el7.x86_64
plexus-containers-container-default-1.5.5-14.el7.noarch
docker-ce-20.10.2-3.el7.x86_64
container-selinux-2.119.2-1.911c772.el7_8.noarch
docker-devel-1.3.2-4.el7.centos.x86_64
ghc-containers-0.5.0.0-26.4.el7.x86_64
libnvidia-container1-1.3.1-1.x86_64
containers-common-0.1.40-11.el7_8.x86_64
plexus-containers-component-metadata-1.5.5-14.el7.noarch
container-storage-setup-0.11.0-2.git5eaf76c.el7.noarch
docker-ce-rootless-extras-20.10.2-3.el7.x86_64
docker-ce-cli-20.10.2-3.el7.x86_64
ghc-unordered-containers-0.2.3.0-3.el7.x86_64
nvidia-container-toolkit-1.4.0-2.x86_64
plexus-containers-component-annotations-1.5.5-14.el7.noarch
docker-ce-debuginfo-18.06.3.ce-3.el7.x86_64
libnvidia-container-tools-1.3.1-1.x86_64
docker-unit-test-1.13.1-68.gitdded712.el7.centos.x86_64
docker-distribution-2.6.2-2.git48294d9.el7.x86_64
$ cat /var/run/docker/goroutine-stacks-2021-01-26T165641+0800.log
goroutine 139 [running]:
github.com/docker/docker/pkg/signal.DumpStacks(0x55d5a5576870, 0xf, 0x0, 0x0, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/pkg/signal/trap.go:83 +0x9d
github.com/docker/docker/daemon.(*Daemon).setupDumpStackTrap.func1(0xc000051560, 0x55d5a5576870, 0xf)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/debugtrap_unix.go:19 +0x74
created by github.com/docker/docker/daemon.(*Daemon).setupDumpStackTrap
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/debugtrap_unix.go:17 +0xc1

goroutine 1 [chan receive, 22940 minutes]:
main.(*DaemonCli).start(0xc000933500, 0xc000050a20, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/cmd/dockerd/daemon.go:249 +0xc04
main.runDaemon(...)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/cmd/dockerd/docker_unix.go:13
main.newDaemonCommand.func1(0xc00020eb00, 0xc0009334a0, 0x0, 0x3, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/cmd/dockerd/docker.go:34 +0x7c
github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).execute(0xc00020eb00, 0xc00004e0d0, 0x3, 0x3, 0xc00020eb00, 0xc00004e0d0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:850 +0x462
github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc00020eb00, 0x0, 0x0, 0x10)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:958 +0x34b
github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).Execute(...)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:895
main.main()
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/cmd/dockerd/docker.go:97 +0x191

goroutine 6 [syscall]:
os/signal.signal_recv(0x55d5a6944340)
    /usr/local/go/src/runtime/sigqueue.go:147 +0x9e
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:23 +0x24
created by os/signal.init.0
    /usr/local/go/src/os/signal/signal_unix.go:29 +0x43

goroutine 116 [select]:
github.com/docker/docker/vendor/go.opencensus.io/stats/view.(*worker).start(0xc0001d05a0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/go.opencensus.io/stats/view/worker.go:154 +0x102
created by github.com/docker/docker/vendor/go.opencensus.io/stats/view.init.0
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/go.opencensus.io/stats/view/worker.go:32 +0x59

goroutine 149 [select, 22940 minutes]:
github.com/docker/docker/libcontainerd/remote.(*client).processEventStream(0xc0008530a0, 0x55d5a696d280, 0xc00004e7c0, 0x55d5a5571adf, 0xc)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/libcontainerd/remote/client.go:782 +0x314
created by github.com/docker/docker/libcontainerd/remote.NewClient
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/libcontainerd/remote/client.go:70 +0x27f

goroutine 137 [chan receive, 1023 minutes]:
github.com/docker/docker/pkg/signal.Trap.func1(0xc000051200, 0x55d5a6907ca0, 0xc00012be30, 0xc00000ee80)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/pkg/signal/trap.go:38 +0x5d
created by github.com/docker/docker/pkg/signal.Trap
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/pkg/signal/trap.go:36 +0x11d

goroutine 142 [IO wait, 22940 minutes]:
internal/poll.runtime_pollWait(0x7f3a2220bc60, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc0001c4918, 0x72, 0x0, 0x0, 0x55d5a556692e)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc0001c4900, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1fa
net.(*netFD).accept(0xc0001c4900, 0xc000520700, 0x7f3a22336d98, 0x0)
    /usr/local/go/src/net/fd_unix.go:238 +0x44
net.(*UnixListener).accept(0xc000876690, 0xc0008cdd98, 0x55d5a39a23fa, 0x30)
    /usr/local/go/src/net/unixsock_posix.go:162 +0x34
net.(*UnixListener).Accept(0xc000876690, 0x55d5a66c12a0, 0xc0008bc150, 0x55d5a647d880, 0x55d5a7c9b290)
    /usr/local/go/src/net/unixsock.go:260 +0x49
net/http.(*Server).Serve(0xc0008da000, 0x55d5a695d000, 0xc000876690, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2925 +0x282
net/http.Serve(...)
    /usr/local/go/src/net/http/server.go:2497
github.com/docker/docker/daemon.(*Daemon).listenMetricsSock.func1(0x55d5a695d000, 0xc000876690, 0xc00004e8c0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/metrics_unix.go:33 +0x101
created by github.com/docker/docker/daemon.(*Daemon).listenMetricsSock
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/metrics_unix.go:31 +0x1af

goroutine 144 [select, 22940 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc00004eb80)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:69 +0xc4
created by github.com/docker/docker/vendor/google.golang.org/grpc.newCCBalancerWrapper
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:60 +0x16f

goroutine 145 [chan receive, 22940 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc00045ab00)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:1179 +0x6ec
created by github.com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).connect
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:815 +0x12a

goroutine 163 [IO wait, 20 minutes]:
internal/poll.runtime_pollWait(0x7f3a2220bb90, 0x72, 0xffffffffffffffff)
    /usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc0001c4b18, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0001c4b00, 0xc0008dc000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1d1
net.(*netFD).Read(0xc0001c4b00, 0xc0008dc000, 0x8000, 0x8000, 0x0, 0x800000601, 0x7f3900000000)
    /usr/local/go/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc0006c0810, 0xc0008dc000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:184 +0x6a
bufio.(*Reader).Read(0xc000051d40, 0xc00022c3b8, 0x9, 0x9, 0xc0000c4de0, 0x55d5a399a7ce, 0xc000239920)
    /usr/local/go/src/bufio/bufio.go:226 +0x26c
io.ReadAtLeast(0x55d5a6904ce0, 0xc000051d40, 0xc00022c3b8, 0x9, 0x9, 0x9, 0x55d5a39ef890, 0xc000239920, 0xc0000c4e18)
    /usr/local/go/src/io/io.go:310 +0x89
io.ReadFull(...)
    /usr/local/go/src/io/io.go:329
github.com/docker/docker/vendor/golang.org/x/net/http2.readFrameHeader(0xc00022c3b8, 0x9, 0x9, 0x55d5a6904ce0, 0xc000051d40, 0x0, 0xc000000000, 0x10055d5a39a23fa, 0xc000412d30)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/golang.org/x/net/http2/frame.go:237 +0x89
github.com/docker/docker/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00022c380, 0xc007822000, 0xc007822000, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/golang.org/x/net/http2/frame.go:492 +0xa3
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc000198380)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1265 +0x185
created by github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:300 +0xd25

goroutine 164 [select, 20 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000412d20, 0x1, 0x0, 0x0, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:395 +0x124
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000051e60, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:513 +0x1e5
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc000198380)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:346 +0x7d
created by github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:344 +0xedd

goroutine 121 [select, 22940 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0004537c0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:69 +0xc4
created by github.com/docker/docker/vendor/google.golang.org/grpc.newCCBalancerWrapper
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:60 +0x16f

goroutine 122 [chan receive, 22940 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc0001e3b80)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:1179 +0x6ec
created by github.com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).connect
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:815 +0x12a

goroutine 124 [IO wait, 20 minutes]:
internal/poll.runtime_pollWait(0x7f3a2220bac0, 0x72, 0xffffffffffffffff)
    /usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc0007e2498, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0007e2480, 0xc000982000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1d1
net.(*netFD).Read(0xc0007e2480, 0xc000982000, 0x8000, 0x8000, 0x0, 0x800000601, 0x7f3900000000)
    /usr/local/go/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc0006842d8, 0xc000982000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:184 +0x6a
bufio.(*Reader).Read(0xc0001cf0e0, 0xc000972118, 0x9, 0x9, 0xc0008d3de0, 0x55d5a399a7ce, 0xc0000f6e40)
    /usr/local/go/src/bufio/bufio.go:226 +0x26c
io.ReadAtLeast(0x55d5a6904ce0, 0xc0001cf0e0, 0xc000972118, 0x9, 0x9, 0x9, 0x55d5a39ef890, 0xc0000f6e40, 0xc0008d3e18)
    /usr/local/go/src/io/io.go:310 +0x89
io.ReadFull(...)
    /usr/local/go/src/io/io.go:329
github.com/docker/docker/vendor/golang.org/x/net/http2.readFrameHeader(0xc000972118, 0x9, 0x9, 0x55d5a6904ce0, 0xc0001cf0e0, 0x0, 0xc000000000, 0x10055d5a39a23fa, 0xc0001d0ba0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/golang.org/x/net/http2/frame.go:237 +0x89
github.com/docker/docker/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0009720e0, 0xc007822080, 0xc007822080, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/golang.org/x/net/http2/frame.go:492 +0xa3
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00099a000)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1265 +0x185
created by github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:300 +0xd25

goroutine 125 [select, 20 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0001d0b90, 0x1, 0x0, 0x0, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:395 +0x124
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0001cf200, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:513 +0x1e5
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc00099a000)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:346 +0x7d
created by github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:344 +0xedd

goroutine 165 [select, 22940 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc.newClientStream.func5(0xc000465500, 0xc00005de60, 0x55d5a696d340, 0xc000876fc0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:319 +0xd9
created by github.com/docker/docker/vendor/google.golang.org/grpc.newClientStream
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:318 +0x9a7

goroutine 166 [select, 22940 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc0001de800)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/transport.go:318 +0xce
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/transport.go:333
github.com/docker/docker/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc0001c4d80, 0x55d5a67c77a0, 0xc0001cf3e0, 0x0, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:871 +0x755
github.com/docker/docker/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc0001c4d80, 0x203000, 0x203000)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:736 +0x48
github.com/docker/docker/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc00005de60, 0xc0008d4e48, 0xc0008d4e18, 0x0, 0x48)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:594 +0x9e
github.com/docker/docker/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc00005de60, 0x55d5a67c77a0, 0xc0001cf3e0, 0x55d5a39c3f2c, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:735 +0x105
github.com/docker/docker/vendor/github.com/containerd/containerd/api/services/events/v1.(*eventsSubscribeClient).Recv(0xc0003e4db0, 0x0, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/containerd/containerd/api/services/events/v1/events.pb.go:354 +0x64
github.com/docker/docker/vendor/github.com/containerd/containerd.(*eventRemote).Subscribe.func1(0xc000051f80, 0x55d5a698e060, 0xc0003e4db0, 0xc0002399e0, 0x55d5a696d280, 0xc00004ed80)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/containerd/containerd/events.go:99 +0x8f
created by github.com/docker/docker/vendor/github.com/containerd/containerd.(*eventRemote).Subscribe
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/containerd/containerd/events.go:95 +0x1c6

goroutine 178 [sync.Cond.Wait, 22940 minutes]:
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc000f9d190, 0x0)
    /usr/local/go/src/runtime/sema.go:510 +0xfc
sync.(*Cond).Wait(0xc000f9d180)
    /usr/local/go/src/sync/cond.go:56 +0x9f
github.com/docker/docker/daemon/stats.(*Collector).Run(0xc000f9d140)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/stats/collector.go:95 +0x4a
created by github.com/docker/docker/daemon.(*Daemon).newStatsCollector
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/stats_collector.go:24 +0x7f

goroutine 128 [select, 628 minutes]:
github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).watchLoop(0xc0007e0200)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/store.go:430 +0xc8
created by github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).startWatch
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/store.go:447 +0xcf

goroutine 83 [chan receive]:
github.com/docker/docker/daemon.(*Daemon).execCommandGC(0xc00000c1e0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/exec.go:309 +0x14b
created by github.com/docker/docker/daemon.NewDaemon
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/daemon.go:1136 +0x2aa8

goroutine 84 [select, 7153 minutes]:
github.com/docker/docker/libcontainerd/remote.(*client).processEventStream(0xc0009020e0, 0x55d5a696d280, 0xc00004e7c0, 0x55d5a5562049, 0x4)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/libcontainerd/remote/client.go:782 +0x314
created by github.com/docker/docker/libcontainerd/remote.NewClient
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/libcontainerd/remote/client.go:70 +0x27f

goroutine 194 [select, 22940 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc.newClientStream.func5(0xc000a4dc00, 0xc001042000, 0x55d5a696d340, 0xc001036120)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:319 +0xd9
created by github.com/docker/docker/vendor/google.golang.org/grpc.newClientStream
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:318 +0x9a7

goroutine 195 [select, 7153 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc0001d0050, 0xc001034150, 0x5, 0x5, 0x60, 0x0, 0x55d5a68bc628)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/transport.go:186 +0xd5
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0001d0050, 0xc001034150, 0x5, 0x5, 0xc000d44ef0, 0xc00104a000, 0xc000d44990)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/transport.go:166 +0x197
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0010361b0, 0xc001034150, 0x5, 0x5, 0xc000a7a044, 0xc000d449b8, 0x55d5a4119b0e)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/transport.go:479 +0x57
io.ReadAtLeast(0x55d5a6908d20, 0xc0010361b0, 0xc001034150, 0x5, 0x5, 0x5, 0x0, 0x0, 0xc00103a180)
    /usr/local/go/src/io/io.go:310 +0x89
io.ReadFull(...)
    /usr/local/go/src/io/io.go:329
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc001048000, 0xc001034150, 0x5, 0x5, 0x24, 0x55d5a64f3780, 0x55d5a67c77a0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/transport.go:463 +0xcf
github.com/docker/docker/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc001034140, 0x1000000, 0xc002992840, 0xc000a7a000, 0x9a, 0x9a, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/rpc_util.go:508 +0x65
github.com/docker/docker/vendor/google.golang.org/grpc.recvAndDecompress(0xc001034140, 0xc001048000, 0x0, 0x0, 0x1000000, 0x0, 0x0, 0x0, 0xc000a7a000, 0x9a, ...)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4f
github.com/docker/docker/vendor/google.golang.org/grpc.recv(0xc001034140, 0x7f3a2224f1c0, 0x55d5a7d619e8, 0xc001048000, 0x0, 0x0, 0x55d5a67c77a0, 0xc004944240, 0x1000000, 0x0, ...)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/rpc_util.go:707 +0x9d
github.com/docker/docker/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc001044000, 0x55d5a67c77a0, 0xc004944240, 0x0, 0xc000d44d38, 0x55d5a39f26b0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:885 +0xef
github.com/docker/docker/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc001044000, 0x203001, 0x203001)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:736 +0x48
github.com/docker/docker/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc001042000, 0xc000d44e48, 0xc000d44e18, 0x1, 0x48)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:590 +0x362
github.com/docker/docker/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc001042000, 0x55d5a67c77a0, 0xc004944240, 0xc000d44eec, 0x2)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:735 +0x105
github.com/docker/docker/vendor/github.com/containerd/containerd/api/services/events/v1.(*eventsSubscribeClient).Recv(0xc001032070, 0xc000d44ee8, 0x2, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/containerd/containerd/api/services/events/v1/events.pb.go:354 +0x64
github.com/docker/docker/vendor/github.com/containerd/containerd.(*eventRemote).Subscribe.func1(0xc00102e000, 0x55d5a698e060, 0xc001032070, 0xc00102c000, 0x55d5a696d280, 0xc001026000)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/containerd/containerd/events.go:99 +0x8f
created by github.com/docker/docker/vendor/github.com/containerd/containerd.(*eventRemote).Subscribe
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/containerd/containerd/events.go:95 +0x1c6

goroutine 226 [chan receive, 22940 minutes]:
github.com/docker/docker/vendor/github.com/godbus/dbus/v5.newConn.func1(0xc0008da0e0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/godbus/dbus/v5/conn.go:253 +0x4d
created by github.com/docker/docker/vendor/github.com/godbus/dbus/v5.newConn
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/godbus/dbus/v5/conn.go:252 +0x13a

goroutine 242 [IO wait, 22940 minutes]:
internal/poll.runtime_pollWait(0x7f3a2220b9f0, 0x72, 0x1000)
    /usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc000b1c198, 0x72, 0x0, 0x10, 0xc00086f320)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).ReadMsg(0xc000b1c180, 0xc000ab09c0, 0x10, 0x10, 0xc00086f320, 0x1000, 0x1000, 0x0, 0x0, 0x0, ...)
    /usr/local/go/src/internal/poll/fd_unix.go:243 +0x230
net.(*netFD).readMsg(0xc000b1c180, 0xc000ab09c0, 0x10, 0x10, 0xc00086f320, 0x1000, 0x1000, 0xc0008c9d50, 0x55d5a7ce3bf3, 0x1, ...)
    /usr/local/go/src/net/fd_unix.go:214 +0x92
net.(*UnixConn).readMsg(0xc00105a028, 0xc000ab09c0, 0x10, 0x10, 0xc00086f320, 0x1000, 0x1000, 0xc000ab0890, 0xc000ab088c, 0xc000ae2140, ...)
    /usr/local/go/src/net/unixsock_posix.go:115 +0x93
net.(*UnixConn).ReadMsgUnix(0xc00105a028, 0xc000ab09c0, 0x10, 0x10, 0xc00086f320, 0x1000, 0x1000, 0xc000ae2520, 0x20, 0x18, ...)
    /usr/local/go/src/net/unixsock.go:143 +0x9a
github.com/docker/docker/vendor/github.com/godbus/dbus/v5.(*oobReader).Read(0xc00086f300, 0xc000ab09c0, 0x10, 0x10, 0xc0008c9df0, 0x55d5a39a23fa, 0x20)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/godbus/dbus/v5/transport_unix.go:21 +0x8f
io.ReadAtLeast(0x55d5a6906dc0, 0xc00086f300, 0xc000ab09c0, 0x10, 0x10, 0x10, 0xc001026a90, 0x0, 0x0)
    /usr/local/go/src/io/io.go:310 +0x89
io.ReadFull(...)
    /usr/local/go/src/io/io.go:329
github.com/docker/docker/vendor/github.com/godbus/dbus/v5.(*unixTransport).ReadMessage(0xc001056120, 0xc00000006a, 0x6a, 0xc000a92c01)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/godbus/dbus/v5/transport_unix.go:91 +0x11a
github.com/docker/docker/vendor/github.com/godbus/dbus/v5.(*Conn).inWorker(0xc0008da0e0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/godbus/dbus/v5/conn.go:348 +0x42
created by github.com/docker/docker/vendor/github.com/godbus/dbus/v5.(*Conn).Auth
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/godbus/dbus/v5/auth.go:118 +0x623

goroutine 227 [chan receive, 22940 minutes]:
github.com/docker/docker/vendor/github.com/docker/libnetwork/iptables.signalHandler()
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/iptables/firewalld.go:125 +0x56
created by github.com/docker/docker/vendor/github.com/docker/libnetwork/iptables.FirewalldInit
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/iptables/firewalld.go:79 +0x106

goroutine 346 [chan receive, 22940 minutes]:
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*scheduler).loop.func2(0xc000902d90)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/scheduler.go:76 +0x3e
created by github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*scheduler).loop
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/scheduler.go:75 +0x79

goroutine 273 [IO wait, 718 minutes]:
internal/poll.runtime_pollWait(0x7f3a2220b920, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc000861018, 0x72, 0x0, 0x0, 0x55d5a556692e)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc000861000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1fa
net.(*netFD).accept(0xc000861000, 0xc000010470, 0x0, 0x0)
    /usr/local/go/src/net/fd_unix.go:238 +0x44
net.(*UnixListener).accept(0xc000883f80, 0xc001060f48, 0xc001060f50, 0x18)
    /usr/local/go/src/net/unixsock_posix.go:162 +0x34
net.(*UnixListener).Accept(0xc000883f80, 0x55d5a68b6ea8, 0x55d5a6998020, 0xc000010470, 0xc0007e0200)
    /usr/local/go/src/net/unixsock.go:260 +0x49
github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).acceptClientConnections(0xc0007e0200, 0xc00045eb70, 0x2c, 0x55d5a695d000, 0xc000883f80)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/sandbox_externalkey_unix.go:144 +0x3d
created by github.com/docker/docker/vendor/github.com/docker/libnetwork.(*controller).startExternalKeyListener
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/sandbox_externalkey_unix.go:138 +0x339

goroutine 272 [select, 22940 minutes]:
github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/overlay.(*driver).peerOpRoutine(0xc001049700, 0x55d5a696d280, 0xc0001e9f80, 0xc00106afc0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/overlay/peerdb.go:279 +0x10c
created by github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/overlay.Init
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/overlay/overlay.go:77 +0x17f

goroutine 150 [chan receive, 22940 minutes]:
main.(*DaemonCli).setupConfigReloadTrap.func1(0xc000ebe0c0, 0xc000933500)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/cmd/dockerd/daemon_unix.go:83 +0x57
created by main.(*DaemonCli).setupConfigReloadTrap
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/cmd/dockerd/daemon_unix.go:82 +0xb7

goroutine 347 [IO wait, 718 minutes]:
internal/poll.runtime_pollWait(0x7f3a2220bd30, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc0001c4798, 0x72, 0x0, 0x0, 0x55d5a556692e)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc0001c4780, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1fa
net.(*netFD).accept(0xc0001c4780, 0x55d5a3df9ca6, 0xc00022c0a0, 0xc00118e090)
    /usr/local/go/src/net/fd_unix.go:238 +0x44
net.(*UnixListener).accept(0xc000876120, 0x600f30e2, 0xc000d45da0, 0x55d5a3a5f878)
    /usr/local/go/src/net/unixsock_posix.go:162 +0x34
net.(*UnixListener).Accept(0xc000876120, 0xc000d45df0, 0x18, 0xc0006aa300, 0x55d5a3df91c6)
    /usr/local/go/src/net/unixsock.go:260 +0x49
net/http.(*Server).Serve(0xc00022c000, 0x55d5a695d000, 0xc000876120, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2925 +0x282
github.com/docker/docker/api/server.(*HTTPServer).Serve(...)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/server.go:112
github.com/docker/docker/api/server.(*Server).serveAPI.func1(0xc000fc21e0, 0xc00000ee60)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/server.go:86 +0xd1
created by github.com/docker/docker/api/server.(*Server).serveAPI
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/server.go:83 +0x8d

goroutine 331 [sync.Cond.Wait, 628 minutes]:
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:310
sync.runtime_notifyListWait(0xc000bd7cd0, 0xc0000017e8)
    /usr/local/go/src/runtime/sema.go:510 +0xfc
sync.(*Cond).Wait(0xc000bd7cc0)
    /usr/local/go/src/sync/cond.go:56 +0x9f
github.com/docker/docker/vendor/github.com/moby/buildkit/util/cond.(*StatefulCond).Wait(0xc000c35dd0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/cond/cond.go:28 +0xb1
github.com/docker/docker/vendor/github.com/moby/buildkit/solver.(*scheduler).loop(0xc000902d90)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/scheduler.go:101 +0x18c
created by github.com/docker/docker/vendor/github.com/moby/buildkit/solver.newScheduler
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/moby/buildkit/solver/scheduler.go:35 +0x1ad

goroutine 332 [select, 22940 minutes]:
github.com/docker/docker/daemon.(*Daemon).ProcessClusterNotifications(0xc00000c1e0, 0x55d5a696d280, 0xc00004e7c0, 0xc001041380)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/daemon/events.go:126 +0xe7
created by main.(*DaemonCli).start
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/cmd/dockerd/daemon.go:234 +0xb71

goroutine 151 [chan receive, 22940 minutes]:
github.com/docker/docker/api/server.(*Server).serveAPI(0xc0004128c0, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/server.go:94 +0x14c
github.com/docker/docker/api/server.(*Server).Wait(0xc0004128c0, 0xc00106a060)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/server.go:199 +0x31
created by main.(*DaemonCli).start
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/cmd/dockerd/daemon.go:242 +0xbd6

goroutine 1091861 [runnable]:
syscall.Syscall(0x1, 0x17, 0xc00152d000, 0x2d, 0x2d, 0x2d, 0x0)
    /usr/local/go/src/syscall/asm_linux_amd64.s:18 +0x5
syscall.write(0x17, 0xc00152d000, 0x2d, 0x1000, 0x0, 0x2, 0xc000000078)
    /usr/local/go/src/syscall/zsyscall_linux_amd64.go:1005 +0x5c
syscall.Write(...)
    /usr/local/go/src/syscall/syscall_unix.go:202
internal/poll.(*FD).Write(0xc000a8c900, 0xc00152d000, 0x2d, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:268 +0x1a3
net.(*netFD).Write(0xc000a8c900, 0xc00152d000, 0x2d, 0x1000, 0x4, 0x0, 0x0)
    /usr/local/go/src/net/fd_unix.go:220 +0x51
net.(*conn).Write(0xc000010138, 0xc00152d000, 0x2d, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:196 +0x6a
net/http.checkConnErrorWriter.Write(0xc0001fef00, 0xc00152d000, 0x2d, 0x1000, 0xc0029a0d38, 0x55d5a3a945b7, 0xc0013482e0)
    /usr/local/go/src/net/http/server.go:3448 +0x58
bufio.(*Writer).Flush(0xc000b9df40, 0x0, 0x30)
    /usr/local/go/src/bufio/bufio.go:593 +0x77
net/http.(*chunkWriter).flush(0xc0013482e0)
    /usr/local/go/src/net/http/server.go:396 +0x3d
net/http.(*response).Flush(0xc0013482a0)
    /usr/local/go/src/net/http/server.go:1667 +0x49
github.com/docker/docker/pkg/ioutils.(*WriteFlusher).Flush(0xc0019d2230)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/pkg/ioutils/writeflusher.go:49 +0x7c
github.com/docker/docker/pkg/ioutils.(*WriteFlusher).Write(0xc0019d2230, 0xc0018a8060, 0x27, 0x30, 0x0, 0x30, 0x25)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/pkg/ioutils/writeflusher.go:34 +0xd9
github.com/docker/docker/api/server/router/build.(*syncWriter).Write(0xc000fa6bc0, 0xc0018a8060, 0x27, 0x30, 0x30, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/router/build/build_routes.go:321 +0x7a
github.com/docker/docker/pkg/streamformatter.(*AuxFormatter).Emit(0xc009c26180, 0x55d5a558188a, 0x13, 0x55d5a634b100, 0xc001901340, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/pkg/streamformatter/streamformatter.go:154 +0x17b
github.com/docker/docker/builder/builder-next.(*Builder).Build.func4(0xc000ab2768, 0x18)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/builder/builder-next/builder.go:406 +0xe4
github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc005ad7bc0, 0xc000fa6d20)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x66
created by github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x68

goroutine 107056 [chan send, 20827 minutes]:
github.com/docker/docker/builder/builder-next.(*statusProxy).SendMsg(0xc000c0ec00, 0x55d5a67f1480, 0xc0014cc150, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/builder/builder-next/builder.go:452 +0x78
github.com/docker/docker/vendor/github.com/moby/buildkit/control.(*Controller).Status.func2(0xc0027c1768, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/moby/buildkit/control/control.go:352 +0x757
github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc0013fdce0, 0xc000c0ec20)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x66
created by github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x68

goroutine 1091858 [IO wait, 718 minutes]:
internal/poll.runtime_pollWait(0x7f3a186b2610, 0x72, 0xffffffffffffffff)
    /usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc000a8c918, 0x72, 0x0, 0x1, 0xffffffffffffffff)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000a8c900, 0xc005ad7001, 0x1, 0x1, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1d1
net.(*netFD).Read(0xc000a8c900, 0xc005ad7001, 0x1, 0x1, 0x2, 0x0, 0x55d5a399ad00)
    /usr/local/go/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc000010138, 0xc005ad7001, 0x1, 0x1, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:184 +0x6a
net/http.(*connReader).backgroundRead(0xc005ad6ff0)
    /usr/local/go/src/net/http/server.go:688 +0x5a
created by net/http.(*connReader).startBackgroundRead
    /usr/local/go/src/net/http/server.go:684 +0xd6

goroutine 1091537 [chan receive, 718 minutes]:
github.com/docker/docker/vendor/github.com/moby/buildkit/session.(*Manager).handleConn(0xc000ae2ca0, 0x55d5a696d280, 0xc001502340, 0x55d5a6998020, 0xc000010148, 0xc005d077d0, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/moby/buildkit/session/manager.go:144 +0x4c4
github.com/docker/docker/vendor/github.com/moby/buildkit/session.(*Manager).HandleHTTPRequest(0xc000ae2ca0, 0x55d5a696d340, 0xc001cad6e0, 0x55d5a695d280, 0xc0008dad20, 0xc0007e0c00, 0xc0000bb500, 0x55d5a39a8b14)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/moby/buildkit/session/manager.go:91 +0x7cf
github.com/docker/docker/api/server/router/session.(*sessionRouter).startSession(0xc000bf7830, 0x55d5a696d340, 0xc001cad6e0, 0x55d5a695d280, 0xc0008dad20, 0xc0007e0c00, 0xc005d078f0, 0xc0000bb501, 0xc001522f20)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/router/session/session_routes.go:11 +0x6b
github.com/docker/docker/api/server/middleware.ExperimentalMiddleware.WrapHandler.func1(0x55d5a696d340, 0xc001cad6e0, 0x55d5a695d280, 0xc0008dad20, 0xc0007e0c00, 0xc005d078f0, 0x55d5a696d340, 0xc001cad6e0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/middleware/experimental.go:26 +0x177
github.com/docker/docker/api/server/middleware.VersionMiddleware.WrapHandler.func1(0x55d5a696d340, 0xc005d079e0, 0x55d5a695d280, 0xc0008dad20, 0xc0007e0c00, 0xc005d078f0, 0x55d5a39a1b72, 0xc0012134e0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/middleware/version.go:62 +0x5fb
github.com/docker/docker/pkg/authorization.(*Middleware).WrapHandler.func1(0x55d5a696d340, 0xc005d079e0, 0x55d5a695d280, 0xc0008dad20, 0xc0007e0c00, 0xc005d078f0, 0xc0012134e0, 0x55d5a6562901)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/pkg/authorization/middleware.go:59 +0x826
github.com/docker/docker/api/server/middleware.DebugRequestMiddleware.func1(0x55d5a696d340, 0xc005d079e0, 0x55d5a695d280, 0xc0008dad20, 0xc0007e0c00, 0xc005d078f0, 0x55d5a696d340, 0xc005d079e0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/middleware/debug.go:53 +0x4a0
github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1(0x55d5a695d280, 0xc0008dad20, 0xc0007e0b00)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/server.go:141 +0x241
net/http.HandlerFunc.ServeHTTP(0xc000a8b2e0, 0x55d5a695d280, 0xc0008dad20, 0xc0007e0b00)
    /usr/local/go/src/net/http/server.go:2036 +0x46
github.com/docker/docker/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc00084e300, 0x55d5a695d280, 0xc0008dad20, 0xc001f0d300)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/gorilla/mux/mux.go:210 +0xe4
net/http.serverHandler.ServeHTTP(0xc00022c000, 0x55d5a695d280, 0xc0008dad20, 0xc001f0d300)
    /usr/local/go/src/net/http/server.go:2831 +0xa6
net/http.(*conn).serve(0xc00019cfa0, 0x55d5a696d280, 0xc00150e6c0)
    /usr/local/go/src/net/http/server.go:1919 +0x877
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:2957 +0x386

goroutine 1091 [select]:
github.com/docker/docker/vendor/github.com/docker/libnetwork/osl.removeUnusedPaths()
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/osl/namespace_linux.go:105 +0x2dd
created by github.com/docker/docker/vendor/github.com/docker/libnetwork/osl.createBasePath
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/osl/namespace_linux.go:90 +0xbd

goroutine 107097 [semacquire, 21691 minutes]:
sync.runtime_Semacquire(0xc0013fdcf0)
    /usr/local/go/src/runtime/sema.go:56 +0x44
sync.(*WaitGroup).Wait(0xc0013fdce8)
    /usr/local/go/src/sync/waitgroup.go:130 +0x66
github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Wait(0xc0013fdce0, 0xc000c0ec20, 0x55d5a696d280)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:40 +0x33
github.com/docker/docker/vendor/github.com/moby/buildkit/control.(*Controller).Status(0xc000f4e510, 0xc0013fdc50, 0x55d5a698d8e0, 0xc000c0ec00, 0xc0044a0eb8, 0xc000b7ef60)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/moby/buildkit/control/control.go:362 +0x1a5
github.com/docker/docker/builder/builder-next.(*Builder).Build.func3(0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/builder/builder-next/builder.go:397 +0x14b
github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc001df5740, 0xc001df5770)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x66
created by github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x68

goroutine 1091839 [select, 718 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0015021c0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:69 +0xc4
created by github.com/docker/docker/vendor/google.golang.org/grpc.newCCBalancerWrapper
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:60 +0x16f

goroutine 107016 [select, 21691 minutes]:
github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress.(*MultiReader).Reader.func1(0x55d5a696d340, 0xc00321c180, 0xc001df5a10, 0xc0001b7880)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress/multireader.go:37 +0xc3
created by github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress.(*MultiReader).Reader
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress/multireader.go:36 +0x1c4

goroutine 102478 [semacquire, 21691 minutes]:
sync.runtime_Semacquire(0xc001df5750)
    /usr/local/go/src/runtime/sema.go:56 +0x44
sync.(*WaitGroup).Wait(0xc001df5748)
    /usr/local/go/src/sync/waitgroup.go:130 +0x66
github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Wait(0xc001df5740, 0xc000c3d5c0, 0xc003676960)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:40 +0x33
github.com/docker/docker/builder/builder-next.(*Builder).Build(0xc000c0f180, 0x55d5a696d280, 0xc00381d880, 0x55d5a6944200, 0x55d5a7d619e8, 0x55d5a6905200, 0xc000c3d440, 0x55d5a6905a60, 0xc000c3d460, 0x55d5a6905a60, ...)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/builder/builder-next/builder.go:413 +0xe6d
github.com/docker/docker/api/server/backend/build.(*Backend).Build(0xc000c35e30, 0x55d5a696d340, 0xc001df5290, 0x55d5a6944200, 0x55d5a7d619e8, 0x55d5a6905200, 0xc000c3d440, 0x55d5a6905a60, 0xc000c3d460, 0x55d5a6905a60, ...)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/backend/build/backend.go:64 +0x141
github.com/docker/docker/api/server/router/build.(*buildRouter).postBuild(0xc000bd7ec0, 0x55d5a696d340, 0xc001df5290, 0x55d5a695d280, 0xc00366ce00, 0xc00148a500, 0xc001df4f90, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/router/build/build_routes.go:282 +0x731
github.com/docker/docker/api/server/middleware.ExperimentalMiddleware.WrapHandler.func1(0x55d5a696d340, 0xc001df5290, 0x55d5a695d280, 0xc00366ce00, 0xc00148a500, 0xc001df4f90, 0x55d5a696d340, 0xc001df5290)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/middleware/experimental.go:26 +0x177
github.com/docker/docker/api/server/middleware.VersionMiddleware.WrapHandler.func1(0x55d5a696d340, 0xc001df5050, 0x55d5a695d280, 0xc00366ce00, 0xc00148a500, 0xc001df4f90, 0xc00003c000, 0x55d5a64e1800)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/middleware/version.go:62 +0x5fb
github.com/docker/docker/pkg/authorization.(*Middleware).WrapHandler.func1(0x55d5a696d340, 0xc001df5050, 0x55d5a695d280, 0xc00366ce00, 0xc00148a500, 0xc001df4f90, 0x2, 0x55d5a6562901)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/pkg/authorization/middleware.go:59 +0x826
github.com/docker/docker/api/server/middleware.DebugRequestMiddleware.func1(0x55d5a696d340, 0xc001df5050, 0x55d5a695d280, 0xc00366ce00, 0xc00148a500, 0xc001df4f90, 0x55d5a696d340, 0xc001df5050)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/middleware/debug.go:25 +0x8ae
github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1(0x55d5a695d280, 0xc00366ce00, 0xc00148a400)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/server.go:141 +0x241
net/http.HandlerFunc.ServeHTTP(0xc000a8ad80, 0x55d5a695d280, 0xc00366ce00, 0xc00148a400)
    /usr/local/go/src/net/http/server.go:2036 +0x46
github.com/docker/docker/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc00084e300, 0x55d5a695d280, 0xc00366ce00, 0xc002ad2900)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/gorilla/mux/mux.go:210 +0xe4
net/http.serverHandler.ServeHTTP(0xc00022c000, 0x55d5a695d280, 0xc00366ce00, 0xc002ad2900)
    /usr/local/go/src/net/http/server.go:2831 +0xa6
net/http.(*conn).serve(0xc000d17540, 0x55d5a696d280, 0xc0011c8580)
    /usr/local/go/src/net/http/server.go:1919 +0x877
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:2957 +0x386

goroutine 1091840 [chan receive, 718 minutes]:
github.com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc00174a2c0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:1179 +0x6ec
created by github.com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).connect
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:815 +0x12a

goroutine 1091863 [chan send]:
github.com/docker/docker/builder/builder-next.(*statusProxy).SendMsg(0xc000fa6d60, 0x55d5a67f1480, 0xc00192bc00, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/builder/builder-next/builder.go:452 +0x78
github.com/docker/docker/vendor/github.com/moby/buildkit/control.(*Controller).Status.func2(0xc00121c768, 0x55d5a399a2a5)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/moby/buildkit/control/control.go:352 +0x757
github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc005ad7ce0, 0xc000fa6d80)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x66
created by github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x68

goroutine 1091535 [semacquire, 718 minutes]:
sync.runtime_Semacquire(0xc005ad7bd0)
    /usr/local/go/src/runtime/sema.go:56 +0x44
sync.(*WaitGroup).Wait(0xc005ad7bc8)
    /usr/local/go/src/sync/waitgroup.go:130 +0x66
github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Wait(0xc005ad7bc0, 0xc000fa6d20, 0xc002e68180)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:40 +0x33
github.com/docker/docker/builder/builder-next.(*Builder).Build(0xc000c0f180, 0x55d5a696d280, 0xc000f4b680, 0x55d5a6944200, 0x55d5a7d619e8, 0x55d5a6905200, 0xc000fa6bc0, 0x55d5a6905a60, 0xc000fa6be0, 0x55d5a6905a60, ...)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/builder/builder-next/builder.go:413 +0xe6d
github.com/docker/docker/api/server/backend/build.(*Backend).Build(0xc000c35e30, 0x55d5a696d340, 0xc005ad76e0, 0x55d5a6944200, 0x55d5a7d619e8, 0x55d5a6905200, 0xc000fa6bc0, 0x55d5a6905a60, 0xc000fa6be0, 0x55d5a6905a60, ...)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/backend/build/backend.go:64 +0x141
github.com/docker/docker/api/server/router/build.(*buildRouter).postBuild(0xc000bd7ec0, 0x55d5a696d340, 0xc005ad76e0, 0x55d5a695d280, 0xc0013482a0, 0xc001f0d600, 0xc005ad7530, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/router/build/build_routes.go:282 +0x731
github.com/docker/docker/api/server/middleware.ExperimentalMiddleware.WrapHandler.func1(0x55d5a696d340, 0xc005ad76e0, 0x55d5a695d280, 0xc0013482a0, 0xc001f0d600, 0xc005ad7530, 0x55d5a696d340, 0xc005ad76e0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/middleware/experimental.go:26 +0x177
github.com/docker/docker/api/server/middleware.VersionMiddleware.WrapHandler.func1(0x55d5a696d340, 0xc005ad7650, 0x55d5a695d280, 0xc0013482a0, 0xc001f0d600, 0xc005ad7530, 0xc00003c000, 0x55d5a64e1800)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/middleware/version.go:62 +0x5fb
github.com/docker/docker/pkg/authorization.(*Middleware).WrapHandler.func1(0x55d5a696d340, 0xc005ad7650, 0x55d5a695d280, 0xc0013482a0, 0xc001f0d600, 0xc005ad7530, 0x2, 0x55d5a6562901)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/pkg/authorization/middleware.go:59 +0x826
github.com/docker/docker/api/server/middleware.DebugRequestMiddleware.func1(0x55d5a696d340, 0xc005ad7650, 0x55d5a695d280, 0xc0013482a0, 0xc001f0d600, 0xc005ad7530, 0x55d5a696d340, 0xc005ad7650)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/middleware/debug.go:25 +0x8ae
github.com/docker/docker/api/server.(*Server).makeHTTPHandler.func1(0x55d5a695d280, 0xc0013482a0, 0xc001f0d500)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/api/server/server.go:141 +0x241
net/http.HandlerFunc.ServeHTTP(0xc000a8ad80, 0x55d5a695d280, 0xc0013482a0, 0xc001f0d500)
    /usr/local/go/src/net/http/server.go:2036 +0x46
github.com/docker/docker/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc00084e300, 0x55d5a695d280, 0xc0013482a0, 0xc001f0d200)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/gorilla/mux/mux.go:210 +0xe4
net/http.serverHandler.ServeHTTP(0xc00022c000, 0x55d5a695d280, 0xc0013482a0, 0xc001f0d200)
    /usr/local/go/src/net/http/server.go:2831 +0xa6
net/http.(*conn).serve(0xc0001fef00, 0x55d5a696d280, 0xc000c19b40)
    /usr/local/go/src/net/http/server.go:1919 +0x877
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:2957 +0x386

goroutine 1091874 [IO wait]:
internal/poll.runtime_pollWait(0x7f3a2220b5e0, 0x72, 0xffffffffffffffff)
    /usr/local/go/src/runtime/netpoll.go:184 +0x57
internal/poll.(*pollDesc).wait(0xc000a8cc98, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x47
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000a8cc80, 0xc001b1a000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:169 +0x1d1
net.(*netFD).Read(0xc000a8cc80, 0xc001b1a000, 0x8000, 0x8000, 0x0, 0x800010601, 0x7f3900000000)
    /usr/local/go/src/net/fd_unix.go:202 +0x51
net.(*conn).Read(0xc000010148, 0xc001b1a000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:184 +0x6a
bufio.(*Reader).Read(0xc00539af00, 0xc0021001f8, 0x9, 0x9, 0xc000520e00, 0x7f3a22336d98, 0x0)
    /usr/local/go/src/bufio/bufio.go:226 +0x26c
io.ReadAtLeast(0x55d5a6904ce0, 0xc00539af00, 0xc0021001f8, 0x9, 0x9, 0x9, 0x55d5a40d5e77, 0xc00782206c, 0xc002887de8)
    /usr/local/go/src/io/io.go:310 +0x89
io.ReadFull(...)
    /usr/local/go/src/io/io.go:329
github.com/docker/docker/vendor/golang.org/x/net/http2.readFrameHeader(0xc0021001f8, 0x9, 0x9, 0x55d5a6904ce0, 0xc00539af00, 0x0, 0xbffc143200000000, 0x4e3d5b037f3df, 0x55d5a7d419e0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/golang.org/x/net/http2/frame.go:237 +0x89
github.com/docker/docker/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0021001c0, 0xc007822060, 0xc007822060, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/golang.org/x/net/http2/frame.go:492 +0xa3
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00022ea80)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1265 +0x185
created by github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:300 +0xd25

goroutine 1091841 [select]:
github.com/docker/docker/vendor/github.com/moby/buildkit/session.monitorHealth(0x55d5a696d280, 0xc001502340, 0xc000521500, 0xc000278e30)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/moby/buildkit/session/grpc.go:87 +0x1cd
created by github.com/docker/docker/vendor/github.com/moby/buildkit/session.grpcClientConn
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/moby/buildkit/session/grpc.go:73 +0x335

goroutine 1091785 [select, 718 minutes]:
github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress.(*MultiReader).Reader.func1(0x55d5a696d340, 0xc001c8f830, 0xc0022ad2c0, 0xc001169a00)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress/multireader.go:37 +0xc3
created by github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress.(*MultiReader).Reader
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/progress/multireader.go:36 +0x1c4

goroutine 1091875 [select]:
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00269dd10, 0x1, 0x0, 0x0, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:395 +0x124
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00539afc0, 0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:513 +0x1e5
github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc00022ea80)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:346 +0x7d
created by github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:344 +0xedd

goroutine 1091860 [semacquire, 718 minutes]:
sync.runtime_Semacquire(0xc005ad7cf0)
    /usr/local/go/src/runtime/sema.go:56 +0x44
sync.(*WaitGroup).Wait(0xc005ad7ce8)
    /usr/local/go/src/sync/waitgroup.go:130 +0x66
github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Wait(0xc005ad7ce0, 0xc000fa6d80, 0x55d5a696d280)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:40 +0x33
github.com/docker/docker/vendor/github.com/moby/buildkit/control.(*Controller).Status(0xc000f4e510, 0xc005ad7cb0, 0x55d5a698d8e0, 0xc000fa6d60, 0xc000ab5f10, 0x10)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/github.com/moby/buildkit/control/control.go:362 +0x1a5
github.com/docker/docker/builder/builder-next.(*Builder).Build.func3(0x0, 0x0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/builder/builder-next/builder.go:397 +0x14b
github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc005ad7bc0, 0xc005ad7bf0)
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x66
created by github.com/docker/docker/vendor/golang.org/x/sync/errgroup.(*Group).Go
    /root/rpmbuild/BUILD/src/engine/.gopath/src/github.com/docker/docker/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x68
thompson-shaun commented 3 months ago

Going waaay back here, is this still an issues @xkszltl?