search

moby / buildkit

concurrent, cache-efficient, and Dockerfile-agnostic builder toolkit
https://github.com/moby/moby/issues/34227
Apache License 2.0
8.12k stars 1.15k forks source link

Panic during build #3085

Open ciaranmcnulty opened 2 years ago

ciaranmcnulty commented 2 years ago

Context: a docker compose build on a fresh checkout of a project. Some images were pulling, presumably something timed out when authenticating?

Hoping you can see what's happened from the line numbers maybe

 => => transferring context: 2B                                                                                                                                        0.0s
panic: send on closed channelrunner runtime_base  1/11] FROM public.ecr.aws/lambda/provided                                                                            0.0s
 => [[REDACTED]-mysql internal] load metadata for [REDACTED]/mysql:develop                                               1.7s
goroutine 466 [running]:tification-api-mock internal] load metadata for docker.io/wiremock/wiremock:2.31.0                                                             1.2s
github.com/docker/buildx/util/progress.(*Printer).Write(0x140001e7701?, 0x10142659c?)                                                                                  0.0s
 => [autgithub.com/docker/buildx@v0.8.2/util/progress/printer.go:41 +0x2c                                                                                              0.0s
github.com/docker/buildx/build.Build.func3(0x1014d8ee3?)da-layer-cache:pull token for [REDACTED]                                                      0.0s
 => [autgithub.com/docker/buildx@v0.8.2/build/build.go:681 +0x34 => [[REDACTED]-mysql internal] load .dockerignore                                                                                                          0.0s

github.com/moby/buildkit/util/progress/progresswriter.Wrap({0x140006e6770, 0x6d}, 0x1400000ca38, 0x140001e7920)
        github.com/moby/buildkit@v0.10.1-0.20220403220257-10e6f94bf90d/util/progress/progresswriter/progress.go: => => transferring context: 2B                                                                                                                                        0.0s
24 +0x1c8
github.com/moby/buildkit/session/auth/authprovider.(*authProvider).Fetch => [[REDACTED]_lambda-runner internal] load metadata for public.ecr.aws/lambda/provided:latest                                                                  0.0s
(0x14000240990, {0x1019a35c8, 0x1400045eae0}, 0x140002bfbc0)
        github.com/moby/buildkit@v0.10.1-0.20220403220257-10e6f94bf90d/session/auth/authprovider/authprovider.go:88 +0x364
github.com/moby/buildkit/session/auth._Auth_FetchToken_Handler.func1({0x1019a35c8, 0x1400045eae0}, {0x101951b40?, 0x140002bfbc0})
        github.com/moby/buildkit@v0.10.1-0.20220403220257-10e6f94bf90d/session/auth/auth.pb.go:954 +0x78
github.com/moby/buildkit/util/grpcerrors.UnaryServerInterceptor({0x1019a35c8?, 0x1400045eae0?}, {0x101951b40?, 0x140002bfbc0?}, 0x14000746ad8?, 0x101170b08?)
        github.com/moby/buildkit@v0.10.1-0.20220403220257-10e6f94bf90d/util/grpcerrors/intercept.go:14 +0x38
github.com/moby/buildkit/session/auth._Auth_FetchToken_Handler({0x1018d66c0?, 0x14000240990}, {0x1019a35c8, 0x1400045eae0}, 0x140002bfb60, 0x10198f0e0)
        github.com/moby/buildkit@v0.10.1-0.20220403220257-10e6f94bf90d/session/auth/auth.pb.go:956 +0x13c
google.golang.org/grpc.(*Server).processUnaryRPC(0x140001d8000, {0x1019a88f8, 0x1400072fcb0}, 0x140000d9e60, 0x14000343620, 0x10218d718, 0x0)
        google.golang.org/grpc@v1.45.0/server.go:1282 +0xb3c
google.golang.org/grpc.(*Server).handleStream(0x140001d8000, {0x1019a88f8, 0x1400072fcb0}, 0x140000d9e60, 0x0)
        google.golang.org/grpc@v1.45.0/server.go:1619 +0x840
google.golang.org/grpc.(*Server).serveStreams.func1.2()
        google.golang.org/grpc@v1.45.0/server.go:921 +0x88
created by google.golang.org/grpc.(*Server).serveStreams.func1
        google.golang.org/grpc@v1.45.0/server.go:919 +0x298
make: *** [test] Error 2
ciaranmcnulty commented 2 years ago

Ah, I was not authenticated with the [REDACTED] registry - retrying gave me a 401 error instead

jedevc commented 2 years ago

So had a brief investigation: I think what's causing this is that the AuthProvider on the client is queried for some token, but at the point that we receive the request, we've now finished the build and shut down the progress writer - however the AuthProvider is still listening and handling incoming requests potentially.

Not quite sure, it's not immediately obvious why that channel has been closed.

tonistiigi commented 2 years ago

So this is actually a docker buildx issue but I can't transfer cross-org. The progress writer lib should make it sure that if the progresswriter is closed then running goroutines can't write to it. Also, all child goroutines should be closed before the progresswriter gets closed.