search

vmware-archive / buildkit-cli-for-kubectl

BuildKit CLI for kubectl is a tool for building container images with your Kubernetes cluster
Other
501 stars 40 forks source link

stuck on "sending tarball" #40

Open cppforlife opened 3 years ago

cppforlife commented 3 years ago

What steps did you take and what happened

things were working ok for a few hours. then i issued 3 builds in parallel (same single go file app submitted 3 times). they got stuck (not sure where; lost the output). after that i've tried to issue single build and now buildkit is stuck (been sitting at #10 sending tarball for 10+ minutes). getting stuck on sending tarball happens consistently. this continues to happen even after rm/create cycle.

% kbld -f <(pbpaste )
simple-app-two | starting build (using kubectl buildkit): assets/simple-app -> kbld:rand-1605358502697084000-246112051777-simple-app-two
simple-app-two | #2 [internal] load .dockerignore
simple-app-two | #2 transferring context: 2B 0.1s done
simple-app-two | #2 DONE 0.1s
simple-app-two |
simple-app-two | #1 [internal] load build definition from Dockerfile.dev
simple-app-two | #1 transferring dockerfile: 36B 0.1s done
simple-app-two | #1 DONE 0.1s
simple-app-two |
simple-app-two | #3 [internal] load metadata for docker.io/library/golang:1.10.1
simple-app-two | #3 DONE 0.6s
simple-app-two |
simple-app-two | #4 [1/5] FROM docker.io/library/golang:1.10.1@sha256:4826b5c314a498142c7291...
simple-app-two | #4 resolve docker.io/library/golang:1.10.1@sha256:4826b5c314a498142c7291ad835ab6be1bf02f7813d6932d01f1f0f1383cdda1 done
simple-app-two | #4 DONE 0.0s
simple-app-two |
simple-app-two | #6 [internal] load build context
simple-app-two | #6 transferring context: 229B 0.2s done
simple-app-two | #6 DONE 0.2s
simple-app-two |
simple-app-two | #7 [3/5] COPY . .
simple-app-two | #7 CACHED
simple-app-two |
simple-app-two | #8 [4/5] RUN CGO_ENABLED=0 GOOS=linux go build -v -o app
simple-app-two | #8 CACHED
simple-app-two |
simple-app-two | #5 [2/5] WORKDIR /go/src/github.com/mchmarny/simple-app/
simple-app-two | #5 CACHED
simple-app-two |
simple-app-two | #9 [5/5] RUN cp /go/src/github.com/mchmarny/simple-app/app /app
simple-app-two | #9 CACHED
simple-app-two |
simple-app-two | #10 exporting to oci image format
simple-app-two | #10 exporting layers 0.0s done
simple-app-two | #10 exporting manifest sha256:1d7a647a5fc69531d12481f9d82dd0ca07b9cb17ee545cd304166fab21ef6ab9 0.0s done
simple-app-two | #10 exporting config sha256:753b4895cb1ec7ffa40877a4c47edbf762685358f67d06e5303de78ee747fe69 0.0s done
simple-app-two | #10 sending tarball

(command: /usr/local/bin/kubectl-buildkit build --progress=plain --file dev/Dockerfile.dev --tag kbld:rand-1605358502697084000-246112051777-simple-app-two .)

What did you expect to happen

not get stuck.

Environment Details:

error is happening on single pod buildkit. ive got 5 nodes in cluster. previously had buildkit installed with 5 nodes.

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.12-gke.1504", GitCommit:"17061f5bd4ee34f72c9281d49f94b4f3ac31ac25", GitTreeState:"clean", BuildDate:"2020-10-19T17:00:22Z", GoVersion:"go1.13.15b4", Compiler:"gc", Platform:"linux/amd64"}

Builder Logs Logs: 

time="2020-11-14T12:46:42Z" level=info msg="auto snapshotter: using overlayfs"
time="2020-11-14T12:46:42Z" level=warning msg="using host network as the default"
time="2020-11-14T12:46:42Z" level=info msg="found worker \"2vfh56sddvftg38os08qsxhuh\", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:buildkit-7cbb8bf988-wdpxt org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/amd64 linux/386]"
time="2020-11-14T12:46:42Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
time="2020-11-14T12:46:42Z" level=info msg="found 1 workers, default=\"2vfh56sddvftg38os08qsxhuh\""
time="2020-11-14T12:46:42Z" level=warning msg="currently, only the default worker can be used."
time="2020-11-14T12:46:42Z" level=info msg="running server on /run/buildkit/buildkitd.sock"
time="2020-11-14T12:54:51Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unavailable desc = transport is closing\ngithub.com/moby/buildkit/session/filesync.(*streamWriterCloser).Write\n\t/src/session/filesync/diffcopy.go:45\nbufio.(*Writer).Write\n\t/usr/local/go/src/bufio/bufio.go:625\narchive/tar.(*regFileWriter).Write\n\t/usr/local/go/src/archive/tar/writer.go:497\narchive/tar.(*Writer).Write\n\t/usr/local/go/src/archive/tar/writer.go:435\nio.(*multiWriter).Write\n\t/usr/local/go/src/io/multi.go:60\nio.copyBuffer\n\t/usr/local/go/src/io/io.go:404\nio.Copy\n\t/usr/local/go/src/io/io.go:364\ngithub.com/containerd/containerd/images/archive.blobRecord.func1\n\t/src/vendor/github.com/containerd/containerd/images/archive/exporter.go:299\ngithub.com/containerd/containerd/images/archive.writeTar\n\t/src/vendor/github.com/containerd/containerd/images/archive/exporter.go:456\ngithub.com/containerd/containerd/images/archive.Export\n\t/src/vendor/github.com/containerd/containerd/images/archive/exporter.go:248\ngithub.com/moby/buildkit/exporter/oci.(*imageExporterInstance).Export\n\t/src/exporter/oci/export.go:183\ngithub.com/moby/buildkit/solver/llbsolver.(*Solver).Solve.func3\n\t/src/solver/llbsolver/solver.go:218\ngithub.com/moby/buildkit/solver/llbsolver.inVertexContext\n\t/src/solver/llbsolver/solver.go:359\ngithub.com/moby/buildkit/solver/llbsolver.(*Solver).Solve\n\t/src/solver/llbsolver/solver.go:217\ngithub.com/moby/buildkit/control.(*Controller).Solve\n\t/src/control/control.go:276\ngithub.com/moby/buildkit/api/services/control._Control_Solve_Handler.func1\n\t/src/api/services/control/control.pb.go:1489\ngithub.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1\n\t/src/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/server.go:57\nmain.unaryInterceptor.func1\n\t/src/cmd/buildkitd/main.go:531\ngithub.com/moby/buildkit/api/services/control._Control_Solve_Handler\n\t/src/api/services/control/control.pb.go:1491\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/src/vendor/google.golang.org/grpc/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/src/vendor/google.golang.org/grpc/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/src/vendor/google.golang.org/grpc/server.go:722\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357\nfailed to copy to tar\ngithub.com/containerd/containerd/images/archive.blobRecord.func1\n\t/src/vendor/github.com/containerd/containerd/images/archive/exporter.go:301\ngithub.com/containerd/containerd/images/archive.writeTar\n\t/src/vendor/github.com/containerd/containerd/images/archive/exporter.go:456\ngithub.com/containerd/containerd/images/archive.Export\n\t/src/vendor/github.com/containerd/containerd/images/archive/exporter.go:248\ngithub.com/moby/buildkit/exporter/oci.(*imageExporterInstance).Export\n\t/src/exporter/oci/export.go:183\ngithub.com/moby/buildkit/solver/llbsolver.(*Solver).Solve.func3\n\t/src/solver/llbsolver/solver.go:218\ngithub.com/moby/buildkit/solver/llbsolver.inVertexContext\n\t/src/solver/llbsolver/solver.go:359\ngithub.com/moby/buildkit/solver/llbsolver.(*Solver).Solve\n\t/src/solver/llbsolver/solver.go:217\ngithub.com/moby/buildkit/control.(*Controller).Solve\n\t/src/control/control.go:276\ngithub.com/moby/buildkit/api/services/control._Control_Solve_Handler.func1\n\t/src/api/services/control/control.pb.go:1489\ngithub.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1\n\t/src/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/server.go:57\nmain.unaryInterceptor.func1\n\t/src/cmd/buildkitd/main.go:531\ngithub.com/moby/buildkit/api/services/control._Control_Solve_Handler\n\t/src/api/services/control/control.pb.go:1491\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/src/vendor/google.golang.org/grpc/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/src/vendor/google.golang.org/grpc/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/src/vendor/google.golang.org/grpc/server.go:722\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"
time="2020-11-14T13:00:55Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Canceled desc = context canceled\ngithub.com/moby/buildkit/session/filesync.(*streamWriterCloser).Write\n\t/src/session/filesync/diffcopy.go:45\nbufio.(*Writer).Write\n\t/usr/local/go/src/bufio/bufio.go:625\narchive/tar.(*regFileWriter).Write\n\t/usr/local/go/src/archive/tar/writer.go:497\narchive/tar.(*Writer).Write\n\t/usr/local/go/src/archive/tar/writer.go:435\nio.(*multiWriter).Write\n\t/usr/local/go/src/io/multi.go:60\nio.copyBuffer\n\t/usr/local/go/src/io/io.go:404\nio.Copy\n\t/usr/local/go/src/io/io.go:364\ngithub.com/containerd/containerd/images/archive.blobRecord.func1\n\t/src/vendor/github.com/containerd/containerd/images/archive/exporter.go:299\ngithub.com/containerd/containerd/images/archive.writeTar\n\t/src/vendor/github.com/containerd/containerd/images/archive/exporter.go:456\ngithub.com/containerd/containerd/images/archive.Export\n\t/src/vendor/github.com/containerd/containerd/images/archive/exporter.go:248\ngithub.com/moby/buildkit/exporter/oci.(*imageExporterInstance).Export\n\t/src/exporter/oci/export.go:183\ngithub.com/moby/buildkit/solver/llbsolver.(*Solver).Solve.func3\n\t/src/solver/llbsolver/solver.go:218\ngithub.com/moby/buildkit/solver/llbsolver.inVertexContext\n\t/src/solver/llbsolver/solver.go:359\ngithub.com/moby/buildkit/solver/llbsolver.(*Solver).Solve\n\t/src/solver/llbsolver/solver.go:217\ngithub.com/moby/buildkit/control.(*Controller).Solve\n\t/src/control/control.go:276\ngithub.com/moby/buildkit/api/services/control._Control_Solve_Handler.func1\n\t/src/api/services/control/control.pb.go:1489\ngithub.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1\n\t/src/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/server.go:57\nmain.unaryInterceptor.func1\n\t/src/cmd/buildkitd/main.go:531\ngithub.com/moby/buildkit/api/services/control._Control_Solve_Handler\n\t/src/api/services/control/control.pb.go:1491\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/src/vendor/google.golang.org/grpc/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/src/vendor/google.golang.org/grpc/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/src/vendor/google.golang.org/grpc/server.go:722\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357\nfailed to copy to tar\ngithub.com/containerd/containerd/images/archive.blobRecord.func1\n\t/src/vendor/github.com/containerd/containerd/images/archive/exporter.go:301\ngithub.com/containerd/containerd/images/archive.writeTar\n\t/src/vendor/github.com/containerd/containerd/images/archive/exporter.go:456\ngithub.com/containerd/containerd/images/archive.Export\n\t/src/vendor/github.com/containerd/containerd/images/archive/exporter.go:248\ngithub.com/moby/buildkit/exporter/oci.(*imageExporterInstance).Export\n\t/src/exporter/oci/export.go:183\ngithub.com/moby/buildkit/solver/llbsolver.(*Solver).Solve.func3\n\t/src/solver/llbsolver/solver.go:218\ngithub.com/moby/buildkit/solver/llbsolver.inVertexContext\n\t/src/solver/llbsolver/solver.go:359\ngithub.com/moby/buildkit/solver/llbsolver.(*Solver).Solve\n\t/src/solver/llbsolver/solver.go:217\ngithub.com/moby/buildkit/control.(*Controller).Solve\n\t/src/control/control.go:276\ngithub.com/moby/buildkit/api/services/control._Control_Solve_Handler.func1\n\t/src/api/services/control/control.pb.go:1489\ngithub.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1\n\t/src/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/server.go:57\nmain.unaryInterceptor.func1\n\t/src/cmd/buildkitd/main.go:531\ngithub.com/moby/buildkit/api/services/control._Control_Solve_Handler\n\t/src/api/services/control/control.pb.go:1491\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/src/vendor/google.golang.org/grpc/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/src/vendor/google.golang.org/grpc/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/src/vendor/google.golang.org/grpc/server.go:722\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"
time="2020-11-14T13:03:04Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unavailable desc = transport is closing\ngithub.com/moby/buildkit/session/filesync.(*streamWriterCloser).Write\n\t/src/session/filesync/diffcopy.go:45\nbufio.(*Writer).Write\n\t/usr/local/go/src/bufio/bufio.go:625\narchive/tar.(*regFileWriter).Write\n\t/usr/local/go/src/archive/tar/writer.go:497\narchive/tar.(*Writer).Write\n\t/usr/local/go/src/archive/tar/writer.go:435\nio.(*multiWriter).Write\n\t/usr/local/go/src/io/multi.go:60\nio.copyBuffer\n\t/usr/local/go/src/io/io.go:404\nio.Copy\n\t/usr/local/go/src/io/io.go:364\ngithub.com/containerd/containerd/images/archive.blobRecord.func1\n\t/src/vendor/github.com/containerd/containerd/images/archive/exporter.go:299\ngithub.com/containerd/containerd/images/archive.writeTar\n\t/src/vendor/github.com/containerd/containerd/images/archive/exporter.go:456\ngithub.com/containerd/containerd/images/archive.Export\n\t/src/vendor/github.com/containerd/containerd/images/archive/exporter.go:248\ngithub.com/moby/buildkit/exporter/oci.(*imageExporterInstance).Export\n\t/src/exporter/oci/export.go:183\ngithub.com/moby/buildkit/solver/llbsolver.(*Solver).Solve.func3\n\t/src/solver/llbsolver/solver.go:218\ngithub.com/moby/buildkit/solver/llbsolver.inVertexContext\n\t/src/solver/llbsolver/solver.go:359\ngithub.com/moby/buildkit/solver/llbsolver.(*Solver).Solve\n\t/src/solver/llbsolver/solver.go:217\ngithub.com/moby/buildkit/control.(*Controller).Solve\n\t/src/control/control.go:276\ngithub.com/moby/buildkit/api/services/control._Control_Solve_Handler.func1\n\t/src/api/services/control/control.pb.go:1489\ngithub.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1\n\t/src/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/server.go:57\nmain.unaryInterceptor.func1\n\t/src/cmd/buildkitd/main.go:531\ngithub.com/moby/buildkit/api/services/control._Control_Solve_Handler\n\t/src/api/services/control/control.pb.go:1491\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/src/vendor/google.golang.org/grpc/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/src/vendor/google.golang.org/grpc/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/src/vendor/google.golang.org/grpc/server.go:722\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357\nfailed to copy to tar\ngithub.com/containerd/containerd/images/archive.blobRecord.func1\n\t/src/vendor/github.com/containerd/containerd/images/archive/exporter.go:301\ngithub.com/containerd/containerd/images/archive.writeTar\n\t/src/vendor/github.com/containerd/containerd/images/archive/exporter.go:456\ngithub.com/containerd/containerd/images/archive.Export\n\t/src/vendor/github.com/containerd/containerd/images/archive/exporter.go:248\ngithub.com/moby/buildkit/exporter/oci.(*imageExporterInstance).Export\n\t/src/exporter/oci/export.go:183\ngithub.com/moby/buildkit/solver/llbsolver.(*Solver).Solve.func3\n\t/src/solver/llbsolver/solver.go:218\ngithub.com/moby/buildkit/solver/llbsolver.inVertexContext\n\t/src/solver/llbsolver/solver.go:359\ngithub.com/moby/buildkit/solver/llbsolver.(*Solver).Solve\n\t/src/solver/llbsolver/solver.go:217\ngithub.com/moby/buildkit/control.(*Controller).Solve\n\t/src/control/control.go:276\ngithub.com/moby/buildkit/api/services/control._Control_Solve_Handler.func1\n\t/src/api/services/control/control.pb.go:1489\ngithub.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1\n\t/src/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/server.go:57\nmain.unaryInterceptor.func1\n\t/src/cmd/buildkitd/main.go:531\ngithub.com/moby/buildkit/api/services/control._Control_Solve_Handler\n\t/src/api/services/control/control.pb.go:1491\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/src/vendor/google.golang.org/grpc/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/src/vendor/google.golang.org/grpc/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/src/vendor/google.golang.org/grpc/server.go:722\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"

Dockerfile app: https://github.com/vmware-tanzu/carvel-kbld/tree/develop/test/e2e/assets/simple-app

dhiltgen commented 3 years ago

During early development, I saw some hangs that sound similar, so I think we do have some corner cases or flakiness somewhere in the kubectl exec flow. Getting some parallel tests running in CI will also be helpful to squash these parallelism bugs.

dhiltgen commented 3 years ago

We've added various hardening improvements on parallel build scenarios in the most recent release. It might be worth giving your test scenario another try on 0.1.3 to see if the problem is resolved or if we've still got more raciness to squash.