Build fails with large context over SSH

andrewrynhard commented 3 years ago

I have a builder that is setup with SSH, and when building projects with larger contexts I see the following:

#10 [internal] load build context
#10 sha256:1b6ab60be38a09b7ca89d54bee19f0b64482887ff6b887eef34c84ab89671f0d
#10 transferring context: 29.14MB 3.4s
#10 transferring context: 73.51MB 8.5s
#10 transferring context: 73.89MB 15.3s
#10 transferring context: 74.21MB 17.4s done
#10 ERROR: rpc error: code = Canceled desc = grpc: the client connection is closing
------
 > [internal] load build context:
------
error: failed to solve: rpc error: code = Unknown desc = rpc error: code = Canceled desc = grpc: the client connection is closing

Version of buildx is 0.5.1.

tonistiigi commented 3 years ago

Anything more interesting in the logs? Is the connection just dropping?

andrewrynhard commented 3 years ago

Not much. This is all that I get really. The connection drops and that is the end of it.

MichalAugustyn commented 3 years ago

I have the same problem while using kubernetes driver. You can find the pod's log below

time="2021-05-27T15:16:52Z" level=warning msg="using host network as the default"
time="2021-05-27T15:16:52Z" level=info msg="found worker \"z8nw3lfxs3fvricxi7s2fch0s\", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:pr-823-2270-6c87477976-mr7vb org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/amd64 linux/386]"
time="2021-05-27T15:16:52Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
time="2021-05-27T15:16:52Z" level=info msg="found 1 workers, default=\"z8nw3lfxs3fvricxi7s2fch0s\""
time="2021-05-27T15:16:52Z" level=warning msg="currently, only the default worker can be used."
time="2021-05-27T15:16:52Z" level=info msg="running server on /run/buildkit/buildkitd.sock"
time="2021-05-27T15:17:12Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Canceled desc = grpc: the client connection is closing\n"
time="2021-05-27T15:17:12Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Canceled desc = grpc: the client connection is closing\n"
time="2021-05-27T15:17:13Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unavailable desc = transport is closing\n"
time="2021-05-27T15:17:13Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Canceled desc = grpc: the client connection is closing\n"
time="2021-05-27T15:17:13Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Canceled desc = context canceled\n"
time="2021-05-27T15:17:13Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Canceled desc = context canceled\n"
time="2021-05-27T15:17:13Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Canceled desc = context canceled\n"
time="2021-05-27T15:17:13Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unavailable desc = transport is closing\n"
time="2021-05-27T15:17:13Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unavailable desc = transport is closing\n"
time="2021-05-27T15:17:13Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Canceled desc = context canceled\n"
time="2021-05-27T15:17:13Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Canceled desc = context canceled\n"
time="2021-05-27T15:17:13Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Canceled desc = grpc: the client connection is closing\n"
time="2021-05-27T15:17:13Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Canceled desc = context canceled\n"
time="2021-05-27T15:17:13Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Canceled desc = context canceled\n"
time="2021-05-27T15:17:14Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Canceled desc = grpc: the client connection is closing\n"
time="2021-05-27T15:17:14Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unavailable desc = transport is closing\n"
time="2021-05-27T15:17:14Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Canceled desc = context canceled\n"
time="2021-05-27T15:17:14Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Canceled desc = grpc: the client connection is closing\n"
time="2021-05-27T15:17:14Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Canceled desc = context canceled\n"
time="2021-05-27T15:17:14Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unavailable desc = transport is closing\n"
time="2021-05-27T15:17:14Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Canceled desc = context canceled\n"
time="2021-05-27T15:17:15Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Canceled desc = context canceled\n"
time="2021-05-27T15:17:15Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unavailable desc = transport is closing\n"
time="2021-05-27T15:17:15Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unavailable desc = transport is closing\n"
time="2021-05-27T15:17:15Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Canceled desc = context canceled\n"
time="2021-05-27T15:17:15Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Canceled desc = context canceled\n"
time="2021-05-27T15:17:15Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Canceled desc = grpc: the client connection is closing\n"
time="2021-05-27T15:17:15Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unavailable desc = transport is closing\n"
time="2021-05-27T15:17:16Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Canceled desc = context canceled\n"
time="2021-05-27T15:17:16Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unavailable desc = transport is closing\n"
time="2021-05-27T15:17:16Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Canceled desc = context canceled\n"
time="2021-05-27T15:17:16Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Canceled desc = context canceled\n"
time="2021-05-27T15:17:29Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Canceled desc = context canceled\n"
time="2021-05-27T15:17:29Z" level=info msg="stopping server"

javidaslan commented 3 years ago

Are there any updates on this? Were you able solve this?

sakoht commented 2 years ago

Same issue. I have a large compile/build step cached, and it seems to be "unpushable".

 => ERROR exporting to oci image format                                                                                                                                                      57.6s
 => => exporting layers                                                                                                                                                                       0.0s
 => => exporting manifest sha256:ae226ccf6e09dfa3365a93608b0fe93cdf93b8c13f0c34f0f23e552f19ed6d36                                                                                             0.0s
 => => exporting config sha256:668415544e0e1043c1c5ec63f722fd57eda9f26fae7801378a2129895f088b55                                                                                               0.0s
 => => sending tarball                                                                                                                                                                       57.0s
------
 > exporting to oci image format:
------
error: failed to solve: Canceled: grpc: the client connection is closing

dioguerra commented 1 year ago

Same here, same error as @sakoht

dgageot commented 1 year ago

Have you tried to setup SSH multiplexing? I found that it improves performance and stability a lot because, otherwise, buildx tends to open a lot of connections.

https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Multiplexing

docker / buildx

Build fails with large context over SSH #485