search

docker / buildx

Docker CLI plugin for extended build capabilities with BuildKit
Apache License 2.0
3.47k stars 467 forks source link

Buildx bake large groups fails when setting DOCKER_HOST=ssh://... #976

Open fenollp opened 2 years ago

fenollp commented 2 years ago 
λ DOCKER_HOST=ssh://toaster.local docker version
Client: Docker Engine - Community
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.16.12
 Git commit:        e91ed57
 Built:             Mon Dec 13 11:45:33 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.12
  Git commit:       459d0df
  Built:            Mon Dec 13 11:43:41 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Reproduce with commit: https://github.com/fenollp/mediapipe-desktop-linux-demos/commit/79e73bc29c635b64872ea79cc1169dad0ccb09d8

Builds fail in various states of progression but always end up complaining about stderr=kex_exchange_identification: Connection closed by remote host

λ DOCKER_HOST=ssh://toaster.local docker buildx bake git://github.com/fenollp/mediapipe-desktop-linux-demos#main bins
[+] Building 3.5s (1/1) FINISHED                                                                                                                                                              
 => ERROR [face_detection_cpu internal] load git source git://github.com/fenollp/mediapipe-desktop-linux-demos#main                                                                      0.2s
------                                                                                                                                                                                        
 > [face_detection_cpu internal] load git source git://github.com/fenollp/mediapipe-desktop-linux-demos#main:                                                                                 
#1 0.787 79e73bc29c635b64872ea79cc1169dad0ccb09d8       refs/heads/main                                                                                                                       
#1 0.616 From git://github.com/fenollp/mediapipe-desktop-linux-demos                                                                                                                          
#1 0.616  * [new branch]      main       -> main
#1 0.617  * [new branch]      main       -> origin/main
------
error: failed to dial gRPC: command [ssh -- toaster.local docker system dial-stdio] has exited with exit status 255, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=kex_exchange_identification: Connection closed by remote host
Connection closed by 192.168.1.29 port 22
λ DOCKER_HOST=ssh://toaster.local docker buildx bake git://github.com/fenollp/mediapipe-desktop-linux-demos#main bins
[+] Building 3.6s (2/2) FINISHED                                                                                                                                                              
 => CACHED [libs internal] load git source git://github.com/fenollp/mediapipe-desktop-linux-demos#main                                                                                   0.0s
 => CANCELED [iris_tracking_gpu] resolve image config for docker.io/docker/dockerfile:1@sha256:42399d4635eddd7a9b8a24be879d2f9a930d0ed040a61324cfdf59ef1357b3b2                          0.0s 
error: failed to dial gRPC: command [ssh -- toaster.local docker system dial-stdio] has exited with exit status 255, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=kex_exchange_identification: Connection closed by remote host
Connection closed by 192.168.1.29 port 22
λ DOCKER_HOST=ssh://toaster.local docker buildx bake git://github.com/fenollp/mediapipe-desktop-linux-demos#main bins
[+] Building 3.5s (1/1) FINISHED                                                                                                                                                              
 => [object_detection_gpu internal] load git source git://github.com/fenollp/mediapipe-desktop-linux-demos#main                                                                          0.3s
error: failed to dial gRPC: command [ssh -- toaster.local docker system dial-stdio] has exited with exit status 255, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=kex_exchange_identification: Connection closed by remote host
Connection closed by 192.168.1.29 port 22

With no DOCKER_HOST the builds make more progress but seem to hit a race condition looking for cached data (--mount=type=cache,target=/root/.cache/bazel):

λ unset DOCKER_HOST; docker buildx bake git://github.com/fenollp/mediapipe-desktop-linux-demos#main bins
[+] Building 12.8s (18/66)                                                                                                                                                                    
 => CACHED [face_mesh_cpu internal] load git source git://github.com/fenollp/mediapipe-desktop-linux-demos#main                                                                          0.0s
 => [face_mesh_cpu] resolve image config for docker.io/docker/dockerfile:1@sha256:42399d4635eddd7a9b8a24be879d2f9a930d0ed040a61324cfdf59ef1357b3b2                                       0.0s 
 => CACHED [face_mesh_cpu] docker-image://docker.io/docker/dockerfile:1@sha256:42399d4635eddd7a9b8a24be879d2f9a930d0ed040a61324cfdf59ef1357b3b2                                          0.0s
 => [face_mesh_cpu internal] load metadata for docker.io/library/ubuntu:18.04@sha256:c2aa13782650aa7ade424b12008128b60034c795f25456e8eb552d0a0f447cad                                    0.0s
 => [face_mesh_cpu internal] load metadata for docker.io/library/alpine@sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300                                          0.0s
 => [face_mesh_cpu ubuntu 1/1] FROM docker.io/library/ubuntu:18.04@sha256:c2aa13782650aa7ade424b12008128b60034c795f25456e8eb552d0a0f447cad                                               0.0s
 => [face_mesh_cpu alpine 1/1] FROM docker.io/library/alpine@sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300                                                     0.0s
 => CACHED [face_mesh_cpu mediapipe-src 1/2] WORKDIR /w                                                                                                                                  0.0s
 => CACHED [face_mesh_cpu mediapipe-src 2/2] RUN set -ux  && apk add --no-cache git  && git init  && git remote add origin https://github.com/google/mediapipe  && git fetch --depth 1   0.0s
 => CACHED [face_mesh_cpu base 1/6] WORKDIR /mediapipe                                                                                                                                   0.0s
 => CACHED [face_mesh_cpu base 2/6] RUN set -ux  && apt update  && apt install -y --no-install-recommends         build-essential         gcc-8 g++-8         ca-certificates         c  0.0s
 => CACHED [face_mesh_cpu base 3/6] RUN set -ux  && update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-8 800 --slave /usr/bin/g++ g++ /usr/bin/g++-8  && update-alternatives -  0.0s
 => CACHED [face_mesh_cpu base 4/6] RUN set -ux  && pip3 install --upgrade setuptools  && pip3 install wheel  && pip3 install future  && pip3 install six==1.14.0                        0.0s
 => CACHED [face_mesh_cpu base 5/6] RUN set -ux  && mkdir /bazel  && wget --no-check-certificate -O /bazel/installer.sh "https://github.com/bazelbuild/bazel/releases/download/4.2.1/ba  0.0s
 => CACHED [face_mesh_cpu base 6/6] RUN set -ux  && mkdir /x  && echo ' --platform_suffix=-cpu' >/bazelflags                                                                             0.0s
 => CACHED [hand_tracking_gpu base-gpu 1/5] RUN set -ux  && apt install -y --no-install-recommends         mesa-common-dev         libegl1-mesa-dev         libgles2-mesa-dev         m  0.0s
 => ERROR [hand_tracking_gpu base-gpu 2/5] RUN   --mount=from=mediapipe-src,source=/w,target=/mediapipe,rw   --mount=type=cache,target=/root/.cache/bazel     set -ux  && bazel build $  7.5s
 => CANCELED [selfie_segmentation_cpu base-cpu 1/4] RUN   --mount=from=mediapipe-src,source=/w,target=/mediapipe,rw   --mount=type=cache,target=/root/.cache/bazel     set -ux  && baze  8.2s
------                                                                                                                                                                                        
 > [hand_tracking_gpu base-gpu 2/5] RUN   --mount=from=mediapipe-src,source=/w,target=/mediapipe,rw   --mount=type=cache,target=/root/.cache/bazel     set -ux  && bazel build $(cat /bazelflags)                 -c opt                 --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11                 mediapipe/examples/desktop/hello_world:hello_world  && cp ./bazel-bin/mediapipe/examples/desktop/hello_world/hello_world /x/hello_world_gpu:                                                                                                                                
#17 1.331 + cat /bazelflags                                                                                                                                                                   
#17 0.909 + cat /bazelflags                                                                                                                                                                   
#17 1.331 + cat /bazelflags                                                                                                                                                                   
#17 1.331 + cat /bazelflags                                                                                                                                                                   
#17 1.331 + cat /bazelflags                                                                                                                                                                   
#17 1.331 + cat /bazelflags                                                                                                                                                                   
#17 0.910 + cat /bazelflags                                                                                                                                                                   
#17 0.909 + cat /bazelflags                                                                                                                                                                   
#17 1.331 + cat /bazelflags
#17 1.331 + cat /bazelflags
#17 1.332 + bazel build --platform_suffix=-gpu -c opt --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 mediapipe/examples/desktop/hello_world:hello_world
#17 1.332 + bazel build --platform_suffix=-gpu -c opt --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 mediapipe/examples/desktop/hello_world:hello_world
#17 1.332 + bazel build --platform_suffix=-gpu -c opt --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 mediapipe/examples/desktop/hello_world:hello_world
#17 0.910 + bazel build --platform_suffix=-gpu -c opt --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 mediapipe/examples/desktop/hello_world:hello_world
#17 1.332 + bazel build --platform_suffix=-gpu -c opt --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 mediapipe/examples/desktop/hello_world:hello_world
#17 1.332 + bazel build --platform_suffix=-gpu -c opt --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 mediapipe/examples/desktop/hello_world:hello_world
#17 1.332 + bazel build --platform_suffix=-gpu -c opt --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 mediapipe/examples/desktop/hello_world:hello_world
#17 0.910 + bazel build --platform_suffix=-gpu -c opt --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 mediapipe/examples/desktop/hello_world:hello_world
#17 1.332 + bazel build --platform_suffix=-gpu -c opt --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 mediapipe/examples/desktop/hello_world:hello_world
#17 0.910 + bazel build --platform_suffix=-gpu -c opt --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 mediapipe/examples/desktop/hello_world:hello_world
#17 0.940 Starting local Bazel server and connecting to it...
#17 1.362 Starting local Bazel server and connecting to it...
#17 1.362 Starting local Bazel server and connecting to it...
#17 1.362 Starting local Bazel server and connecting to it...
#17 0.940 Starting local Bazel server and connecting to it...
#17 1.362 Starting local Bazel server and connecting to it...
#17 1.362 Starting local Bazel server and connecting to it...
#17 0.940 Starting local Bazel server and connecting to it...
#17 1.362 Starting local Bazel server and connecting to it...
#17 1.362 Starting local Bazel server and connecting to it...
#17 3.059 Loading: 
#17 3.480 Loading: 
#17 3.480 Loading: 
#17 3.059 Loading: 
#17 3.480 Loading: 
#17 3.059 Loading: 
#17 3.480 Loading: 
#17 3.480 Loading: 
#17 3.480 Loading: 
#17 3.480 Loading: 
#17 3.063 Loading: 0 packages loaded
#17 3.485 Loading: 0 packages loaded
#17 3.063 Loading: 0 packages loaded
#17 3.063 Loading: 0 packages loaded
#17 3.485 Loading: 0 packages loaded
#17 3.485 Loading: 0 packages loaded
#17 3.485 Loading: 0 packages loaded
#17 3.485 Loading: 0 packages loaded
#17 3.485 Loading: 0 packages loaded
#17 3.485 Loading: 0 packages loaded
#17 4.502 Loading: 0 packages loaded
#17 4.080 Loading: 0 packages loaded
#17 4.502 Loading: 0 packages loaded
#17 4.080 Loading: 0 packages loaded
#17 4.080 Loading: 0 packages loaded
#17 4.502 Loading: 0 packages loaded
#17 4.502 Loading: 0 packages loaded
#17 4.502 Loading: 0 packages loaded
#17 4.502 Loading: 0 packages loaded
#17 4.502 Loading: 0 packages loaded
#17 5.498 Loading: 0 packages loaded
#17 5.498 Loading: 0 packages loaded
#17 5.498 Loading: 0 packages loaded
#17 5.498 Loading: 0 packages loaded
#17 5.076 Loading: 0 packages loaded
#17 5.076 Loading: 0 packages loaded
#17 5.076 Loading: 0 packages loaded
#17 5.498 Loading: 0 packages loaded
#17 5.498 Loading: 0 packages loaded
#17 5.498 Loading: 0 packages loaded
#17 6.498 Loading: 0 packages loaded
#17 6.498 Loading: 0 packages loaded
#17 6.077 Loading: 0 packages loaded
#17 6.077 Loading: 0 packages loaded
#17 6.077 Loading: 0 packages loaded
#17 6.498 Loading: 0 packages loaded
#17 6.499 Loading: 0 packages loaded
#17 6.499 Loading: 0 packages loaded
#17 6.498 Loading: 0 packages loaded
#17 6.498 Loading: 0 packages loaded
#17 7.761 
#17 7.761 Server terminated abruptly (error code: 14, error message: 'Socket closed', log file: '/root/.cache/bazel/_bazel_root/4884d566396e9b67b62185751879ad14/server/jvm.out')
#17 7.761 
#17 7.339 
#17 7.339 Server terminated abruptly (error code: 14, error message: 'Socket closed', log file: '/root/.cache/bazel/_bazel_root/4884d566396e9b67b62185751879ad14/server/jvm.out')
#17 7.339 
#17 7.761 
#17 7.761 Server terminated abruptly (error code: 14, error message: 'Socket closed', log file: '/root/.cache/bazel/_bazel_root/4884d566396e9b67b62185751879ad14/server/jvm.out')
#17 7.761 
#17 7.761 
#17 7.761 Server terminated abruptly (error code: 14, error message: 'Socket closed', log file: '/root/.cache/bazel/_bazel_root/4884d566396e9b67b62185751879ad14/server/jvm.out')
#17 7.761 
#17 7.761 
#17 7.761 Server terminated abruptly (error code: 14, error message: 'Socket closed', log file: '/root/.cache/bazel/_bazel_root/4884d566396e9b67b62185751879ad14/server/jvm.out')
#17 7.761 
#17 7.761 
#17 7.761 Server terminated abruptly (error code: 14, error message: 'Socket closed', log file: '/root/.cache/bazel/_bazel_root/4884d566396e9b67b62185751879ad14/server/jvm.out')
#17 7.761 
#17 7.339 
#17 7.339 Server terminated abruptly (error code: 14, error message: 'Socket closed', log file: '/root/.cache/bazel/_bazel_root/4884d566396e9b67b62185751879ad14/server/jvm.out')
#17 7.339 
#17 7.339 
#17 7.339 Server terminated abruptly (error code: 14, error message: 'Socket closed', log file: '/root/.cache/bazel/_bazel_root/4884d566396e9b67b62185751879ad14/server/jvm.out')
#17 7.339 
#17 7.761 
#17 7.761 Server terminated abruptly (error code: 14, error message: 'Socket closed', log file: '/root/.cache/bazel/_bazel_root/4884d566396e9b67b62185751879ad14/server/jvm.out')
#17 7.761 
#17 7.761 
#17 7.761 Server terminated abruptly (error code: 14, error message: 'Socket closed', log file: '/root/.cache/bazel/_bazel_root/4884d566396e9b67b62185751879ad14/server/jvm.out')
#17 7.761 
------
error: failed to solve: executor failed running [/bin/sh -c set -ux  && bazel build $(cat /bazelflags)                 -c opt                 --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11                 mediapipe/examples/desktop/hello_world:hello_world  && cp ./bazel-bin/mediapipe/examples/desktop/hello_world/hello_world /x/hello_world_gpu]: exit code: 37

Note: all these are attempting to build a large-ish group of targets which independently all compile without issue:

λ DOCKER_HOST=ssh://toaster.local docker buildx bake git://github.com/fenollp/mediapipe-desktop-linux-demos#main 
[+] Building 577.4s (27/27) FINISHED                                                                                                                                                          
 => CACHED [libs internal] load git source git://github.com/fenollp/mediapipe-desktop-linux-demos#main                                                                                   0.0s
 => [libs] resolve image config for docker.io/docker/dockerfile:1@sha256:42399d4635eddd7a9b8a24be879d2f9a930d0ed040a61324cfdf59ef1357b3b2                                                1.5s 
 => [auth] docker/dockerfile:pull token for registry-1.docker.io                                                                                                                         0.0s
 => [face_detection_cpu] docker-image://docker.io/docker/dockerfile:1@sha256:42399d4635eddd7a9b8a24be879d2f9a930d0ed040a61324cfdf59ef1357b3b2                                            1.1s
 => => resolve docker.io/docker/dockerfile:1@sha256:42399d4635eddd7a9b8a24be879d2f9a930d0ed040a61324cfdf59ef1357b3b2                                                                     0.0s
 => => sha256:e532695ddd93ca7c85a816c67afdb352e91052fab7ac19a675088f80915779a7 1.21kB / 1.21kB                                                                                           0.0s
 => => sha256:24a639a53085eb680e1d11618ac62f3977a3926fedf5b8471ace519b8c778030 9.67MB / 9.67MB                                                                                           0.3s
 => => sha256:42399d4635eddd7a9b8a24be879d2f9a930d0ed040a61324cfdf59ef1357b3b2 2.00kB / 2.00kB                                                                                           0.0s
 => => sha256:93f32bd6dd9004897fed4703191f48924975081860667932a4df35ba567d7426 528B / 528B                                                                                               0.0s
 => => extracting sha256:24a639a53085eb680e1d11618ac62f3977a3926fedf5b8471ace519b8c778030                                                                                                0.6s
 => [face_detection_cpu internal] load metadata for docker.io/library/ubuntu:18.04@sha256:c2aa13782650aa7ade424b12008128b60034c795f25456e8eb552d0a0f447cad                               1.1s
 => [libs internal] load metadata for docker.io/library/alpine@sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300                                                   1.0s
 => [auth] library/alpine:pull token for registry-1.docker.io                                                                                                                            0.0s
 => [auth] library/ubuntu:pull token for registry-1.docker.io                                                                                                                            0.0s
 => [libs ubuntu 1/1] FROM docker.io/library/ubuntu:18.04@sha256:c2aa13782650aa7ade424b12008128b60034c795f25456e8eb552d0a0f447cad                                                        5.8s
 => => resolve docker.io/library/ubuntu:18.04@sha256:c2aa13782650aa7ade424b12008128b60034c795f25456e8eb552d0a0f447cad                                                                    0.0s
 => => sha256:c2aa13782650aa7ade424b12008128b60034c795f25456e8eb552d0a0f447cad 1.41kB / 1.41kB                                                                                           0.0s
 => => sha256:98706f0f213dbd440021993a82d2f70451a73698315370ae8615cc468ac06624 529B / 529B                                                                                               0.0s
 => => sha256:dcf4d4bef137f695d11ed187ba6a135362dca3de36955c4da0905d596ce521bc 1.46kB / 1.46kB                                                                                           0.0s
 => => sha256:68e7bb398b9ff421236990bfeaf5c1feab26c590eed93489e245375c23551e2a 26.71MB / 26.71MB                                                                                         2.5s
 => => extracting sha256:68e7bb398b9ff421236990bfeaf5c1feab26c590eed93489e245375c23551e2a                                                                                                3.0s
 => [face_detection_cpu alpine 1/1] FROM docker.io/library/alpine@sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300                                                2.6s
 => => resolve docker.io/library/alpine@sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300                                                                          0.0s
 => => sha256:c059bfaa849c4d8e4aecaeb3a10c2d9b3d85f5165c66ad3a4d937758128c4d18 1.47kB / 1.47kB                                                                                           0.0s
 => => sha256:59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3 2.82MB / 2.82MB                                                                                           0.3s
 => => sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300 1.64kB / 1.64kB                                                                                           0.0s
 => => sha256:e7d88de73db3d3fd9b2d63aa7f447a10fd0220b7cbf39803c803f2af9ba256b3 528B / 528B                                                                                               0.0s
 => => extracting sha256:59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3                                                                                                0.9s
 => [libs mediapipe-src 1/2] WORKDIR /w                                                                                                                                                  0.5s
 => [face_detection_cpu mediapipe-src 2/2] RUN set -ux  && apk add --no-cache git  && git init  && git remote add origin https://github.com/google/mediapipe  && git fetch --depth 1 o  23.5s
 => [libs base 1/6] WORKDIR /mediapipe                                                                                                                                                   0.6s
 => [libs base 2/6] RUN set -ux  && apt update  && apt install -y --no-install-recommends         build-essential         gcc-8 g++-8         ca-certificates         curl         ff  186.7s
 => [libs base 3/6] RUN set -ux  && update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-8 800 --slave /usr/bin/g++ g++ /usr/bin/g++-8  && update-alternatives --install /usr/bi  0.5s
 => [libs base 4/6] RUN set -ux  && pip3 install --upgrade setuptools  && pip3 install wheel  && pip3 install future  && pip3 install six==1.14.0                                       11.5s 
 => [libs base 5/6] RUN set -ux  && mkdir /bazel  && wget --no-check-certificate -O /bazel/installer.sh "https://github.com/bazelbuild/bazel/releases/download/4.2.1/bazel-4.2.1-instal  4.3s 
 => [face_detection_cpu base 6/6] RUN set -ux  && mkdir /x  && echo ' --platform_suffix=-cpu' >/bazelflags                                                                               0.6s 
 => [face_detection_cpu base-cpu 1/4] RUN   --mount=from=mediapipe-src,source=/w,target=/mediapipe,rw   --mount=type=cache,target=/root/.cache/bazel     set -ux  && bazel build $(cat  10.8s 
 => [libs base-cpu 2/4] RUN   --mount=from=mediapipe-src,source=/w,target=/mediapipe,rw   --mount=type=cache,target=/root/.cache/bazel     set -ux  && bazel build $(cat /bazelflags)  234.6s 
 => [face_detection_cpu base-cpu 3/4] RUN   --mount=from=mediapipe-src,source=/w,target=/mediapipe,rw   --mount=type=cache,target=/root/.cache/bazel     set -ux  && bazel build $(cat  24.0s 
 => [libs base-cpu 4/4] RUN   --mount=from=mediapipe-src,source=/w,target=/mediapipe,rw   --mount=type=cache,target=/root/.cache/bazel     set -ux  && bazel build $(cat /bazelflags)   22.8s 
 => [face_detection_cpu builder-face_detection_cpu 1/1] RUN   --mount=from=mediapipe-src,source=/w,target=/mediapipe,rw   --mount=type=cache,target=/root/.cache/bazel     set -ux  &&  69.7s 
 => [libs libs 1/2] COPY --from=base-cpu     /usr/lib/x86_64-linux-gnu/libCharLS.so.1     /usr/lib/x86_64-linux-gnu/libEGL.so.1     /usr/lib/x86_64-linux-gnu/libGLdispatch.so.0     /u  0.9s 
 => [libs libs 2/2] COPY --from=base-cpu     /usr/lib/libgdal.so.20     /lib/x86_64-linux-gnu/libjson-c.so.3     /usr/lib/libarmadillo.so.8     /usr/lib/libogdi.so.3.2     /            5.2s 
 => [face_detection_cpu] exporting to client                                                                                                                                             0.5s 
 => => copying files 8.67MB                                                                                                                                                              0.5s 
 => [face_detection_cpu face_detection_cpu 1/1] COPY --from=builder-face_detection_cpu /x/face_detection_cpu /                                                                           0.1s

Can this be failing because "too many" SSH connections are being attempted and something rate-limits this either in openssh-server or docker and so fails one job and cancels the rest? Aren't connections pooled and shouldn't they be?

I'm happy to help on this and maybe help write the fix as well.

tonistiigi commented 2 years ago

I think the connection should be multiplexed(ControlPath) already but didn't verify atm. Same should be for the grpc level where we don't need more than one connection. You could verify with just ps aux how many processes there are and if they all use the same master socket path.

Server terminated abruptly does not look like related to our code.