Stubborn buildx error: mounting "overlay" to rootfs at "/var/lib/buildkit/runc-overlayfs/executor/bnlx2eoyoa5lq8oie878pvqss/rootfs/dest" caused: no such file or directory

[jamshid]

Sorry I haven't been able to reduce this to a small reproducible example but is anyone else getting mounting "overlay" to rootfs ... no such fileerrors like below?

I just started using buildx to make multiplatform images. Getting the failure on a simple COPY command in my Dockerfile. It builds fine with docker build ... (non-multiplatform).

Running Docker for Mac 3.3.1. I get the failure even when building only linux/amd64, I removed linux/arm64.

docker buildx build  --push --platform linux/amd64 --progress=plain --build-arg=... --secret id=sshknownhosts,src=.../known_hosts --secret id=sshpriv,src=.../.ssh/id_mygitkey -t=192.168.1.50:5000/myimage:mytag myimage)

...
#46 sha256:f8242732b621eb082cade21e7b6545ac733b9fd14649a7be1a27ccd92283e764
#46 CACHED

#47 [stage-2 34/40] COPY socat.service /etc/systemd/system/socat.service
#47 sha256:bbdce2136343d1ab75457a330c556412b3c9e7e6ff75e67aa2265c9db2b3bbca
#47 0.079 container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: rootfs_linux.go:60: mounting "overlay" to rootfs at "/var/lib/buildkit/runc-overlayfs/executor/6ptgmfj4a5oclrgp6phoispsb/rootfs/dest" caused: no such file or directory
#47 ERROR: executor failed running [copy /src-0/socat.service etc/systemd/system/socat.service]: exit code: 1
------
 > [stage-2 34/40] COPY socat.service /etc/systemd/system/socat.service:
------
error: failed to solve: rpc error: code = Unknown desc = executor failed running [copy /src-0/socat.service etc/systemd/system/socat.service]: exit code: 1

I'm building a multiplatform docker image, pulling in some files from other builds.

ARG REGISTRY_URL
ARG MYIMAGE_BRANCH_TAG
...
FROM ${REGISTRY_URL}myimage-src:${MYIMAGE_BRANCH_TAG} AS my-image-src
FROM ${REGISTRY_URL}ui:${CSUI_GIT_BUILD} AS my-ui

FROM centos:7.9.2009

RUN mkdir /deliverables

ARG MYIMAGE_BRANCH_TAG
COPY --from=my-image-src /deliverables/*.rpm /deliverables/
...
ARG UI_GIT_BUILD
COPY --from=my-ui /deliverables/*.rpm /deliverables/
...
RUN yum install -y socat && yum clean all
# Used when using systemd (instead of starting socats in foreground in /myinit.sh)
COPY socat.service /etc/systemd/system/socat.service
...
COPY myinit.sh /myinit.sh
RUN chmod +x /myinit.sh
COPY supervisord.conf.in /supervisord.conf.in
...
CMD ["/myinit.sh"]

I've tried building with --no-cache, doesn't help. Tried deleting and recreating the builder, doesn't help. Tried docker system prune --volumes, doesn't help.

[tonistiigi]

As you suggested, we would need a reproducer to debug this.

Looking at the error though, what is the version of buildkit you are using with buildx? The codepath that is producing the error should only run in quite old buildkit.

[jamshid]

I'm using Docker for Mac 3.3.1 connecting to another Docker for Mac server. Let me know if I can provide any other info or enable debugging to help. I'll continue trying to narrow down to reproduce.

% docker buildx version
github.com/docker/buildx v0.5.1-docker 11057da37336192bfc57d81e02359ba7ba848e4a
% docker buildx ls     
NAME/NODE    DRIVER/ENDPOINT             STATUS  PLATFORMS
mybuilder *  docker-container                    
  mybuilder0 unix:///var/run/docker.sock running linux/amd64*, linux/arm64*, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
default      docker                              
  default    default                     running linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

% docker context ls
NAME                TYPE                DESCRIPTION                               DOCKER ENDPOINT           KUBERNETES ENDPOINT                                 ORCHESTRATOR
default *           moby                Current DOCKER_HOST based configuration   tcp://192.168.1.50:2376   https://kubernetes.docker.internal:6443 (default)   kubernetes

[tonistiigi]

docker run --rm $(docker inspect -f '{{.Config.Image}}' buildx_buildkit_ mybuilder0) --version

[jamshid]

Intersting, I guess the mybuilder instance is on my local docker server, not the remote docker server.

% echo $DOCKER_HOST $DOCKER_TLS_VERIFY $DOCKER_CERT_PATH
tcp://192.168.1.50:2376 1 /Users/jamshid/.docker/machine/machines/mini
% docker inspect -f '{{.Config.Image}}' buildx_buildkit_mybuilder0
Error: No such object: buildx_buildkit_mybuilder0

% DOCKER_HOST= DOCKER_TLS_VERIFY= docker inspect -f '{{.Config.Image}}' buildx_buildkit_mybuilder0
moby/buildkit:buildx-stable-1

% DOCKER_HOST= DOCKER_TLS_VERIFY= docker run --rm moby/buildkit:buildx-stable-1 --version                            
buildkitd github.com/moby/buildkit v0.8.2 9065b18ba4633c75862befca8188de4338d9f94a