Race condition with --mount=type=cache when uid/gid is set

[griffous]

I am experiencing an easily reproducable issue trying to build multistage images, when the following conditions are true:

• docker-container based buildkitd
• Multistage build with buildx bake
• Sub-stages use '--mount=type=cache' with uid/gid defined e.g RUN --mount=type=cache,uid=101,gid=1001,target=/cache cmd_here
• It is buildkitd's first build. (on retry, it will work)

Attached is a simple test case, showing the problem against both v0.8.2 and master

docker-compose.yml.txt Dockerfile.txt testscript.txt

The sh error messages are typically something like:

#16 0.152 /bin/sh: can't create /cache/shared: nonexistent directory
#16 0.586 /bin/sh: /cache/shared: No such file or directory                                                                                                                  

On one occasion, I saw this slightly more interesting error: error: failed to solve: rpc error: code = Unknown desc = executor failed running [/bin/sh -c ls -alh ${CACHE_DIR} && echo "shared_cache1" > ${CACHE_DIR}/shared && echo "local_cache1" > ${CACHE_DIR}/local_1 && cat ${CACHE_DIR}/* && sleep 5]: failed to mount /tmp/buildkit-mount103821201: [{Type:overlay Source:overlay Options:[index=off workdir=/var/lib/buildkit/runc-overlayfs/snapshots/snapshots/18/work upperdir=/var/lib/buildkit/runc-overlayfs/snapshots/snapshots/18/fs lowerdir=/var/lib/buildkit/runc-overlayfs/snapshots/snapshots/11/fs]}]: no such file or directory

[tonistiigi]

I tried to debug this and can reproduce but not quite sure where it is coming from. I wonder if this is just a side effect of parallel file write access on overlay filesystem. The difference to uid/gid is that without it there is only one layer and no overlay. I can reproduce same case without uid if I mount a subfolder with from.

@AkihiroSuda Any thoughts?

[griffous]

Any updates on this one? It's still broken in the latest master.