[v0.14] dockerfile: mkdir xxxx not a directory - with COPY --link

[tetienne-zenchef]

Hi,

we face this issue with v0.14.1 (and 0.14.0), but was OK with v0.13.2.

0.038 runc run failed: unable to start container process: error during container init: mkdir /var/www: not a directory

To reproduce

FROM public.ecr.aws/debian/debian:11-slim@sha256:1fec6f6a4f281dc401bf40553e9ddc5524714020be06c865d42870c549662dcd AS base-production

WORKDIR /var/www
COPY --link readme.md . # whatever the file
RUN echo "Hello world"

❯ docker buildx create --name ko --driver=docker-container --driver-opt image=moby/buildkit:v0.14.1
ko
❯ docker buildx build  --builder=ko .                                                                                                                 
[+] Building 1.2s (10/10) FINISHED                                                                                                                          docker-container:ko
 => [internal] load build definition from Dockerfile                                                                                                                       0.0s
 => => transferring dockerfile: 240B                                                                                                                                       0.0s
 => [internal] load metadata for public.ecr.aws/debian/debian:11-slim@sha256:1fec6f6a4f281dc401bf40553e9ddc5524714020be06c865d42870c549662dcd                              1.2s
 => [internal] load .dockerignore                                                                                                                                          0.0s
 => => transferring context: 201B                                                                                                                                          0.0s
 => [internal] load build context                                                                                                                                          0.0s
 => => transferring context: 30B                                                                                                                                           0.0s
 => [1/4] FROM public.ecr.aws/debian/debian:11-slim@sha256:1fec6f6a4f281dc401bf40553e9ddc5524714020be06c865d42870c549662dcd                                                0.0s
 => => resolve public.ecr.aws/debian/debian:11-slim@sha256:1fec6f6a4f281dc401bf40553e9ddc5524714020be06c865d42870c549662dcd                                                0.0s
 => CACHED [2/4] WORKDIR /var/www                                                                                                                                          0.0s
 => ERROR [4/4] RUN echo "Hello world"                                                                                                                                     0.0s
 => CACHED [3/4] COPY --link readme.md .                                                                                                                                   0.0s
------
 > [4/4] RUN echo "Hello world":
0.030 runc run failed: unable to start container process: error during container init: mkdir /var/www: not a directory
------
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
Dockerfile:8
--------------------
   6 |     
   7 |     COPY --link readme.md .
   8 | >>> RUN echo "Hello world"
   9 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c echo \"Hello world\"" did not complete successfully: exit code: 1

View build details: docker-desktop://dashboard/build/ko/ko0/t1r68fqzqgnzp9y4v5xcpmyis

❯ docker buildx create --name ok --driver=docker-container --driver-opt image=moby/buildkit:v0.13.2
ok
❯ docker buildx build --builder=ok .                                                               
[+] Building 6.3s (11/11) FINISHED                                                                                                                          docker-container:ok
 => [internal] booting buildkit                                                                                                                                            2.3s
 => => pulling image moby/buildkit:v0.13.2                                                                                                                                 1.6s
 => => creating container buildx_buildkit_ok0                                                                                                                              0.7s
 => [internal] load build definition from Dockerfile                                                                                                                       0.0s
 => => transferring dockerfile: 240B                                                                                                                                       0.0s
 => [internal] load metadata for public.ecr.aws/debian/debian:11-slim@sha256:1fec6f6a4f281dc401bf40553e9ddc5524714020be06c865d42870c549662dcd                              1.5s
 => [internal] load .dockerignore                                                                                                                                          0.0s
 => => transferring context: 2B                                                                                                                                            0.0s
 => [internal] load build context                                                                                                                                          0.0s
 => => transferring context: 139B                                                                                                                                          0.0s
 => [1/4] FROM public.ecr.aws/debian/debian:11-slim@sha256:1fec6f6a4f281dc401bf40553e9ddc5524714020be06c865d42870c549662dcd                                                1.9s
 => => resolve public.ecr.aws/debian/debian:11-slim@sha256:1fec6f6a4f281dc401bf40553e9ddc5524714020be06c865d42870c549662dcd                                                0.0s
 => => sha256:c982e8b303792a129014d0d049810d7251866836c1f20cd11d8dc5d6690f035a 28.74MB / 28.74MB                                                                           1.5s
 => => extracting sha256:c982e8b303792a129014d0d049810d7251866836c1f20cd11d8dc5d6690f035a                                                                                  0.4s
 => [2/4] WORKDIR /var/www                                                                                                                                                 0.2s
 => [3/4] COPY --link readme.md .                                                                                                                                          0.0s
 => => merging                                                                                                                                                             0.0s
 => [4/4] RUN echo "Hello world"                                                                                                                                           0.0s
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load

View build details: docker-desktop://dashboard/build/ok/ok0/k9dy6ewmfufpkwwbqx0otnozv

What's next:
    View a summary of image vulnerabilities and recommendations → docker scout quickview 

When doing the copy without --link, build is OK.

❯ docker buildx build --builder=ko --no-cache . 
[+] Building 0.1s (8/8) FINISHED                                                                                                                            docker-container:ko
 => [internal] load build definition from Dockerfile                                                                                                                       0.0s
 => => transferring dockerfile: 233B                                                                                                                                       0.0s
 => [internal] load metadata for public.ecr.aws/debian/debian:11-slim@sha256:1fec6f6a4f281dc401bf40553e9ddc5524714020be06c865d42870c549662dcd                              0.0s
 => [internal] load .dockerignore                                                                                                                                          0.0s
 => => transferring context: 2B                                                                                                                                            0.0s
 => [1/4] FROM public.ecr.aws/debian/debian:11-slim@sha256:1fec6f6a4f281dc401bf40553e9ddc5524714020be06c865d42870c549662dcd                                                0.0s
 => => resolve public.ecr.aws/debian/debian:11-slim@sha256:1fec6f6a4f281dc401bf40553e9ddc5524714020be06c865d42870c549662dcd                                                0.0s
 => [internal] load build context                                                                                                                                          0.0s
 => => transferring context: 30B                                                                                                                                           0.0s
 => CACHED [2/4] WORKDIR /var/www                                                                                                                                          0.0s
 => [3/4] COPY readme.md .                                                                                                                                                 0.0s
 => [4/4] RUN echo "Hello world"                                                                                                                                           0.0s
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load

View build details: docker-desktop://dashboard/build/ko/ko0/mcs9oact4tuj55bq7i22cp1oj

What's next:
    View a summary of image vulnerabilities and recommendations → docker scout quickview 

[ufou]

Yeah, we ran into this same issue, I wonder if this new buildkit image is more strict on COPY command syntax? It seems a potential fix is to ensure that the COPY command ends in a trailing slash

[tonistiigi]

Indeed, changing to COPY --link readme ./ seems to be the workaround.

018155fa68e3ab267588f4f1735532bcfb7f02aa is the first bad commit
commit 018155fa68e3ab267588f4f1735532bcfb7f02aa
Author: Anthony Nandaa <profnandaa@gmail.com>
Date:   Fri Apr 5 14:32:58 2024 +0300

    fix: use unix path separator since path already normalized

    In the case for Windows, this line at
    frontend/dockerfile/dockerfile2llb/convert.go#L1142
    ```go
    dest += string(filepath.Separator)

was adding the `\\` to a path that is already normalized
to unix-format, hence ending up with dest paths like
`/\\` for `C:\\` and `/test\\` for `C:\\test\\`.

the src paths are well normalized too at ~L1290.

This change removes the block of code and instead
does the "/" appending using the keepSlash logic
that is in system.NormalizePath called in
pathRelativeToWorkingDir() function before.

fixes #4696

Signed-off-by: Anthony Nandaa <profnandaa@gmail.com>

frontend/dockerfile/dockerfile2llb/convert.go | 8 ++------ solver/llbsolver/file/backend.go | 2 +- 2 files changed, 3 insertions(+), 7 deletions(-)

https://github.com/moby/buildkit/pull/4825

@profnandaa @gabriel-samfira 

[profnandaa]

@tonistiigi -- will take a look. it should be more forgiving for such a case.

[profnandaa]

Fix raised, but I've also opened an issue to add a few test cases on our integration tests to cover these path scenarios.