docker-compose up fails with symlinked Dockerfile

davidhewitt commented 4 years ago

Description of the issue

I have several microservices which share an identical Dockerfile. To simplify maintenance and reduce duplication I would like to replace all the Dockerfiles with a symlink to a single "common" Dockerfile.

Before symlinking, my directory structure looks like this:

- docker-compose.yaml
- microservice_a/
  - Dockerfile
- microservice_b/
  - Dockerfile

after, it looks like this:

- docker-compose.yaml
- common/
  - Dockerfile
- microservice_a/
  - Dockerfile  ->  ../common/Dockerfile
- microservice_b/
  - Dockerfile  ->  ../common/Dockerfile

Without symlinking, docker-compose up works as expected.

Once I switch to the symlink model, this no longer works, and I get errors related to Dockerfile: no such file or directory. See below for more detail:

Context information (for bug reports)

Output of docker-compose version

docker-compose version 1.25.4, build 8d51620a

Output of docker version

Docker version 19.03.8, build afacb8b

(ommited docker-compose config because it contains confidential information like private keys which I'm configuring using env vars)

Steps to reproduce the issue

Create two services with idential dockerfiles and reference them both in a compose file.
Move their dockerfile to a subdirectory 'common' and symlink it in the services' Dockerfiles.

Observed result

docker-compose can no longer find the Dockerfile, even though it's still in "the same place" because of the symlink.

Expected result

docker-compose continues to function, and can read the Dockerfile through the new symlink.

Stacktrace / full error message

[+] Building 0.0s (2/2) FINISHED
 => [internal] load .dockerignore                                                                                                  0.0s
 => => transferring context: 34B                                                                                                   0.0s
 => [internal] load build definition from Dockerfile                                                                               0.0s
 => => transferring dockerfile: 70B                                                                                                0.0s
failed to solve with frontend dockerfile.v0: failed to read dockerfile: open /var/lib/docker/tmp/buildkit-mount384971895/common/Dockerfile: no such file or directory
Traceback (most recent call last):
  File "docker-compose", line 6, in <module>
  File "compose/cli/main.py", line 72, in main
  File "compose/cli/main.py", line 128, in perform_command
  File "compose/cli/main.py", line 1077, in up
  File "compose/cli/main.py", line 1073, in up
  File "compose/project.py", line 548, in up
  File "compose/service.py", line 351, in ensure_image_exists
  File "compose/service.py", line 1110, in build
  File "compose/progress_stream.py", line 25, in stream_output
  File "compose/utils.py", line 61, in split_buffer
  File "compose/utils.py", line 37, in stream_as_text
  File "compose/service.py", line 1818, in build
FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/7_/wk5b0_312sq4_9x59jhq0q_r0000gn/T/tmp0jhuoqwn'
[20348] Failed to execute script docker-compose

Additional information

MacOS Catalina 10.15.4

Code0x58 commented 4 years ago

From some investigation as an interested party:

This log was generated while _CLIBuilder is in use, which happens when COMPOSE_DOCKER_CLI_BUILD=1 which you may do if you want want to use Buildkit (so you'd have DOCKER_BUILDKIT=1 set too). If that is not set, you get a docker.api.client.APIClient instance instead - both share the same interface, so this issue would occur regardless of that config.

The issue could be fixed with a fairly noninvasive change by always providing the Dockerfile using the fileobj argument of the low level build interface. This can be done by making the abstracted build method pass a fileobj instead, and updating _CLIBuilder to pass that through stdin [docker CLI docs]. An optimisation that may be tempting is to only pass via fileobj iff the Dockerfile is a symlink, but I don't think that should be done at all.

thaJeztah commented 4 years ago

Trying to reproduce the problem, but first checking the behavior with docker build (taking docker-compose out of the equation)

Create directories, create Dockerfile, and create symlink:

mkdir repro-7397 && cd repro-7397
mkdir common microservice_a

cat > common/Dockerfile <<EOF
FROM nginx:alpine
RUN echo hello
EOF

cd microservice_a && ln -s ../common/Dockerfile Dockerfile && cd ../

Result should look like:

tree
.
├── common
│   └── Dockerfile
└── microservice_a
    └── Dockerfile -> ../common/Dockerfile

Performing a build in some combinations:

DOCKER_BUILDKIT=1 docker build -t repro-7397 microservice_a/
[+] Building 0.0s (2/2) FINISHED
=> [internal] load build definition from Dockerfile                                                                                                                                   0.0s
=> => transferring dockerfile: 60B                                                                                                                                                    0.0s
=> [internal] load .dockerignore                                                                                                                                                      0.0s
=> => transferring context: 2B                                                                                                                                                        0.0s
failed to solve with frontend dockerfile.v0: failed to read dockerfile: open /var/lib/docker/tmp/buildkit-mount184854581/common/Dockerfile: no such file or directory

DOCKER_BUILDKIT=0 docker build -t repro-7397 microservice_a/
unable to open Dockerfile: open : no such file or directory

DOCKER_BUILDKIT=1 docker build -t repro-7397 -f microservice_a/Dockerfile microservice_a/
[+] Building 0.0s (2/2) FINISHED
 => [internal] load .dockerignore                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                        0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                   0.0s
 => => transferring dockerfile: 60B                                                                                                                                                    0.0s
failed to solve with frontend dockerfile.v0: failed to read dockerfile: open /var/lib/docker/tmp/buildkit-mount620518338/common/Dockerfile: no such file or directory

DOCKER_BUILDKIT=0 docker build -t repro-7397 -f microservice_a/Dockerfile microservice_a/
Sending build context to Docker daemon  2.095kB
Step 1/2 : FROM nginx:alpine
 ---> 377c0837328f
Step 2/2 : RUN echo hello
 ---> Using cache
 ---> 490312e05310
Successfully built 490312e05310
Successfully tagged repro-7397:latest

So looking at those combinations:

:x: DOCKER_BUILDKIT=0 and only build-context specified
:x: DOCKER_BUILDKIT=1 and only build-context specified
:white_check_mark: DOCKER_BUILDKIT=0 and both build-context and -f (path to Dockerfile)
:x: DOCKER_BUILDKIT=1 and both build-context and -f (path to Dockerfile)

Testing the same with docker-compose:

cat > docker-compose.yml <<EOF
version: "3.7"
services:
  test:
    build:
      context: microservice_a
      dockerfile: Dockerfile
EOF

:white_check_mark: DOCKER_BUILDKIT=0 COMPOSE_DOCKER_CLI_BUILD=1 docker-compose build
:x: DOCKER_BUILDKIT=1 COMPOSE_DOCKER_CLI_BUILD=1 docker-compose build

Results there look the same; it looks like there's a difference in behavior between buildkit and non-buildkit; I think that should be fixed in the docker cli (or buildkit)

thaJeztah commented 4 years ago

ping @tonistiigi @tiborvass PTAL ^

thaJeztah commented 4 years ago

From a conversation with @tonistiigi on Slack;

this is similar to https://github.com/docker/buildx/issues/1781 . same “fix” and workaround buildkit shares always need to have root path and symlinks are not allowed to escape out of it so if we don’t want this to be true for dockerfile and think that dockerfile can be anywhere in the system we need to make a copy and share that instead.

Regarding this;

buildkit shares always need to have root path and symlinks are not allowed to escape out of it

I think it makes sense to limit scope if only a build-context is passed (as in: if a build-context is passed, scope the build to that context). However for the "root" of the build-context, we allow that root itself to be a symlink; this is what's currently supported:

resolve the root of the build-context
- if the specified path is a symlink, follow that symlink, and use its target as the build-context
start the build using the resolved path

One use-case for this is, if (for example) one maintains a directory linking to projects stored elsewhere (the "real" project directory may be a path somewhere nested deeply);

mkdir repro-7397 && cd repro-7397
mkdir -p go/src/github.com/bar/project-a myprojects

cat > go/src/github.com/bar/project-a/Dockerfile <<EOF
FROM nginx:alpine
RUN echo hello project-a
EOF

cd myprojects \
&& ln -s ../go/src/github.com/bar/project-a project-a \
&& cd ..

Directory now looks like;

tree
.
├── go
│   └── src
│       └── github.com
│           └── bar
│               └── project-a
│                   └── Dockerfile
└── myprojects
    └── project-a -> ../go/src/github.com/bar/project-a

With the above, these work:

DOCKER_BUILDKIT=0 docker build -t project-a myprojects/project-a
DOCKER_BUILDKIT=1 docker build -t project-a myprojects/project-a

However, escaping out of the specified build-context should not work, so in the example below, "project-b"'s Dockerfile is a symlink to "project-a"'s Dockerfile:

mkdir -p go/src/github.com/bar/project-b-symlinked-dockerfile \
&& cd go/src/github.com/bar/project-b-symlinked-dockerfile \
&& ln -s ../project-a/Dockerfile Dockerfile \
&& cd ../../../../../myprojects/ \
&& ln -s ../go/src/github.com/bar/project-b-symlinked-dockerfile/ project-b \
&& cd ..

tree
.
├── go
│   └── src
│       └── github.com
│           └── bar
│               ├── project-a
│               │   └── Dockerfile
│               └── project-b-symlinked-dockerfile
│                   └── Dockerfile -> ../project-a/Dockerfile
└── myprojects
    ├── project-a -> ../go/src/github.com/bar/project-a
    └── project-b -> ../go/src/github.com/bar/project-b-symlinked-dockerfile/

This currently fails (as expected), because although "project-b"'s directory was found, the Dockerfile is outside of its context;

DOCKER_BUILDKIT=0 docker build -t project-b myprojects/project-b
unable to open Dockerfile: open : no such file or directory

DOCKER_BUILDKIT=1 docker build -t project-b myprojects/project-b
[+] Building 0.0s (2/2) FINISHED
 => [internal] load build definition from Dockerfile      0.0s
 => => transferring dockerfile: 63B                       0.0s
 => [internal] load .dockerignore                         0.0s
 => => transferring context: 2B                           0.0s
failed to solve with frontend dockerfile.v0: failed to read dockerfile: open /var/lib/docker/tmp/buildkit-mount011317083/project-a/Dockerfile: no such file or directory

I think the situation when explicitly providing the path to the Dockerfile should work, and follow the same rules;

resolve the path of the specified Dockerfile
- if the specified path is a symlink, follow that symlink, and use its target as the path for the Dockerfile
start the build using resolved Dockerfile

This currently works without buildkit (in this example, both the specified context and Dockerfile are resolved through symlinks before starting the build);

DOCKER_BUILDKIT=0 docker build -t project-b  -f myprojects/project-b/Dockerfile myprojects/project-b

But fails with BuildKit;

DOCKER_BUILDKIT=1 docker build -t project-b  -f myprojects/project-b/Dockerfile myprojects/project-b

One issue would not be "resolved" (and would have to be discussed, probably should not be supported);

BuildKit supports having a per-Dockerfile .dockerignore (Dockerfile.dockerignore), so if theDockerfile` is resolved through a symlink, should;

the Dockerfile.dockerignore also be looked for in the resolved directory of the Dockerfile? Doing so would break the "sandbox" where no content outside of the context should be read
should the Dockerfile.dockerignore be looked for in the "build-context"? (resolved path of myprojects/project-b in this example)?

@tonistiigi WDYT?

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

thaJeztah commented 3 years ago

.

stale[bot] commented 3 years ago

This issue has been automatically closed because it had not recent activity during the stale period.

thaJeztah commented 3 years ago

Yes, it did have activity, lol. Let me reopen

stale[bot] commented 3 years ago

This issue has been automatically marked as not stale anymore due to the recent activity.

stale[bot] commented 3 years ago

This issue has been automatically marked as not stale anymore due to the recent activity.

fsaintjacques commented 3 years ago

I'm facing this issue.

vviikk commented 3 years ago

How simple is this to fix? It should just work.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] commented 2 years ago

This issue has been automatically closed because it had not recent activity during the stale period.

fredericoschardong commented 2 years ago

same issue

simon-liebehenschel commented 1 year ago

@laurazard @thaJeztah Can someone just disable this useless bot? There are dozens of high-voted issues that were closed automatically by the bot https://github.com/docker/compose/issues?q=is%3Aissue+label%3Astale+is%3Aclosed+sort%3Areactions-%2B1-desc

JekRock commented 1 year ago

I have the same issue but with symlinks to .env files inside a docker-compose.yml file. If env_file points to a file, everything works. If it points to a symlink, docker-compose can't find the file with no such file or directory

docker / compose