Windows: missing secrets file creates file-named directory

philliphoff commented 3 years ago

Description of the issue

On Windows, when starting a composition that refers to a secret file which does not exist, an "invalid mount config for type" error is generated and a directory with the same name as the file is created.

This behavior is not seen on Mac, with the same version of Docker Desktop and Docker Compose.

This may be related to #5377 but seems only to affect Windows.

Context information (for bug reports)

Output of docker-compose version

docker-compose version 1.29.0, build 07737305
docker-py version: 5.0.0
CPython version: 3.9.0
OpenSSL version: OpenSSL 1.1.1g  21 Apr 2020

Output of docker version

Client: Docker Engine - Community
 Cloud integration: 1.0.12
 Version:           20.10.5
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        55c4c88
 Built:             Tue Mar  2 20:14:53 2021
 OS/Arch:           windows/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.5
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       363e9a8
  Built:            Tue Mar  2 20:15:47 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.4
  GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc:
  Version:          1.0.0-rc93
  GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker-compose config (Make sure to add the relevant -f and other flags)

D:\Repos\Scratch\compose-test>docker-compose config
secrets:
  postgres_db:
    file: D:\Repos\Scratch\compose-test\postgres_db.txt
  postgres_password:
    file: D:\Repos\Scratch\compose-test\postgres_password.txt
  postgres_user:
    file: D:\Repos\Scratch\compose-test\postgres_user.txt
services:
  db:
    environment:
      POSTGRES_DB_FILE: /run/secrets/postgres_db
      POSTGRES_INITDB_ARGS: --auth=md5
      POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
      POSTGRES_USER_FILE: /run/secrets/postgres_user
    healthcheck:
      interval: 10s
      retries: 5
      test:
      - CMD-SHELL
      - pg_isready -U postgres
      timeout: 5s
    image: postgres:latest
    labels:
      com.microsoft.vscode.dev-container-name: webapi-nodejs-dev-container
    restart: unless-stopped
    secrets:
    - source: postgres_db
    - source: postgres_password
    - source: postgres_user
    volumes:
    - postgres-data:/var/lib/postgresql/data:rw
version: '3.7'
volumes:
  postgres-data: {}

Steps to reproduce the issue

Add the following to docker-compose.yml in an otherwise empty directory:

version: '3.7'

secrets:
  postgres_db:
    file: postgres_db.txt
  postgres_password:
    file: postgres_password.txt
  postgres_user:
    file: postgres_user.txt

services:
  db:
    image: postgres:latest
    restart: unless-stopped
    volumes:
      - postgres-data:/var/lib/postgresql/data
    environment:
      - POSTGRES_DB_FILE=/run/secrets/postgres_db
      - POSTGRES_INITDB_ARGS=--auth=md5
      - POSTGRES_PASSWORD_FILE=/run/secrets/postgres_password
      - POSTGRES_USER_FILE=/run/secrets/postgres_user
    labels:
      com.microsoft.vscode.dev-container-name: webapi-nodejs-dev-container
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 5
    secrets:
      - postgres_db
      - postgres_password
      - postgres_user

    # Add "forwardPorts": ["5432"] to **devcontainer.json** to forward Postgress locally.
    # (Adding the "ports" property to this file will not forward from a Codespace.)

volumes:
  postgres-data:

docker-compose up --remove-orphans

Observed result

Composition fails to start (which is expected).
Folders are created in the directory with the same names as the declared secrets file (not expected).

NOTE: This does not repro on every attempt. What I observe is that it works on the first attempt, but not (always) subsequent attempts. However, if you tweak the names of the secrets files, then you can (usually) repro it on the next attempt.

Expected result

Composition fails to start.
No directories (or files) created.

Stacktrace / full error message

D:\Repos\Scratch\compose-test>docker-compose up --remove-orphans
WARNING: Service "db" uses an undefined secret file "D:\Repos\Scratch\compose-test\postgres_db.txt", the following file should be created "D:\Repos\Scratch\compose-test\postgres_db.txt"
WARNING: Service "db" uses an undefined secret file "D:\Repos\Scratch\compose-test\postgres_password.txt", the 
following file should be created "D:\Repos\Scratch\compose-test\postgres_password.txt"
WARNING: Service "db" uses an undefined secret file "D:\Repos\Scratch\compose-test\postgres_user.txt", the following file should be created "D:\Repos\Scratch\compose-test\postgres_user.txt"
Docker Compose is now in the Docker CLI, try `docker compose up`

Removing orphan container "compose-test_app_1"
Starting compose-test_db_1 ... error

ERROR: for compose-test_db_1  Cannot start service db: invalid mount config for type "bind": bind source path does not exist: /run/desktop/mnt/host/d/Repos/Scratch/compose-test/postgres_db.txt

ERROR: for db  Cannot start service db: invalid mount config for type "bind": bind source path does not exist: 
/run/desktop/mnt/host/d/Repos/Scratch/compose-test/postgres_db.txt
ERROR: Encountered errors while bringing up the project.

Additional information

OS version / distribution, docker-compose install method, etc.

Windows 10 (Version 21H1, OS Build 19043.964)

pmorch commented 3 years ago

I'm not 100% sure, but I think this issue has gotten worse recently on Docker Deskop for Windows.

If the file: ./mypass.txt is missing, it creates a directory in the current directory called mypass.txt. It has been like that "for a long time".

(I'm pretty sure) it used to be that I could then:

docker-compose down
rmdir mypass.txt
echo contents > mypass.txt
docker-compose up -d

and that would be the end of it.

But now when I do that last step, I get:

WSL2@~/idea/app» docker-compose up -d
Creating network "app_default" with the default driver
Creating app_app_1 ... error

ERROR: for app_app_1  Cannot create container for service app: not a directory

ERROR: for app  Cannot create container for service app: not a directory
ERROR: Encountered errors while bringing up the project.

To get rid of that error, I now have to:

WSL2@~/idea/app» sudo rm -r /mnt/wsl/docker-desktop-bind-mounts/Ubuntu-20.04-v2/e47c2d04a4b94e64079cbd50e60e132e8e5670dbd7d9fc84721344f3f2b4e105

and then it works:

WSL2@~/idea/app» docker-compose up -d
Creating network "app_default" with the default driver
Creating app_app_1 ... done

I have no idea how to identify the correct e47c2d04a4b94e64079cbd50e60e132e8e5670dbd7d9fc84721344f3f2b4e105 file under /mnt/wsl/docker-desktop-bind-mounts/Ubuntu-20.04-v2 but the timestamp may help and I found that quitting Docker Desktop for Windows and then:

WSL2@~/idea/app» sudo rm -r /mnt/wsl/docker-desktop-bind-mounts/Ubuntu-20.04-v2/*

And then restarting Docker Desktop works too.

Note: I was worried that this sudo rm -r .../* might loose data somehow, but I didn't have any data I didn't mind loosing. You might. Be careful and investigate what you're deleting... Even though I didn't mind loosing them, I checked and all my volume contents were still intact afterwards

WSL2@~/idea/app» docker-compose version
docker-compose version 1.29.1, build c34c88b2
docker-py version: 5.0.0
CPython version: 3.7.10
OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019

spwoodcock commented 2 years ago

For anyone reading the comment above - be VERY careful following the advice to delete some directories under /mnt/wsl/docker-desktop-bind-mounts/

It has the potential to not only delete Docker volumes, but delete your entire WSL filesystem. I just found out the hard way.

I'm not sure if this is to do with a recent update to Docker Desktop and some sort of symlinking, but I really hope this secrets mounting issue can be fixed at some point.

delyada commented 2 years ago

It's not just about missing secrets file. My secrets files are there but I get the error.

I'm in a WSL 2 VS Code dev container.

If I use docker-from-docker (i.e. the Windows host's Docker) to up my composition, the error happens.
If I use docker-in-docker (i.e. the Linux container's internal installation of Docker) to up my composition, no such error happens.

JustEnoughDucks commented 2 years ago

Interestingly, this happens to me on my brand new debian baremetal docker install.

On my old debian server which has docker at the exact same version of every component, but was upgraded starting from an early 2020 docker version does not have this issue. The difference was the parent folder permissions.

The difference between them for me was that the secrets directory (not the files themselves) needs group execute permissions. I don't know how applicable it is for windows, but it seems to be a permissions issue?

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

MicahZoltu commented 2 years ago

Still a problem, current behavior:

version: '3.8'

secrets:
  my_secret:
    file: my_secret.txt

services:
  hello-world:
    image: hello-world
    secrets:
      - my_secret

> docker-compose up
WARNING: Service "hello-world" uses an undefined secret file "C:\Users\micah\Source\temp\my_secret.txt", the following file should be created "C:\Users\micah\Source\temp\my_secret.txt"
...

my_secret.txt directory is created
The compose executes, despite the missing secret file (it should fail).

If you delete the folder and run docker-compose up again, you now get a failure:

  > docker-compose up
  WARNING: Service "hello-world" uses an undefined secret file "C:\...\my_secret.txt", the following file should be created "C:\...\my_secret.txt"
  Starting temp_hello-world_1 ... error

  ERROR: for temp_hello-world_1  Cannot start service hello-world: invalid mount config for type "bind": bind source path does not exist: /run/desktop/mnt/host/c/.../my_secret.txt

Deleting the folder and then creating a text file in its place results in the following failure:

  > docker-compose up
  Starting temp_hello-world_1 ... error

  ERROR: for temp_hello-world_1  Cannot start service hello-world: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/run/desktop/mnt/host/c/.../my_secret.txt" to rootfs at "/run/secrets/my_secret" caused: mount through procfd: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

Restarting docker reverts behavior back to initial behavior.
Behavior is the same when run from a WSL terminal with docker client installed (connected up to Docker for Windows).

MicahZoltu commented 2 years ago

@stale Not stale.

MicahZoltu commented 2 years ago

The above replicated on: Docker Desktop: 4.4.4 (73704 Docker Engine: 20.10.12 Docker Compose: 1.29.2

glours commented 2 years ago

FYI Docker Compose V1, the python version, reached end-of-life, please use Compose v2 instead and let us know if you still have the problem with this version. You can simply switch to Compose V2 by checking the checkbox in Docker Desktop preferences

MicahZoltu commented 2 years ago

Tested with Docker Compose v2 per above instructions: New behavior (still broken):

I get no warning at all, and a folder is created (as before).

If I delete the folder after the first run then docker-compose up again I get the following error:

> docker-compose up
[+] Running 1/0
- Container temp-hello-world-1  Created
Attaching to temp-hello-world-1
Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /run/desktop/mnt/host/c/.../my_secret.txt

Restarting Docker no longer reverts behavior back to initial behavior, it now seems to get permanently stuck on the second-run behavior.

Creating the secret after second docker-compose up attempt results in this error:

> docker-compose up
[+] Running 1/0
- Container temp-hello-world-1  Created
Attaching to temp-hello-world-1
Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/run/desktop/mnt/host/c/.../my_secret.txt" to rootfs at "/run/secrets/my_secret" caused: mount through procfd: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

stale[bot] commented 2 years ago

This issue has been automatically marked as not stale anymore due to the recent activity.

stale[bot] commented 2 years ago

This issue has been automatically marked as not stale anymore due to the recent activity.

digitalkelvin commented 1 year ago

Any progress? I have same issue as @MicahZoltu

lucasfelber commented 7 months ago

I also have this issue and get the same result now with v3

LordLumineer commented 4 months ago

It appears to be fixed

Progress1 commented 4 months ago

It looks not :-) (Latest Docker Desktop 4.30.0, latest updated Windows)

if secrets are missing first time, it creates directory with secret names
then i delete manually directories and create secrets files

after using docker-compose up i get error:


Error response from daemon: failed to create task for container: failed to create shim task: 
OCI runtime create failed: runc create failed: unable to start container process: error during container init: 
error mounting "/run/desktop/mnt/host/c/_/Dev/XXX/docker/secrets/postgres_password.txt" to rootfs at "/run/secrets/postgres_password": 
mount /run/desktop/mnt/host/c/_/Dev/XXX/docker/secrets/postgres_password.txt:/run/secrets/postgres_password (via /proc/self/fd/6), 
flags: 0x5000: not a directory: unknown: 
Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

file `c:\_\Dev\XXX\docker\secrets\postgres_password.txt`  exists.

docker-compose.yml:

services: database: ... secrets:

postgres_password secrets: postgres_password: file: ./secrets/postgres_password.txt


Any solution to get secrets working?

**UPDATE/FIX:** i comment secrets definition, recreate container (this probably delete old mounts), un-comment secrets, recreate container again (now are mounts correctly mapped) and it start without error

dakkafex commented 3 weeks ago

The only way i have gotten it to work sofar is by running docker-compose up --build by going into the WSL terminal first.

docker / compose