[BUG] Secret mounts in `/run/secrets` throws an error: Could not find the file / in container <container_id>

shantanoo-desai commented 1 year ago

Description

This error started just recently, especially when updating to latest Docker Engine / Docker Compose versions.

When a service is mentioned with user: "1000" the container that have explicit users defined to them e.g. grafana in their images fail to mount the secrets in compose file to the respective /run/secrets directory in the container on boot.

Previously, it was known that a getent was performed on the host to match whether the user ID matches and a container is spun up in order to make /run/secrets readable by a the container's user (if it not root).

Steps To Reproduce

Compose file

services:
  grafana:
    image: docker.io/grafana/grafana-oss:9.5.1
    container_name: komponist_grafana
    environment:
      - GF_SECURITY_ADMIN_USER__FILE=/run/secrets/grafana_admin_username
      - GF_SECURITY_ADMIN_PASSWORD__FILE=/run/secrets/grafana_admin_password
      - GF_SERVER_ROOT_URL=/grafana
      - GF_SERVER_SERVE_FROM_SUB_PATH=true
    secrets:
      - grafana_admin_username
      - grafana_admin_password
    logging:
      options:
        max-size: "5m"
    ports:
     - "3000:3000"
    security_opt:
      - "no-new-privileges:true"
    user: "1000"
    volumes:
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro

secrets:
    grafana_admin_username:
      environment: GRAFANA_ADMIN_USERNAME
    grafana_admin_password:
      environment: GRAFANA_ADMIN_PASSWORD

Environment Variables

.env file

GRAFANA_ADMIN_USERNAME=admin
GRAFANA_ADMIN_PASSWORD=testGrafana

Steps

Upon docker compose up the following error occurs:

Container komponist_grafana  Creating                                                                           
Error response from daemon: Could not find the file / in container 
9714fd659bd2eb795855f9fa292d7e76f3a06fdd40a16dfd47e5c53f759758a9

Upon forcing an up again using docker compose up the following logs show up:

komponist_grafana  | Getting secret GF_SECURITY_ADMIN_PASSWORD from 
/run/secrets/grafana_admin_password
komponist_grafana  | /run.sh: line 59: /run/secrets/grafana_admin_password: No such file or directory
komponist_grafana exited with code 1

Upon removing the user value from the Compose file, error still persists

Compose Version

v2.18.1

Docker Environment

Client: Docker Engine - Community
 Version:    24.0.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.5
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.18.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
 Containers: 3
  Running: 0
  Paused: 0
  Stopped: 3
 Images: 229
 Server Version: 24.0.2
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 5.15.90.1-microsoft-standard-WSL2
 Operating System: Ubuntu 20.04.6 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 15.44GiB

Anything else?

Work-around / Solution

The only way to get the container up is to figure out from the container what the user is using an whoami / id and place this in the user of the compose file.

Working `docker-compose.yml`

services:
  grafana:
    image: docker.io/grafana/grafana-oss:9.5.1
    container_name: komponist_grafana
    environment:
      - GF_SECURITY_ADMIN_USER__FILE=/run/secrets/grafana_admin_username
      - GF_SECURITY_ADMIN_PASSWORD__FILE=/run/secrets/grafana_admin_password
      - GF_SERVER_ROOT_URL=/grafana
      - GF_SERVER_SERVE_FROM_SUB_PATH=true
    secrets:
      - grafana_admin_username
      - grafana_admin_password
    logging:
      options:
        max-size: "5m"
    ports:
     - "3000:3000"
    security_opt:
      - "no-new-privileges:true"
    user: grafana
    volumes:
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro

secrets:
    grafana_admin_username:
      environment: GRAFANA_ADMIN_USERNAME
    grafana_admin_password:
      environment: GRAFANA_ADMIN_PASSWORD

I am not sure if this is a Docker Compose Bug or a Docker Engine thing from the start of Docker Engine v24.x.x.

NOTE: This problem does not persist when the container's user is root

Happy to help reproduce any other examples with similar logic.

ndeloof commented 1 year ago

Tried to reproduce but don't get the same error:

komponist_grafana  | Getting secret GF_SECURITY_ADMIN_PASSWORD from /run/secrets/grafana_admin_password
komponist_grafana  | /run.sh: line 59: /run/secrets/grafana_admin_password: Permission denied

Which I expected: secret is injected under /run/secrets/grafana_admin_password as owner root until secret is declared with explicit uid/gid. To make this readable by non-root user you need to set uid: 472 but then you indeed will be blocked by the getenv bug.

shantanoo-desai commented 1 year ago

@ndeloof 😮 this is rather interesting that you get a permission denied error. I used to have user: "1000" previously in order to mitigate this error ( Docker v23.x ) but from Docker v24.x I don't get the permission error.

Would you happen to know if this is because of certain settings in the Docker Daemon that you have?

I am okay with the permission error, but the secrets not being created in the first place is what was not expected

ndeloof commented 1 year ago

the secrets not being created in the first place is what was not expected

sure, but this one I can't reproduce. Could not find the file / in container is weird, how could a container not have a filesystem root? I guess the actual error is hidden. Could you check engine logs maybe?

shantanoo-desai commented 1 year ago

Steps

using the compose file in the issue with user: "1000"

Logs

Here are the logs from the Docker Daemon via

journalctl -xu docker.service | tail -f

Jun 07 11:51:07 2KLD7G3 dockerd[1787]: time="2023-06-07T11:51:07.656923225+02:00" level=debug msg="form data: {\"AttachStderr\":true,\"AttachStdin\":false,\"AttachStdout\":true,\"Cmd\":null,\"Domainname\":\"\",\"Entrypoint\":null,\"Env\":[\"GF_SERVER_ROOT_URL=/grafana\",\"GF_SERVER_SERVE_FROM_SUB_PATH=true\",\"GF_SECURITY_ADMIN_USER__FILE=/run/secrets/grafana_admin_username\",\"GF_SECURITY_ADMIN_PASSWORD__FILE=/run/secrets/grafana_admin_password\"],\"ExposedPorts\":{\"3000/tcp\":{}},\"HostConfig\":{\"AutoRemove\":false,\"Binds\":[\"/etc/timezone:/etc/timezone:ro\",\"/etc/localtime:/etc/localtime:ro\"],\"BlkioDeviceReadBps\":null,\"BlkioDeviceReadIOps\":null,\"BlkioDeviceWriteBps\":null,\"BlkioDeviceWriteIOps\":null,\"BlkioWeight\":0,\"BlkioWeightDevice\":null,\"CapAdd\":null,\"CapDrop\":null,\"Cgroup\":\"\",\"CgroupParent\":\"\",\"CgroupnsMode\":\"\",\"ConsoleSize\":[0,0],\"ContainerIDFile\":\"\",\"CpuCount\":0,\"CpuPercent\":0,\"CpuPeriod\":0,\"CpuQuota\":0,\"CpuRealtimePeriod\":0,\"CpuRealtimeRuntime\":0,\"CpuShares\":0,\"CpusetCpus\":\"\",\"CpusetMems\":\"\",\"DeviceCgroupRules\":null,\"DeviceRequests\":null,\"Devices\":null,\"Dns\":null,\"DnsOptions\":null,\"DnsSearch\":null,\"ExtraHosts\":[],\"GroupAdd\":null,\"IOMaximumBandwidth\":0,\"IOMaximumIOps\":0,\"IpcMode\":\"\",\"Isolation\":\"\",\"Links\":null,\"LogConfig\":{\"Config\":{\"max-size\":\"5m\"},\"Type\":\"\"},\"MaskedPaths\":null,\"Memory\":0,\"MemoryReservation\":0,\"MemorySwap\":0,\"MemorySwappiness\":null,\"NanoCpus\":0,\"NetworkMode\":\"tmp_default\",\"OomKillDisable\":false,\"OomScoreAdj\":0,\"PidMode\":\"\",\"PidsLimit\":null,\"PortBindings\":{\"3000/tcp\":[{\"HostIp\":\"\",\"HostPort\":\"3000\"}]},\"Privileged\":false,\"PublishAllPorts\":false,\"ReadonlyPaths\":null,\"ReadonlyRootfs\":false,\"RestartPolicy\":{\"MaximumRetryCount\":0,\"Name\":\"\"},\"SecurityOpt\":[\"no-new-privileges:true\"],\"ShmSize\":0,\"UTSMode\":\"\",\"Ulimits\":null,\"UsernsMode\":\"\",\"VolumeDriver\":\"\",\"VolumesFrom\":null},\"Hostname\":\"\",\"Image\":\"docker.io/grafana/grafana-oss:9.5.1\",\"Labels\":{\"com.docker.compose.config-hash\":\"ece9b9cdc59ea88900a3b9495c1d3cfca6940294c0549ea28abcacdaa3a47f99\",\"com.docker.compose.container-number\":\"1\",\"com.docker.compose.depends_on\":\"\",\"com.docker.compose.image\":\"sha256:d365f61c9bc67cef7b043637917063c0929f50e0331a96b6c753f0abae70f9be\",\"com.docker.compose.oneoff\":\"False\",\"com.docker.compose.project\":\"tmp\",\"com.docker.compose.project.config_files\":\"/tmp/docker-compose.yml\",\"com.docker.compose.project.working_dir\":\"/tmp\",\"com.docker.compose.service\":\"grafana\",\"com.docker.compose.version\":\"2.18.1\"},\"NetworkingConfig\":{\"EndpointsConfig\":{\"tmp_default\":{\"Aliases\":[\"grafana\"],\"DriverOpts\":null,\"EndpointID\":\"\",\"Gateway\":\"\",\"GlobalIPv6Address\":\"\",\"GlobalIPv6PrefixLen\":0,\"IPAMConfig\":null,\"IPAddress\":\"\",\"IPPrefixLen\":0,\"IPv6Gateway\":\"\",\"Links\":null,\"MacAddress\":\"\",\"NetworkID\":\"\"}}},\"OnBuild\":null,\"OpenStdin\":false,\"StdinOnce\":false,\"Tty\":false,\"User\":\"1000\",\"Volumes\":null,\"WorkingDir\":\"\"}"
Jun 07 11:51:07 2KLD7G3 dockerd[1787]: time="2023-06-07T11:51:07.657627065+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 17.04, use `=` instead."
Jun 07 11:51:07 2KLD7G3 dockerd[1787]: time="2023-06-07T11:51:07.670237084+02:00" level=debug msg="container mounted via layerStore: /home/shantanoo/docker/overlay2/37893bce20d45d2c8ab16288df50b0863aad4fff62cc1c8fa78ab25b3305c072/merged" container=03c55e6da2e181b570c6631cd641cf8e0800e727179859423c4634d2ec412ed2
Jun 07 11:51:07 2KLD7G3 dockerd[1787]: time="2023-06-07T11:51:07.687593773+02:00" level=debug msg="Calling GET /v1.42/containers/03c55e6da2e181b570c6631cd641cf8e0800e727179859423c4634d2ec412ed2/json"
Jun 07 11:51:07 2KLD7G3 dockerd[1787]: time="2023-06-07T11:51:07.688822443+02:00" level=debug msg="Calling POST /v1.42/networks/tmp_default/disconnect"
Jun 07 11:51:07 2KLD7G3 dockerd[1787]: time="2023-06-07T11:51:07.688890447+02:00" level=debug msg="form data: {\"Container\":\"03c55e6da2e181b570c6631cd641cf8e0800e727179859423c4634d2ec412ed2\",\"Force\":false}"
Jun 07 11:51:07 2KLD7G3 dockerd[1787]: time="2023-06-07T11:51:07.697117616+02:00" level=debug msg="Calling POST /v1.42/networks/tmp_default/connect"
Jun 07 11:51:07 2KLD7G3 dockerd[1787]: time="2023-06-07T11:51:07.697201721+02:00" level=debug msg="form data: {\"Container\":\"03c55e6da2e181b570c6631cd641cf8e0800e727179859423c4634d2ec412ed2\",\"EndpointConfig\":{\"Aliases\":[\"komponist_grafana\",\"grafana\"],\"DriverOpts\":null,\"EndpointID\":\"\",\"Gateway\":\"\",\"GlobalIPv6Address\":\"\",\"GlobalIPv6PrefixLen\":0,\"IPAMConfig\":null,\"IPAddress\":\"\",\"IPPrefixLen\":0,\"IPv6Gateway\":\"\",\"Links\":null,\"MacAddress\":\"\",\"NetworkID\":\"\"}}"
Jun 07 11:51:07 2KLD7G3 dockerd[1787]: time="2023-06-07T11:51:07.705294082+02:00" level=debug msg="Calling PUT /v1.42/containers/03c55e6da2e181b570c6631cd641cf8e0800e727179859423c4634d2ec412ed2/archive?copyUIDGID=true&noOverwriteDirNonDir=true&path=%2F"
Jun 07 11:51:07 2KLD7G3 dockerd[1787]: time="2023-06-07T11:51:07.705673904+02:00" level=debug msg="container mounted via layerStore: /home/shantanoo/docker/overlay2/37893bce20d45d2c8ab16288df50b0863aad4fff62cc1c8fa78ab25b3305c072/merged" container=03c55e6da2e181b570c6631cd641cf8e0800e727179859423c4634d2ec412ed2

Beyond the warning of the security_opts I don't see any errors

shantanoo-desai commented 1 year ago

Maybe this seems like a Moby-related Issue?

ndeloof commented 1 year ago

from Docker v24.x I don't get the permission error

can you please try:

services:
  demo:
    image: alpine
    secrets:
      - foo
    user: "1000"

secrets:
    foo:
      environment: FOO

$ FOO=hello docker compose run demo cat /run/secrets/foo
cat: can't open '/run/secrets/foo': Permission denied

shantanoo-desai commented 1 year ago

No still the same error.

for a docker-compose.test.yml

FOO=hello docker compose -f docker-compose.test.yml run demo cat /run/secrets/foo

provides the logs:

Error response from daemon: Could not find the file / in container 85d8900cd569dde57f81c52349ac47fd33098affdf9fe14bcff8475f8adaa7e2

and the daemon logs

Jun 07 12:20:04 2KLD7G3 dockerd[1787]: time="2023-06-07T12:20:04.167637997+02:00" level=debug msg="Calling POST /v1.42/containers/create?name=tmp-demo-run-f9cd2c95842e"
Jun 07 12:20:04 2KLD7G3 dockerd[1787]: time="2023-06-07T12:20:04.167780598+02:00" level=debug msg="form data: {\"AttachStderr\":true,\"AttachStdin\":true,\"AttachStdout\":true,\"Cmd\":[\"cat\",\"/run/secrets/foo\"],\"Domainname\":\"\",\"Entrypoint\":null,\"Env\":null,\"HostConfig\":{\"AutoRemove\":false,\"Binds\":null,\"BlkioDeviceReadBps\":null,\"BlkioDeviceReadIOps\":null,\"BlkioDeviceWriteBps\":null,\"BlkioDeviceWriteIOps\":null,\"BlkioWeight\":0,\"BlkioWeightDevice\":null,\"CapAdd\":null,\"CapDrop\":null,\"Cgroup\":\"\",\"CgroupParent\":\"\",\"CgroupnsMode\":\"\",\"ConsoleSize\":[0,0],\"ContainerIDFile\":\"\",\"CpuCount\":0,\"CpuPercent\":0,\"CpuPeriod\":0,\"CpuQuota\":0,\"CpuRealtimePeriod\":0,\"CpuRealtimeRuntime\":0,\"CpuShares\":0,\"CpusetCpus\":\"\",\"CpusetMems\":\"\",\"DeviceCgroupRules\":null,\"DeviceRequests\":null,\"Devices\":null,\"Dns\":null,\"DnsOptions\":null,\"DnsSearch\":null,\"ExtraHosts\":[],\"GroupAdd\":null,\"IOMaximumBandwidth\":0,\"IOMaximumIOps\":0,\"IpcMode\":\"\",\"Isolation\":\"\",\"Links\":null,\"LogConfig\":{\"Config\":null,\"Type\":\"\"},\"MaskedPaths\":null,\"Memory\":0,\"MemoryReservation\":0,\"MemorySwap\":0,\"MemorySwappiness\":null,\"NanoCpus\":0,\"NetworkMode\":\"tmp_default\",\"OomKillDisable\":false,\"OomScoreAdj\":0,\"PidMode\":\"\",\"PidsLimit\":null,\"PortBindings\":{},\"Privileged\":false,\"PublishAllPorts\":false,\"ReadonlyPaths\":null,\"ReadonlyRootfs\":false,\"RestartPolicy\":{\"MaximumRetryCount\":0,\"Name\":\"\"},\"SecurityOpt\":null,\"ShmSize\":0,\"UTSMode\":\"\",\"Ulimits\":null,\"UsernsMode\":\"\",\"VolumeDriver\":\"\",\"VolumesFrom\":null},\"Hostname\":\"\",\"Image\":\"alpine\",\"Labels\":{\"com.docker.compose.config-hash\":\"fc3d578fd0799ac40b036f778868a0524778501946b0f7cab4a18c603cdba654\",\"com.docker.compose.container-number\":\"1\",\"com.docker.compose.depends_on\":\"\",\"com.docker.compose.image\":\"sha256:5e2b554c1c45d22c9d1aa836828828e320a26011b76c08631ac896cbc3625e3e\",\"com.docker.compose.oneoff\":\"True\",\"com.docker.compose.project\":\"tmp\",\"com.docker.compose.project.config_files\":\"/tmp/docker-compose.test.yml\",\"com.docker.compose.project.working_dir\":\"/tmp\",\"com.docker.compose.service\":\"demo\",\"com.docker.compose.slug\":\"f9cd2c95842e31b06126ee37096372a0339d68d7483d272713fa773bdb48bdcf\",\"com.docker.compose.version\":\"2.18.1\"},\"NetworkingConfig\":{\"EndpointsConfig\":{\"tmp_default\":{\"Aliases\":[\"demo\"],\"DriverOpts\":null,\"EndpointID\":\"\",\"Gateway\":\"\",\"GlobalIPv6Address\":\"\",\"GlobalIPv6PrefixLen\":0,\"IPAMConfig\":null,\"IPAddress\":\"\",\"IPPrefixLen\":0,\"IPv6Gateway\":\"\",\"Links\":null,\"MacAddress\":\"\",\"NetworkID\":\"\"}}},\"OnBuild\":null,\"OpenStdin\":true,\"StdinOnce\":true,\"Tty\":true,\"User\":\"1000\",\"Volumes\":null,\"WorkingDir\":\"\"}"
Jun 07 12:20:04 2KLD7G3 dockerd[1787]: time="2023-06-07T12:20:04.186759653+02:00" level=debug msg="container mounted via layerStore: /home/shantanoo/docker/overlay2/91e4d0ec5f3553f0d71956469f7a0befe33ccc3f8bfc2300c447223683943ab7/merged" container=ad986c8bc1f23708d2bf44a732f96adef46c98154f3adad7c4b21b6b8d558caa
Jun 07 12:20:04 2KLD7G3 dockerd[1787]: time="2023-06-07T12:20:04.203496801+02:00" level=debug msg="Calling GET /v1.42/containers/ad986c8bc1f23708d2bf44a732f96adef46c98154f3adad7c4b21b6b8d558caa/json"
Jun 07 12:20:04 2KLD7G3 dockerd[1787]: time="2023-06-07T12:20:04.204142403+02:00" level=debug msg="Calling POST /v1.42/networks/tmp_default/disconnect"
Jun 07 12:20:04 2KLD7G3 dockerd[1787]: time="2023-06-07T12:20:04.204197703+02:00" level=debug msg="form data: {\"Container\":\"ad986c8bc1f23708d2bf44a732f96adef46c98154f3adad7c4b21b6b8d558caa\",\"Force\":false}"
Jun 07 12:20:04 2KLD7G3 dockerd[1787]: time="2023-06-07T12:20:04.212076026+02:00" level=debug msg="Calling POST /v1.42/networks/tmp_default/connect"
Jun 07 12:20:04 2KLD7G3 dockerd[1787]: time="2023-06-07T12:20:04.212149926+02:00" level=debug msg="form data: {\"Container\":\"ad986c8bc1f23708d2bf44a732f96adef46c98154f3adad7c4b21b6b8d558caa\",\"EndpointConfig\":{\"Aliases\":[\"tmp-demo-run-f9cd2c95842e\"],\"DriverOpts\":null,\"EndpointID\":\"\",\"Gateway\":\"\",\"GlobalIPv6Address\":\"\",\"GlobalIPv6PrefixLen\":0,\"IPAMConfig\":null,\"IPAddress\":\"\",\"IPPrefixLen\":0,\"IPv6Gateway\":\"\",\"Links\":null,\"MacAddress\":\"\",\"NetworkID\":\"\"}}"
Jun 07 12:20:04 2KLD7G3 dockerd[1787]: time="2023-06-07T12:20:04.221899154+02:00" level=debug msg="Calling PUT /v1.42/containers/ad986c8bc1f23708d2bf44a732f96adef46c98154f3adad7c4b21b6b8d558caa/archive?copyUIDGID=true&noOverwriteDirNonDir=true&path=%2F"
Jun 07 12:20:04 2KLD7G3 dockerd[1787]: time="2023-06-07T12:20:04.222268855+02:00" level=debug msg="container mounted via layerStore: /home/shantanoo/docker/overlay2/91e4d0ec5f3553f0d71956469f7a0befe33ccc3f8bfc2300c447223683943ab7/merged" container=ad986c8bc1f23708d2bf44a732f96adef46c98154f3adad7c4b21b6b8d558caa

ndeloof commented 1 year ago

can you try copying a random file in a test container ? (this replicates the way compose injects secrets)

$ docker create nginx
f692f0ccd344f3f69a4b9e5a2cfb9f0ffb6a6a1d8848af15bb9ad12383f369eb
$ docker cp compose.yaml f6:/foo
                                             Successfully copied 2.05kB to f6:/foo

shantanoo-desai commented 1 year ago

Case 1

docker-compose.yml

services:
  demo:
    image: alpine
    command: sleep 3600

Steps

docker compose up -d Works
touch text.file
docker cp ./text.file <container_name>:/ Works

logs: Successfully copied 1.54kB to secretserror-demo-1:/
docker compose exec -it <container_name> ls / Works

Case 2

$ docker create nginx
7667864fee050c98b6e7e342cb4fcd3a8a2bbe975f0d942cb9241f8e79b480f5
$ docker cp text.file <hash>/foo
Successfully copied 1.54kB to 76:/foo

Works

shantanoo-desai commented 1 year ago

@ndeloof here is a thorough analysis of the same docker compose files in two distinct Docker Engine Versions. This might need to be discussed also on Moby

Code under Test

A proof-of-concept repo I created a couple of months ago: https://github.com/shantanoo-desai/docker-compose-secrets-envvars
Grafana example in Issue

Discrepancy

Docker Engine Version (v24.0.2)

however the moment I introduce user: "1000" I get the same error 🔴
since most alpine images have root user for vanilla images, upon setting user: root works ✔️

This only cements my current conclusion that somehow the user needs to be the same name as that of the container image if not root

Docker Engine Versions (v23.0.6)

upon downgrading

theuser: "1000" works again ✔️
Same Grafana docker-compose.yml file in the issue body works with v23.0.6 again ✔️
Upon setting the user to grafana it will throw Error response from daemon: getent unable to find entry "grafana" in passwd database 🔴

ndeloof commented 1 year ago

tested your example repo

$  docker compose run alpine-test cat /run/secrets/test-credentials
supersecret 
$ docker compose run --user 1000 alpine-test cat /run/secrets/test-credentials
cat: can't open '/run/secrets/test-credentials': Permission denied

still can't reproduce the Could not find the file / in container error

I wonder: do you have containerd image store enabled ?

shantanoo-desai commented 1 year ago

My /etc/docker/daemon.json shows the following:

{
   "data-root": "/home/shantanoo/docker",
   "insecure-registries" : ["artifactory.internal.org"],
   "debug": true,
   "features": {
     "buildkit": true
   },
   "dns": ["10.24.64.11", "8.8.8.8"]
}

I am currently on my work-machine which is WSL2 on Windows 10, but the same error I got was on my personal machine with Manjaro Linux (although the daemon.json is not the same).

I wonder: do you have containerd image store enabled ?

No. on neither one of the instances

shantanoo-desai commented 1 year ago

Vagrant Boxes as Proof

@ndeloof the repository provides two isolated instances of the problem being faced with reproducible environments and working examples with results to back the claim. This is the maximum I can reach when it comes to reproducing the errors I get locally via Vagrant Boxes (VMs)

https://github.com/shantanoo-desai/docker-engine-secrets-error

ndeloof commented 1 year ago

Thanks for you test setup, I was able to reproduce issue also can reproduce in a multipass VM with standard docker installation (using get.docker.com) now investigating ...

ndeloof commented 1 year ago

This indeed is a moby issue, I logged https://github.com/moby/moby/issues/45719 with my debugging notes (basically, this is the same old getent error, but hidden by another one)

ndeloof commented 1 year ago

the reason I was not able to reproduce is I'm running latest codebase which includes https://github.com/docker/compose/pull/10598. The good news is that this will also bring you a fix (actually, workaround) for this issue

ndeloof commented 1 year ago

I'm closing this issue as we can follow up fix in https://github.com/moby/moby/issues/45719 and https://github.com/docker/compose/pull/10598 already reduces the impact of this bug

docker / compose