[BUG] Docker Compose volume creating directory instead of file

TheGroxEmpire commented 1 week ago

Description

I am trying to deploy Crowdsec with docker compose. The volumes config requires it to create acquis.yaml file in its volumes.

    volumes:
      - ./crowdsec/acquis.yaml:/etc/crowdsec/acquis.yaml

However, docker created acquis.yaml file as directory instead of yaml file. I tried forcing the docker to create the yaml file by making an empty acquis.yaml file in the host. But it outputs an error instead: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/home/opc/docker/crowdsec/acquis.yaml" to rootfs at "/etc/crowdsec/acquis.yaml": mount /home/opc/docker/crowdsec/acquis.yaml:/etc/crowdsec/acquis.yaml (via /proc/self/fd/6), flags: 0x5000: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

Steps To Reproduce

Run docker compose up -d on this config https://github.com/crowdsecurity/example-docker-compose/blob/main/npm/docker-compose.yml.
Observe that it creates an acquis.yaml directory.
Delete that directory and create an acquis.yaml file yourself.
Run docker compose up -d again.
Observe the error.

Compose Version

Docker Compose version v2.27.1

Docker Environment

Client: Docker Engine - Community
 Version:    26.1.4
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.14.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.27.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 11
  Running: 9
  Paused: 0
  Stopped: 2
 Images: 16
 Server Version: 26.1.4
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d2d58213f83a351ca8f528a95fbd145f5654e957
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-206.153.7.el9uek.aarch64
 Operating System: Oracle Linux Server 9.4
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 22.98GiB
 Name: cronox
 ID: 3ec56814-2b56-4027-b205-9eb49cc1b34b
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Anything else?

Other person encountering the same issue: https://discourse.crowdsec.net/t/cant-start-docker-container-acquis-yaml-is-a-directory/1891/6

ndeloof commented 1 week ago

I tried to reproduce without success relying on a minimal setup:

$  cat compose.yaml 
services:
  test:
    image: alpine
    volumes:
      ./foo/bar.txt:/foo/bar.txt
$   mkdir foo
$   echo "hello" > foo/bar.txt
$   docker compose run test ls -al foo
[+] Running 1/1
drwxr-xr-x    2 root     root          4096 Jun 25 14:08 .
drwxr-xr-x    1 root     root          4096 Jun 25 14:08 ..
-rw-r--r--    1 root     root             6 Jun 25 14:07 bar.txt

If you want a file to be bind-mounted you indeed need the source to exist before (otherwise engine will assume directory) For you ruse-case, can't you bind mount the whole directory instead ? i.e. ./crowdsec:/etc/crowdsec

idsulik commented 1 week ago

@TheGroxEmpire hi! execute docker rm {containerId} to remove old container. You created a docker container that has a bound volume with destination /etc/crowdsec/acquis.yaml and acquis.yaml is directory, because at the first time when you create the container you had no local file and docker created it as directory, but after you created local file and it sees that you want to bind local file to container's directory.

How to reproduce:

services:
  test:
    image: alpine
    volumes:
      ./foo/bar.txt:/foo/bar.txt

docker compose up -d
touch ./foo/bar.txt
docker-compose up -d

How to fix: remove old container

idsulik commented 1 week ago

@ndeloof Is this a bug or is it done on purpose so that when you create a container with volume, but you don't have a source file/folder, docker creates such a directory?

ndeloof commented 1 week ago

this is indeed default docker engine behavior.

docker / compose