Windows strange file mount behaviour

[meghuizen]

Description

I'm using .NET Aspire with docker to run some docker containers, but with some additional configuration files of my own. The issue I'm talking about here is NOT about Aspire, but is about docker cli on Windows. This however gives me extra pain when using it with Aspire due to conflicting logic there.

What I want to do I have an init.sql file in my folder and want to map that file as a file in a mysql container. For that I ran the following command: docker run -v mysql_data:/var/lib/mysql -v ./init.sql:/data/application/init.sql mysql:5.7.18

When I run this and do a ls -al on the folder, the file is located: drwxr-xr-x 2 root root 4096 Jul 30 10:02 init.sql Noticed the d of directory. While the source on the host machine is actually a file.

Then I ran (change in path delimiter): docker run -v mysql_data:/var/lib/mysql -v ./init.sql:/data/application/init.sql mysql:5.7.18

Now it is a file (good behaviour): -rwxrwxrwx 1 root root 54 Jun 28 14:36 init.sql

Now with full system paths docker run -v mysql_data:/var/lib/mysql -v C:\Users\michiele\source\repos\tp-aspire-architecture\tp-aspire-architecture\Dockerfiles\init.sql:/data/application/init.sql mysql:5.7.18 This results in a file mapping.

With inspect JSON

    "Mounts": [
        {
            "Type": "volume",
            "Name": "mysql_data",
            "Source": "/var/lib/docker/volumes/mysql_data/_data",
            "Destination": "/var/lib/mysql",
            "Driver": "local",
            "Mode": "z",
            "RW": true,
            "Propagation": ""
        },
        {
            "Type": "bind",
            "Source": "/run/desktop/mnt/host/c/Users/michiele/source/repos/tp-aspire-architecture/tp-aspire-architecture/Dockerfiles/init.sql",
            "Destination": "/data/application/init.sql",
            "Mode": "",
            "RW": true,
            "Propagation": "rprivate"
        }
    ],

MTAB:

C:\134 /data/application/init.sql 9p rw,dirsync,noatime,aname=drvfs;path=C:\;uid=0;gid=0;metadata;symlinkroot=/mnt/host/,mmap,access=client,msize=65536,trans=fd,rfd=4,wfd=4 0 0

docker run -v mysql_data:/var/lib/mysql -v C/Users/michiele/source/repos/tp-aspire-architecture/tp-aspire-architecture/Dockerfiles/init.sql:/data/application/init.sql mysql:5.7.18 This results in a directory mapping (which is faulty)

with inspect json:

"Mounts": [
        {
            "Type": "volume",
            "Name": "mysql_data",
            "Source": "/var/lib/docker/volumes/mysql_data/_data",
            "Destination": "/var/lib/mysql",
            "Driver": "local",
            "Mode": "z",
            "RW": true,
            "Propagation": ""
        },
        {
            "Type": "bind",
            "Source": "/run/desktop/mnt/host/C/Users/michiele/source/repos/tp-aspire-architecture/tp-aspire-architecture/Dockerfiles/init.sql",
            "Destination": "/data/application/init.sql",
            "Mode": "",
            "RW": true,
            "Propagation": "rprivate"
        }
    ],

MTAB:

/dev/sdc /data/application/init.sql ext4 rw,relatime,discard,errors=remount-ro,data=ordered 0 0

How does Aspire do this? Aspire always uses a full file path from a Windows notation:

var filePath = "C:\\Users\\michiele\\source\\repos\\tp-aspire-architecture\\tp-aspire-architecture\\init.sql";

var mysql = builder.AddContainer("mysql", "mysql", "5.7.18")
                            .WithEndpoint(5455, 3306, "mysql")
                            .WithVolume("mysql_data", "/var/lib/mysql")
                            .WithAnnotation(new ContainerMountAnnotation(filePath, "/data/application/init.sql", ContainerMountType.BindMount, false));

This will be translated to:

"Mounts": [
            {
                "Type": "volume",
                "Source": "mysql_data",
                "Target": "/var/lib/mysql"
            },
            {
                "Type": "bind",
                "Source": "/run/desktop/mnt/host/c/Users/michiele/source/repos/tp-aspire-architecture/tp-aspire-architecture/init.sql",
                "Target": "/data/application/init.sql"
            }
        ],

but this also results in a directory mapping (faulty one!)

Reproduce

1. create an init.sql file
2. Run: docker run -v mysql_data:/var/lib/mysql -v ./init.sql:/data/application/init.sql mysql:5.7.18
3. Open a shell in your container and do ls -al /data/application and see it's a directory (faulty behaviour)
4. Run: docker run -v mysql_data:/var/lib/mysql -v .\init.sql:/data/application/init.sql mysql:5.7.18
5. Open a shell in your container and do ls -al /data/application and see it's a file (good behaviour)

Expected behavior

I'm expecting in all situations this will be mapped as a file and not a directory.

docker version

Client:
 Version:           27.0.3
 API version:       1.46
 Go version:        go1.21.11
 Git commit:        7d4bcd8
 Built:             Sat Jun 29 00:03:32 2024
 OS/Arch:           windows/amd64
 Context:           desktop-linux

Server: Docker Desktop 4.32.0 (157355)
 Engine:
  Version:          27.0.3
  API version:      1.46 (minimum version 1.24)
  Go version:       go1.21.11
  Git commit:       662f78c
  Built:            Sat Jun 29 00:02:50 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.18
  GitCommit:        ae71819c4f5e67bb4d5ae76a6b735f29cc25774e
 runc:
  Version:          1.7.18
  GitCommit:        v1.1.13-0-g58aa920
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    27.0.3
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.15.1-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.28.1-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-compose.exe
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.32
    Path:     C:\Program Files\Docker\cli-plugins\docker-debug.exe
  desktop: Docker Desktop commands (Alpha) (Docker Inc.)
    Version:  v0.0.14
    Path:     C:\Program Files\Docker\cli-plugins\docker-desktop.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     C:\Program Files\Docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.25
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     C:\Program Files\Docker\cli-plugins\docker-feedback.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.3.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scout: Docker Scout (Docker Inc.)
    Version:  v1.10.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 3
  Running: 1
  Paused: 0
  Stopped: 2
 Images: 14
 Server Version: 27.0.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae71819c4f5e67bb4d5ae76a6b735f29cc25774e
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
 Kernel Version: 5.15.153.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.49GiB
 Name: docker-desktop
 ID: 2a082a0d-bfaa-443b-8978-07e9d757cbfd
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=npipe://\\.\pipe\docker_cli
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: daemon is not using the default seccomp profile

Additional Info

No response

[thaJeztah]

I suspect this may be related to conversions happening in Docker Desktop.

For context; bind-mounts happen on the daemon-side, not from the client side; e.g., if you pass -v /some/path:/inside-container/path (mount /some/path into the container at /inside-container/path), as path to be mounted, that path is sent to the daemon which creates a container, and lets the OCI runtime mount the path into it; when using the -v <src>:<dst> shorthand syntax, the daemon also creates the host-path if it's not found, but this defaults to a directory, so if /some/file.txt does not exist on the host where the daemon runs, it will create a directory named /some/file.txt.

Docker uses a client-server architecture (the CLI uses the daemon API to control it); that means that the docker CLI and the Docker Daemon (dockerd) can run on separate machines, which may be a remote daemon (remote datacenter), or a "VM" running on your local machine; this is the case when running Docker Desktop; the Linux Daemon runs inside a VM (on Windows: Hyper-V or WSL2), and may not have direct access to the files you're trying to bind-mount from your client (where your docker CLI runs).

Docker Desktop tries to make this work transparently; it shares files from your (Windows) host with the Linux VM, and where needed translates paths to the corect location (C:\some\path\on\Windows -> /run/desktop/mnt/host/C/some/path/on/Windows inside the VM). Depending on where the docker CLI is executed (Windows, WSL2), different translations may be needed, and I suspect something goes wrong in some of these permutations, resulting in a non-existing path to be passed to the daemon, which in its turn then creates a directory at that location (as outlined further above).

[meghuizen]

I might circumvent this with creating my own images on top of the original images, however that makes my development pipeline a bit more cumbersome.

@thaJeztah thanks for the deduction. That makes a lot of sense.

[thaJeztah]

Yeah, it's a tricky one as there are many moving parts at play; more so on Windows, where there's multiple combinations possible (in addition, MINGW automatic path conversion has been known to cause havoc; https://stackoverflow.com/a/34386471). I'm not on a Windows machine, so not directly able to try to reproduce, but I can move this ticket to the https://github.com/docker/for-win issue tracker; possibly for the team working on those parts to look.

In general, there's some options that (depending on your situation) could help;

• Where possible, mount directories instead of files; this can help in situations where the daemon created a directory (in a location where you expected a file), but also can help in situations where tools modify the mounted-file; many tools don't update files directly but make a temporary copy (open file -> on save: write to a temporary file, then remove the original file, and write the temporary file in its place); such changes won't show up in bind-mounts, as the mount is tied to the file-descriptor of the original (now "deleted") file.
• You can use the advanced --mount syntax instead of the -v short-hand. The --mount flag, unlike the -v shorthand, does not automatically create missing paths on the host; it requires the path to mount to be an existing location, and otherwise will error out. This can prevent situations where an incorrect path either ends up with an empty directory to be mounted (and created on the host), making such errors a hard failure instead. The --mount syntax provides more options, at the cost of being more verbose, so YMMV (see https://docs.docker.com/storage/bind-mounts/#choose-the--v-or---mount-flag).

[meghuizen]

Might indeed be good to move to the for-win issue tracker.

Good advice on the mount syntax. I might be able to come away with that one, since I can pass those arguments directly.

[thaJeztah]

I moved the ticket 👍