mapping docker host path fails under WSL

rsking commented 3 weeks ago

Is there an existing issue for this?

[X] I have searched the existing issues

Describe the bug

When trying to mount a path in the docker host under WSL, aspire tries to convert it to a windows path, and create the folder locally.

builder.WithAnnotation(new ContainerMountAnnotation("/var/run/docker.sock", "/var/run/docker.sock", ContainerMountType.BindMount, isReadOnly: true))

causes the folder C:\var\run\docker.sock to be created, although the mount is correct.

builder.WithContainerRuntimeArgs("-v /var/run/docker.sock:/var/run/docker.sock");

throws

could not create the container    {"Container": {"name":"localstack-rrzzmnnz"}, "Reconciliation": 3, "error": "docker command 'CreateContainer' returned with non-zero exit code 1: command output: Stdout: '' Stderr: 'Error response from daemon: create  /var/run/docker.sock: \" /var/run/docker.sock\" includes invalid characters for a local volume name, only \"[a-zA-Z0-9][a-zA-Z0-9_.-]\" are allowed. If you intended to pass a host directory, use absolute path\n'"}

and

builder.WithBindMount("/var/run/docker.sock", "/var/run/docker.sock", isReadOnly: true);

does not throw an error, but does not mount /var/run/docker.sock correctly.

Expected Behavior

No error is thrown, and /var/run/docker.sock is mounted from the docker host

Steps To Reproduce

In description

Exceptions (if any)

No response

.NET Version info

.NET SDK:
 Version:           8.0.400
 Commit:            36fe6dda56
 Workload version:  8.0.400-manifests.56cd0383
 MSBuild version:   17.11.3+0c8610977

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.22631
 OS Platform: Windows
 RID:         win-x64
 Base Path:   C:\Program Files\dotnet\sdk\8.0.400\

.NET workloads installed:
Configured to use loose manifests when installing new manifests.
 [aspire]
   Installation Source: VS 17.11.35219.272
   Manifest Version:    8.1.0/8.0.100
   Manifest Path:       C:\Program Files\dotnet\sdk-manifests\8.0.100\microsoft.net.sdk.aspire\8.1.0\WorkloadManifest.json
   Install Type:        FileBased

Host:
  Version:      8.0.8
  Architecture: x64
  Commit:       08338fcaa5

.NET SDKs installed:
  8.0.400 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.33 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.33 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 6.0.33 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 8.0.8 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found:
  x86   [C:\Program Files (x86)\dotnet]
    registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

Anything else?

No response

davidfowl commented 3 weeks ago

Feels like this should be abstracted since we support multiple container runtimes. Maybe we can make this a more first class feature.

cc @karolz-ms

karolz-ms commented 3 weeks ago

@rsking first of all, are you running your program from within Windows OS, or from WSL subsystem? Are you using Docker Desktop, Podman, something else? And are you able to get the mount configured correctly by using Docker CLI directly? If so, please share the command.

Here is what I think is happening. If you are running your Aspire program from Windows, which seems like you do, I am not aware of a solution that would allow you to bind-mount the Docker socket from Windows host into your container. There simply is no "Docker socket" on the Windows host--e.g. Docker Desktop uses a named pipe instead https://docs.docker.com/desktop/faqs/general/#how-do-i-connect-to-the-remote-docker-engine-api and I do not think the bind mount syntax will work to proxy between Unix domain socket on the container side and a Windows named pipe on the Windows host side.

Your best bet is probably to run your Aspire program from within WSL subsystem, then

builder.WithBindMount("/var/run/docker.sock", "/var/run/docker.sock", isReadOnly: true);

should work, assuming the Docker socked is available from within WSL subsystem.

@danegsta please chime in if I confused something here.

rsking commented 3 weeks ago

@karolz-ms whilst I'm not a docker expert, these are my finings.

I am running on Windows, and have tried using both docker, and podman.

This is specifically to get LocalStack to operate correctly. It communicates with the docker API, via the socket.

when I run the command without the volume mount,

docker run --rm -e LOCALSTACK_DEBUG=1 docker.io/localstack/localstack:3.5

I get this debug message from localstack.

LocalStack supervisor: starting
LocalStack supervisor: localstack process (PID 15) starting
2024-08-26T23:58:25.390 DEBUG --- [  MainThread] l.utils.docker_utils       : Using SdkDockerClient. LEGACY_DOCKER_CLIENT: False, SDK installed: True
2024-08-26T23:58:25.392 DEBUG --- [  MainThread] l.u.c.docker_sdk_client    : Creating Docker SDK client failed: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory')). If you want to use Docker as container runtime, make sure to mount the socket at /var/run/docker.sock
2024-08-26T23:58:25.619  WARN --- [  MainThread] l.services.internal        : Enabling diagnose endpoint, please be aware that this can expose sensitive information via your network.
2024-08-26T23:58:25.634 DEBUG --- [  MainThread] plux.runtime.manager       : instantiating plugin PluginSpec(localstack.runtime.components.aws = <class 'localstack.aws.components.AwsComponents'>)
2024-08-26T23:58:25.634 DEBUG --- [  MainThread] plux.runtime.manager       : loading plugin localstack.runtime.components:aws

LocalStack version: 3.5.0
LocalStack build date: 2024-06-13
LocalStack build git hash: 54edb407d

when mounting the socket directly

docker run --rm -e LOCALSTACK_DEBUG=1 -v /var/run/docker.sock:/var/run/docker.sock docker.io/localstack/localstack:3.5

the error about not being able to get the server API version is gone

LocalStack supervisor: starting
LocalStack supervisor: localstack process (PID 16) starting
2024-08-27T00:00:37.458 DEBUG --- [  MainThread] l.utils.docker_utils       : Using SdkDockerClient. LEGACY_DOCKER_CLIENT: False, SDK installed: True
2024-08-27T00:00:37.582  WARN --- [  MainThread] l.services.internal        : Enabling diagnose endpoint, please be aware that this can expose sensitive information via your network.
2024-08-27T00:00:37.593 DEBUG --- [  MainThread] plux.runtime.manager       : instantiating plugin PluginSpec(localstack.runtime.components.aws = <class 'localstack.aws.components.AwsComponents'>)
2024-08-27T00:00:37.593 DEBUG --- [  MainThread] plux.runtime.manager       : loading plugin localstack.runtime.components:aws

LocalStack version: 3.5.0
LocalStack build date: 2024-06-13
LocalStack build git hash: 54edb407d

This indicates to me that the communication to the docker host (i.e. on the WSL machine) has been successful.

The idea is not to mount the windows docker, but rather the linux docker from WSL. Which seems to work when you specify a linux path.

rsking commented 3 weeks ago

In my experimentation through aspire, it seems to be making assumptions about the paths being passed in.

with the command

builder.WithBindMount("/var/run/docker.sock", "/var/run/docker.sock", isReadOnly: true);

it gets the full path to the file, which prepends the AppHost directory. this means that whilst the mount is successful, it mounts a file that doesn't work, as it mounts C:\var\run\docker.sock. Aspire also tried to be helpful here by creating that folder.

This leads to the docker API call in localstack failing with,

2024-08-27T10:23:46.6388250 2024-08-27T00:23:46.638 DEBUG --- [  MainThread] l.u.c.docker_sdk_client    : Creating Docker SDK client failed: Error while fetching server API version: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused')). If you want to use Docker as container runtime, make sure to mount the socket at /var/run/docker.sock

I got around this by specifying the ContainerMountAnnotation directly, which does pass through the mount correctly, but also creates a useless folder at /var/run/docker.sock, which is not actually mounted. This works, apart from the empty folder being created.

I then thought the specifying the mount directly as ContainerRunArgs should get around this completely, but it appears like dcpctrl is doing something to those arguments. These are the same args that have worked when specifying this through the Docker CLI. This is the part that I have not been able to debug as to why this is occurring, as it seems to be occurring in dcpctrl.

rsking commented 3 weeks ago

I have created a minimal repo at https://github.com/rsking/AspireDockerSock

karolz-ms commented 3 weeks ago

OK this is all very helpful, thank you @rsking. I think I know what is going on.

Looks like Docker Desktop is treating the host path /var/run/docker.sock in a very special way. It allows to "bind mount" that path into the container even though it does not exist on the host machine at all; instead it silently creates a bind mount into Docker daemon VM. I am quite surprised that it works and I could not find anything about this special treatment in the Docker Desktop documentation, but it kind of makes sense.

Aspire should probably special-case this mount source (/var/run/docker.sock) as well. Specifically

WithAnnotation() results in an unnecessary C:\var\run\docker.sock folder being created because Aspire (DCP specifically) is trying to be helpful and noticed that you are attempting a bind mount from a folder that does not exist on the machine. So it is trying to create that folder for you. We should stop doing that for /var/run/docker.sock.
WithContainerRuntimeArgs() may not work simply because there should be two paramenters passed to it, that is, the way to call it is WithContainerRuntimeArgs("-v", "/var/run/docker.sock:/var/run/docker.sock"). Would appreciate a verification here; this should work.
WithBindMount() does not work because Aspire is trying to compute the absolute path to /var/run/docker.sock and failing at that https://github.com/dotnet/aspire/blob/main/src/Aspire.Hosting/ContainerResourceBuilderExtensions.cs#L80 We should not do this for /var/run/docker.sock source.

rsking commented 3 weeks ago

I can see the difference between ContainerMountAnnotation and ContainerRunArgs is in the ContainerSpec class, the former populates the VolumeMount collection, and the later populates the RunArgs collection.

rsking commented 3 weeks ago

Ah! I can confirm that the WithContainerRuntimeArgs does work when specifying with two arguments. I had incorrectly assumed that I would put in the exact string that I wanted to be supplied on the command line.

karolz-ms commented 3 weeks ago

Great, thanks for confirmation! That makes me more confident that the proposed fixes https://github.com/dotnet/aspire/issues/5378#issuecomment-2311383462 are the right call.

Thank you for submitting the issue and providing us with more information 🤗

rsking commented 3 weeks ago

I'm not completely sure, but I think it is more of a case of docker through WSL treating windows paths differently. From what I understand, it more changes C:\temp\whatever to /mnt/c/temp/whatever when it actually does the mount. When it sees a direct linux path, then it does not do the win->wsl path conversion. That is more from experience, rather than documentation,

dotnet / aspire