Docker stack deploy do not undersand volumes in windows format

[eiximenis]

Description

I am using Docker CE For Windows (Version 2.0.0.0-beta1-win75 (19925) and using docker stack deploy to deploy a workload against the local k8s cluster.

If I define a volume using "linux like" syntax it works OK:

    volumes:     
        -/c/tfs/eShopOnContainers/src/ApiGateways/Mobile.Bff.Shopping/apigw:${ESHOP_OCELOT_VOLUME_SPEC:-/app/configuration}

But If I use windows syntax it fails:

    volumes:      
        -c:\tfs\eShopOnContainers\src\ApiGateways\Mobile.Bff.Shopping\apigw:${ESHOP_OCELOT_VOLUME_SPEC:-/app/configuration}

Then I receive on error:

Stack.compose.docker.com "eshop" is invalid: eshop: Invalid value: "null": conversion to kube entities failed: c:\tfs\eShopOnContainers\ApiGateways\Mobile.Bff.Marketing\apigw: only absolute paths can be specified in mount source

Note that if instead of c:\tfs\... I use something like \c\tfs\... (linux-syntax with no colon but with windows backslashes) I receive another error:

Stack.compose.docker.com "eshop" is invalid: spec.stack.services[1].volumes[0]: Invalid value: "\\c\\tfs\\eShopOnContainers\\src\\ApiGateways\\Mobile.Bff.Shopping\\apigw": not a valid volume name in Kubernetes: a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character

This is very unfortunate, because under windows, docker-compose changes the format of Linux paths to windows paths when using "docker-compose config". So, if I have a compose file with one volume:

- /c/tfs/eShopOnContainers/ApiGateways/Mobile.Bff.Marketing/apigw:${ESHOP_OCELOT_VOLUME_SPEC:-/app/configuration}

The output of docker-compose config is (linux slashes are converted to backslashes):

    volumes:
    - \c\tfs\eShopOnContainers\ApiGateways\Mobile.Bff.Marketing\apigw:/app/configuration:rw

But this format is not supported by docker stack deploy. This prevents me to use docker-compose config | docker stack deploy -c -.

Output of docker version:

Client: Docker Engine - Community
 Version:           18.09.0-ce-beta1
 API version:       1.39
 Go version:        go1.10.4
 Git commit:        78a6bdb
 Built:             Thu Sep  6 22:42:13 2018
 OS/Arch:           windows/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.0-ce-beta1
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       78a6bdb
  Built:            Thu Sep  6 22:49:35 2018
  OS/Arch:          linux/amd64
  Experimental:     true
 Kubernetes:
  Version:          v1.10.3
  StackAPI:         v1beta2

Output of docker info:

Containers: 84
 Running: 41
 Paused: 0
 Stopped: 43
Images: 212
Server Version: 18.09.0-ce-beta1
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.125-linuxkit
Operating System: Docker for Windows
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.837GiB
Name: linuxkit-00155d016143
ID: M5NF:TR4Q:KLJD:VP6F:SWCS:RINA:BSWB:55AK:U4ON:QAM7:44TV:VKRV
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 241
 Goroutines: 211
 System Time: 2018-10-01T16:58:03.4059414Z
 EventsListeners: 1
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

Additional environment details (AWS, VirtualBox, physical, etc.):

[thaJeztah]

Looks like this has the same root-cause as https://github.com/docker/cli/issues/1404. Does this work if you temporarily hardcode the windows path? (not using the ENV-var substitution?)

[eiximenis]

Hi! It works if I hardcode a windows path but using Linux Syntax (i. e. /c/tfs/...). What is curious is that docker-compose config returns paths in windows syntax even though Linux syntax were used in compose file (and that is what prevents me to use docker-compose config | docker stack deploy -c -)

[mcadac]

I found a solution (the difference is that I am using "docker-app" plugin but if you implement the same solution in your "docker compose" file, I think that it works), you can see it here:

https://github.com/mcadac/docker/blob/master/docker-app/gotemplate/README.md

[Drewster727]

I'm still running into this problem. My swarm masters are all linux (CentOS) and they have no idea how to deploy to windows nodes if I provide a windows volume bind path like so:

services:
  app:
    image: app:latest
    volumes:
      - Z:\hostfolder:C:\containerfolder

It also did not work for me if I changed it to:

services:
  app:
    image: app:latest
    volumes:
      - /z/hostfolder:/c/containerfolder

This is pretty discouraging :(

[mcadac]

The volumes must be defined the verbose form and it must be the type "bind"

[Drewster727]

@mcadac Thanks -- just tried that and still no bueno:

services:
  app:
    image: app:latest
    volumes:
      - type: bind
        source: /z/hostfolder
        target: /c/containerfolder

invalid bind mount source, must be an absolute path: /z/hostfolder

What am I missing?

[Drewster727]

@thaJeztah I saw your merge! Thank you! When can we expect this to get released would you think?

[thaJeztah]

This will likely go out in the August patch release for 18.09, and included in either 19.03.0 or the first patch release for 19.03 (19.03.1). You should be able to download nightly builds of the CLI from download.docker.com later today or tomorrow (not sure at what time "nightly" is built 😂)

[thaJeztah]

Note that the fix will only address the "long" syntax; https://docs.docker.com/compose/compose-file/#long-syntax-3

The shorthand syntax (-v C:\hostpath:C:\container-path) won't be fixed by that PR; the shorthand syntax is unfortunately quite ambiguous, and fixing that is non-trivial (if possible at all).

[Drewster727]

@thaJeztah I had another related question to this issue. Will your merge also resolve cross-platform service creation with npipe mount type? For example, I have a linux (centos) manager, and I want to create a service from that manager that only runs on my windows platform nodes:

docker service create --name portainer-agent-win --mode global --mount "type=npipe,source=\\.\pipe\docker_engine,target=\\.\pipe\docker_engine" --constraint 'node.platform.os == windows' --env AGENT_CLUSTER_ADDR=tasks.portainer-agent --network portainer-agent portainer/agent:windows1903-amd64-1.3.0

Using docker 19.03, from a linux manager node, it seems to complain about the mount path: invalid mount config for type "npipe": invalid mount path: '\.\pipe\docker_engine' I tried the above mount source/target paths in different ways (//, \, \, ...)

However, running that service create command from a windows manager, works like a charm. I do see that npipe support was added for compose/stack files, but perhaps not allowed in a cross-platform situation: https://github.com/docker/cli/pull/1195

[Drewster727]

@thaJeztah when you get a chance! Looking for some info on the above. Thank you!

-Drew

[thaJeztah]

@Drewster727 hm.. could it be that if you're running that from a Linux command-line, that the shell treats the \ as an escape character, so

target=\\.\pipe\docker_engine

will be converted to

target=\.\pipe\docker_engine

Trying the following (I don't have a Windows node, but just to test the escaping idea);

docker service create --detach --name=test --mount "type=npipe,source=\\.\pipe\docker_engine,target=\\.\pipe\docker_engine" nginx:alpine

Inspecting the service spec (output formatted with jq for readability):

docker service inspect --format '{{ json .Spec.TaskTemplate.ContainerSpec.Mounts}}' test 

[
  {
    "Type": "npipe",
    "Source": "\\.\\pipe\\docker_engine",
    "Target": "\\.\\pipe\\docker_engine"
  }
]

Which confirms that both the Source and Target path are now invalid; both have a single leading \ (\.\pipe\docker_engine\)

Changing the command to use either escaping backslashes, or using single quotes make sure that the shell doesn't see the back-slashes as escape characters will prevent that;

Creating the service and escaping the back-slashes (\ -> \\);

docker service create --detach --name=test --mount "type=npipe,source=\\\\.\\pipe\\docker_engine,target=\\\\.\\pipe\\docker_engine" nginx:alpine

docker service inspect --format '{{ json .Spec.TaskTemplate.ContainerSpec.Mounts}}' test

[
  {
    "Type": "npipe",
    "Source": "\\\\.\\pipe\\docker_engine",
    "Target": "\\\\.\\pipe\\docker_engine"
  }
]

Single quotes (probably a better solution as it would work both on Windows and Linux shells):

docker service create --detach --name=test --mount 'type=npipe,source=\\.\pipe\docker_engine,target=\\.\pipe\docker_engine' nginx:alpine

docker service inspect --format '{{ json .Spec.TaskTemplate.ContainerSpec.Mounts}}' test

[
  {
    "Type": "npipe",
    "Source": "\\\\.\\pipe\\docker_engine",
    "Target": "\\\\.\\pipe\\docker_engine"
  }
]

[Drewster727]

@thaJeztah I'm not sure why I didn't try the double slashes. I thought I had tried it, but that did the trick.

docker service create --name portainer-agent-win --mode global --mount "type=npipe,source=\\\\.\\pipe\\docker_engine,target=\\\\.\\pipe\\docker_engine" --constraint 'node.platform.os == windows' --network portainer-agent -e AGENT_CLUSTER_ADDR=tasks.portainer-agent  portainer/agent:windows1903-amd64-1.3.0

Hopefully this helps someone else in the same situation. Thank you! Drew

[thaJeztah]

haha, no worries! I also had to scratch my head a few times before I saw what's happening.

The single quotes are probably a bit easier to use, and might work cross-platform (linux/windows) but haven't tried on a Windows machine

[Drewster727]

@thaJeztah thanks again for the fix for this, 19.03.2 works like a charm!

[bryht]

Note that the fix will only address the "long" syntax; https://docs.docker.com/compose/compose-file/#long-syntax-3

The shorthand syntax (-v C:\hostpath:C:\container-path) won't be fixed by that PR; the shorthand syntax is unfortunately quite ambiguous, and fixing that is non-trivial (if possible at all).

I tried long version, it looks still not working. I am using version 19.03.4, did I config something wrong?

docker stack deploy -c .\test.yml mystack2
Stack.compose.docker.com "mystack2" is invalid: mystack2: Invalid value: "null": conversion to kube entities failed: C:\Users\xxx\Desktop\config\data\mssql: only absolute paths can be specified in mount source

[rong0312]

Hey I am facing the same issue now days (docker version 19.03.5)

  volumes:
      - C:\config:C:\config

Not working! Actually, my service not getting started in the first place some why (normal docker-compose command will start my container!) Anything new with this issue?

[thaJeztah]

@bryht your case looks slightly different, and looks to be related to the Kubernetes conversion. @chris-crone any ideas on that one?

@rong0312 the fix that was made only makes changes to the long-hand syntax, not the shorthand one; https://github.com/docker/cli/issues/1403#issuecomment-510511630

The shorthand syntax (-v C:\hostpath:C:\container-path) won't be fixed by that PR; the shorthand syntax is unfortunately quite ambiguous, and fixing that is non-trivial (if possible at all).

[chris-crone]

@chris-crone any ideas on that one?

It looks like that error is coming from Compose on Kubernetes: https://github.com/docker/compose-on-kubernetes/blob/master/internal/convert/volumes.go#L49

@silvin-lubecki or @aiordache can you PTAL?

[rong0312]

I can't find a proper way to volume my stack's service. I tried to bind & volume in all variations, everything fails to volume. Can someone who succeeded in doing it explain how he has done it?

This problem relates only to 'docker stack deploy', my volume works as normal docker-compose up

[Drewster727]

@rong0312

docker

docker run --mount "type=bind,source=c:\temp\,target=c:\temp" someimage:latest

docker-compose

version: "3.7"

services:
  test:
    image: someimage:latest
    volumes:
        - type: bind
          source: C:\temp
          target: C:\temp

That help at all?

[justinkb]

Finally found out the correct syntax when using Docker Desktop to deploy a stack with the local kubernetes cluster:

/run/desktop/mnt/host/c/Users/bla/bla:/home/bla/bla

as an example if you wanted to bind C:\Users\bla\bla to /home/bla/bla

[thaJeztah]

@justinkb what version of docker (and docker desktop) are you running? IIRC, docker desktop performs some conversion to convert windows paths to the correct (internal) path to use for Docker Desktop (but not sure if that conversion is done when using kubernetes) @djs55 do you know?

[justinkb]

Running the latest "stable" channel version

On Sat, May 16, 2020, 2:43 PM Sebastiaan van Stijn notifications@github.com wrote:

@justinkb https://github.com/justinkb what version of docker (and docker desktop) are you running? IIRC, docker desktop performs some conversion to convert windows paths to the correct (internal) path to use for Docker Desktop (but not sure if that conversion is done when using kubernetes) @djs55 https://github.com/djs55 do you know?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/docker/cli/issues/1403#issuecomment-629640047, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABVHKHOAFJVOWCQQ2K4RHTRR2C6HANCNFSM4FYKBFKA .

[djs55]

I believe /run/desktop/mnt/host is the underlying mount point used in WSL 2 mode.

Docker Desktop uses an API proxy to try to hide some of these differences, but there was a regression in the latest 2.3.0.2 stable which we're attempting to fix. Perhaps try the experimental build mentioned here to see if it helps at all?