Data does not appear after mount volume by container.

[luizhpriotto]

Hello, i need a help, please.

I use Photon 1.0 Build 62c543d

After create volume with:

docker volume create --name=bafofa --driver=vsphere -o size=2gb

and create container:

docker run -ti -v bafofa:/mnt/bafofa busybox

I can access the data with:

mount /tmp/bafofa /dev/sdb

(/dev/disk/by-path/pci-0000:0b:00.0-scsi-0:0:0:0 -> ../../sdb)

But when i access with the container in "/mnt/bafofa/", and i create some file, but this file appears only "/var/lib/docker/plugins/564115c25a21cdfb0ea4e3ae11b66e640549358eb6d99234cd6538d7971496f2/rootfs/mnt/vmdk/bafofa/". I can not create files in VMDK.

I Tried to use in other nodes, but the problem still occurred.

Thanks so much!

[govint]

@luizhpriotto can you attach /var/log/docker-volume-vsphere.log from the host that you are running these commands on. Also, can you post the result of "docker volume inspect bafofa"

[luizhpriotto]

Thanks for helping me @govint! Follow the attachment and the "inspect". { "Driver": "vsphere:latest", "Labels": {}, "Mountpoint": "/mnt/vmdk/bafofa", "Name": "bafofa", "Options": { "size": "2gb" }, "Scope": "global", "Status": { "access": "read-write", "attach-as": "independent_persistent", "attached to VM": "Photon02", "attachedVMDevice": { "ControllerPciSlotNumber": "192", "Unit": "0" }, "capacity": { "allocated": "103MB", "size": "2GB" }, "clone-from": "None", "created": "Tue Oct 10 04:45:34 2017", "created by VM": "Photon02", "datastore": "LUN_FCR5_001", "diskformat": "thin", "fstype": "ext4", "status": "attached" } }

docker-volume-vsphere.log

[govint]

@luizhpriotto thanks for providing these logs, the volume plugin is able to mount the volume with no errors, 2017-10-10 04:45:34.100682773 +0000 UTC [INFO] Attaching volume and creating filesystem fstype=ext4 name=bafofa 2017-10-10 04:45:34.950217394 +0000 UTC [INFO] Scan complete device="/dev/disk/by-path/pci-0000:0b:00.0-scsi-0:0:0:0" event="/dev/disk/by-path/pci-0000:0b:00.0-scsi-0:0:0:0": 0x100 == IN_CREATE 2017-10-10 04:45:35.991076312 +0000 UTC [INFO] Volume and filesystem created name=bafofa fstype=ext4 2017-10-10 04:58:12.236893358 +0000 UTC [INFO] Mounting volume name=bafofa 2017-10-10 04:58:12.848529173 +0000 UTC [INFO] Scan complete device="/dev/disk/by-path/pci-0000:0b:00.0-scsi-0:0:0:0" event="/dev/disk/by-path/pci-0000:0b:00.0-scsi-0:0:0:0": 0x100 == IN_CREATE

Can you try the steps and post the content of /proc/mounts (_cat /proc/mounts) and also the result of docker inspect for the container thats unable to see the data. The mounts section of the container meta-data should indicate the volume thats mounted for the container at the specified mount point, like this: "Mounts": [ { "Type": "volume", "Name": "newvol", "Source": "/vol/docker/plugins/1cb151804dbfeafb009dfc2cb5f649f2eef6632e6faac621a7799a0ccad53503/rootfs/mnt/vmdk/newvol@sharedDS", "Destination": "/mnt", "Driver": "vsphere", "Mode": "", "RW": true, "Propagation": "" } ],

Can you also confirm that the plugin isn't being disabled during the test. Do docker plugin ls when the data isn't visible in the container.

Can you also post the version of docker you are using (docker version and docker info).

[govint]

@luizhpriotto , can you verify that any mounts of the same volume are all unmounted before the test is performed.

[luizhpriotto]

mount.txt @govint i get more information:

"HostConfig": { "Binds": [ "bafofa:/mnt/bafofa" ], "ContainerIDFile": "", "LogConfig": { "Type": "json-file", "Config": {} }, "NetworkMode": "default", "PortBindings": {}, "RestartPolicy": { "Name": "no", "MaximumRetryCount": 0 }

"Mounts": [ { "Type": "volume", "Name": "bafofa", "Source": "/var/lib/docker/plugins/564115c25a21cdfb0ea4e3ae11b66e640549358eb6d99234cd6538d7971496f2/rootfs/mnt/vmdk/bafofa", "Destination": "/mnt/bafofa", "Driver": "vsphere:latest", "Mode": "", "RW": true, "Propagation": "" }

The #cat /proc/mount

mount.txt

When I access "/var/lib/docker/plugins/564115c25a21cdfb0ea4e3ae11b66e640549358eb6d99234cd6538d7971496f2/rootfs/mnt/vmdk/bafofa" is it the same as accessing vmdk (/dev/sdb)?

What is the /dev/root? There is not in my system!

As soon as possible I will do new tests, thank you @govint!

[govint]

@luizhpriotto thanks for posting these data, can you also say what guest OS and version you are running.

[luizhpriotto]

@govint Follow: Linux photon02 4.4.88-1.ph1-esx #1-photon SMP Mon Oct 2 19:27:23 UTC 2017 x86_64 GNU/Linux VMware Photon Linux 1.0 PHOTON_BUILD_NUMBER=62c543d

thx

[luizhpriotto]

Hello @govint , I did a test in Oracle Linux 7, and the result was different:

The mount in the container busybox was: /dev/disk/by-path/pci-0000:0b:00.0-scsi-0:0:0:0 on /mnt/bafofa

And in host was: /dev/sdb on /var/lib/docker/plugins/a7e24d9096d01d01001850b9905b140f5ec8fbb6d8c90cb7803ad15ada8d4b2f/rootfs/mnt/vmdk/bafofa@LUN_FCR5_001 type ext4 (rw,relatime,data=ordered)

In Oracle Linux 7 the VMWARE DVS work!

But in Photon no.

The mount /dev/root like in Photon 1.0 does not exist, could it be a bug?

[govint]

@luizhpriotto, thanks for confirming the plugin works as expected on Oracle Linux, Photon Linux photon-machine 4.4.41-1.ph1-esx #1-photon SMP Tue Jan 10 23:46:44 UTC 2017 x86_64 GNU/Linux also works fine. I'm getting Photon 2.0 and checking the behavior there. Let me check and get back to you.

Regarding the mounts inside the container and outside (in the Linux OS), the mount in the Linux OS is like this:

/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:0 /mnt/vmdk/newvol1@sharedVmfs-0 ext4 rw,relatime,data=ordered 0 0

And the mount recorded inside the container is like this,

/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:0 on /mnt type ext4 (rw,relatime,data=ordered)

The device is the same as seen within and outside the container. Once the volume is created and when its mounted for a container, this is what happens,

1. the plugin mounts the volume
2. The mounted volume path is then bind mounted to the namespace of the container

User doesn't have to make any mounts by themselves so /dev/sd? doesn't come into the picture at all. From checking other issues that reported the same behavior restarting docker is also advised to ensure that docker is in a sane state.

I'll do my tests locally to get a repro and let you know.

[govint]

Tried out on PhotonOS 2.0 (with hw version 13) and am not able to reproduce the issue:

1. [root@a6d5ac82ff77 mnt]# uname -a Linux a6d5ac82ff77 4.9.47-2.ph2-esx #1-photon SMP Thu Sep 28 23:33:40 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

2. Created a volume tesstvol and then ran a container with that mounted.

3. The volume is mounted as shown. /dev/sdb on /var/lib/docker/plugins/0aa861101e478a6996cf99c7539253f561920d8f9b08b05fcc1df2ba5cd6580f/rootfs/mnt/vmdk/testvol@sharedVmfs-0 type ext4 (rw,relatime,data=ordered) /dev/sdb on /var/lib/docker/plugins/0aa861101e478a6996cf99c7539253f561920d8f9b08b05fcc1df2ba5cd6580f/propagated-mount/testvol@sharedVmfs-0 type ext4 (rw,relatime,data=ordered) /dev/sdb on /var/lib/docker/plugins/0aa861101e478a6996cf99c7539253f561920d8f9b08b05fcc1df2ba5cd6580f/propagated-mount/testvol@sharedVmfs-0 type ext4 (rw,relatime,data=ordered)

4. Created files outside of the container as below, ls -l /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:1:0 lrwxrwxrwx 1 root root 9 Oct 16 09:43 /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:1:0 -> ../../sdb

cd /var/lib/docker/plugins/0aa861101e478a6996cf99c7539253f561920d8f9b08b05fcc1df2ba5cd6580f/propagated-mount/testvol@sharedVmfs-0/

ls lost+found

touch x touch y touch z

ls lost+found x y z

5. Inside the container, docker run --rm -it -v testvol:/mnt centos Unable to find image 'centos:latest' locally latest: Pulling from library/centos d9aaf4d82f24: Pull complete Digest: sha256:eba772bac22c86d7d6e72421b4700c3f894ab6e35475a34014ff8de74c10872e Status: Downloaded newer image for centos:latest

cd /mnt

ls -l total 12 drwx------ 2 root root 12288 Oct 16 09:43 lost+found -rw-r----- 1 root root 0 Oct 16 09:47 x -rw-r----- 1 root root 0 Oct 16 09:47 y -rw-r----- 1 root root 0 Oct 16 09:47 z

6. Volume is shown mounted inside the container as, /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:1:0 on /mnt type ext4 (rw,relatime,data=ordered)

I'm able to create and access files from both inside and outside the container. No issues at all.

@luizhpriotto, can you confirm the above steps repros the issue for you, I'm unable to see why the plugin won't work as expected. You could try with the new versions of PhotonOS available at https://github.com/vmware/photon/wiki/Downloading-Photon-OS

[luizhpriotto]

Thanks @govint I'll try Photon 2.0 when the stable version is released. thanks very much!

att

[shuklanirdesh82]

I'll try Photon 2.0 when the stable version is released.

@luizhpriotto photon 2.0 is out and available at https://github.com/vmware/photon/wiki/Downloading-Photon-OS#downloading-photon-os-20-ga

Can you please try out and update the issue?

[shuklanirdesh82]

closing this as per https://github.com/vmware/docker-volume-vsphere/issues/1931#issuecomment-336842253; please reopen if the issue is still reproducible.