docker volumes empty - Githubissues

christhomas commented 7 years ago

Hi,

I have a strange issue where a docker volume I am mounting using docker-compose is partially empty, it contains one folder, but nothing more, but I am unsure why that folder is special and why it displays and the others do not display at all.

Are there problems with docker volumes in general? should I go back to using virtualbox if I need this functionality?

I have this service defined in my docker-compose.yml:

version: "2"
services:
  test_nginx_php_lk:
    build: docker/nginx
    ports:
      - "10000:80"
    links:
      - test_php_fpm_lk:php
    volumes:
      - ./projects:/var/www

in my host operating system, if I do this: find ./projects, I get the following output:

./projects
./projects/hello
./projects/hello/index.php
./projects/testing
./projects/testing/testing.php

lets assume, I have no docker machines configured, so lets make one using the command:

docker-machine create somecompany.lk --driver xhyve

if I shell into it using the command:

docker-machine ssh somecompany.lk

then do the command a directory listing of the root I get the following output:

docker@boot2docker:~$ ls /
bin/     dev/     etc/     home/    init     lib/     lib64    linuxrc  mnt/     opt/     proc/    root/    run/     sbin/    sys/     tmp      usr/     var/

So, there is nothing mounted and no /Users folder, it will appear when I docker-compose up my service in a minute, but I just wanted to prove the machine is fresh and new.

Now in the terminal open in the docker-compose.yml folder, I execute the following two commands:

eval $(docker-machine env somecompany.lk)
docker-compose build --no-cache

and I get the following output:

Building test_php_fpm_lk
Step 1 : FROM php:fpm
fpm: Pulling from library/php
6a5a5368e0c2: Pull complete
de059afc6b5d: Pull complete
9b1ac0143753: Pull complete
c591c61adc6e: Pull complete
b9786831efdb: Pull complete
7d2a28039006: Pull complete
605d865c381c: Pull complete
8e5e9bfe8b98: Pull complete
cf2553e0b0e6: Pull complete
Digest: sha256:9ba138c418dc734a94200282aa80b262a79ac5d1d40c765bc1450ae07d485b4a
Status: Downloaded newer image for php:fpm
 ---> f18274aff56e
Step 2 : MAINTAINER some guy <some.guy@somecompany.com>
 ---> Running in 547dee4a97b1
 ---> 52db28e27906
Removing intermediate container 547dee4a97b1
Step 3 : EXPOSE 9000
 ---> Running in 2b71841505bc
 ---> dfe8b3cb0ec3
Removing intermediate container 2b71841505bc
Successfully built dfe8b3cb0ec3
Building test_nginx_php_lk
Step 1 : FROM nginx:latest
latest: Pulling from library/nginx
6a5a5368e0c2: Already exists
4aceccff346f: Pull complete
c8967f302193: Pull complete
Digest: sha256:1ebfe348d131e9657872de9881fe736612b2e8e1630e0508c354acb0350a4566
Status: Downloaded newer image for nginx:latest
 ---> ba6bed934df2
Step 2 : MAINTAINER some guy <some.guy@somecompany.com>
 ---> Running in b1689b07ca8c
 ---> d2808296f86b
Removing intermediate container b1689b07ca8c
Step 3 : RUN rm /etc/nginx/conf.d/default.conf
 ---> Running in 2099a3ecf34c
 ---> 24782ab28271
Removing intermediate container 2099a3ecf34c
Step 4 : ADD ./default.conf /etc/nginx/conf.d/
 ---> 0963becfa790
Removing intermediate container bab135870b62
Step 5 : VOLUME /var/www
 ---> Running in 50d03b539954
 ---> f85297ef0c9c
Removing intermediate container 50d03b539954
Step 6 : EXPOSE 80
 ---> Running in 3df3665a706f
 ---> 030d9e11820b
Removing intermediate container 3df3665a706f
Step 7 : EXPOSE 443
 ---> Running in 7c444845b383
 ---> cb5b7cbbf7b6
Removing intermediate container 7c444845b383
Successfully built cb5b7cbbf7b6
test_mysql_lk uses an image, skipping

Then I bring up the container I want, by running this command:

docker-compose up test_nginx_php_lk

And get the following output:

Creating network "somecompanystack_default" with the default driver
Creating somecompanystack_test_php_fpm_lk_1
Creating somecompanystack_test_nginx_php_lk_1
Attaching to somecompanystack_test_nginx_php_lk_1

But now, in a new terminal, if I shell into the docker-machine again, and list the root directory, I find that "Users" has been added, so lets do "docker-machine ssh somecompany.lk" and see "ls /" again

docker@boot2docker:~$ ls /
Users/   bin/     dev/     etc/     home/    init     lib/     lib64    linuxrc  mnt/     opt/     proc/    root/    run/     sbin/    sys/     tmp      usr/     var/

and if I do a find /Users, I get the following output:

docker@boot2docker:~$ find /Users
/Users
/Users/somecompany-20150103
/Users/somecompany-20150103/projects
/Users/somecompany-20150103/projects/somecompany-stack
/Users/somecompany-20150103/projects/somecompany-stack/projects

But now I am confused, since when I did the docker-compose up, it appears to have found the absolute path to the location I wanted to mount and this appears in the docker-machine, but where are all the contents of ./projects ? remember it should display a couple of directories, hello, test and two files, index.php and testing.php, but they are not there!

So ok, lets open a bash terminal into the container and see what is there in the /var/www folder, I can find the name using docker ps and using the last column in the row, in my case, its "somecompanystack_test_nginx_php_lk_1"

docker exec -it somecompanystack_test_nginx_php_lk_1 bash

and then we can find /var/www to see what files are there

root@609ac694e545:/# find /var/www/
/var/www/

Nothing, its empty, what about du -sh

root@609ac694e545:/# du -sh /var/www/
0   /var/www/

So it is literally empty and has no contents at all, which I guess should be expected because I suppose docker-machine has the mountpoint in its system and it maps it into the container, since in docker-machine it was empty, I suppose here it would be too, right?

So, if I create a file in this folder, will it appear in the docker-machine?

root@609ac694e545:/var/www# echo "hello world" > test.txt
root@609ac694e545:/var/www# ls
test.txt

yes it will, but does it appear in the correct location in the docker-machine

docker@boot2docker:~$ find /Users/
/Users/
/Users/somecompany-20150103
/Users/somecompany-20150103/projects
/Users/somecompany-20150103/projects/somecompany-stack
/Users/somecompany-20150103/projects/somecompany-stack/projects
/Users/somecompany-20150103/projects/somecompany-stack/projects/test.txt

yes it does.....but because this volume is supposed to be mapped from the host, I wonder if the file is present there?

bash$ find ./projects
./projects
./projects/hello
./projects/hello/index.php
./projects/testing
./projects/testing/testing.php

Nope, so this apparently is not working as expected, its got the right directory structure, but there are no contents even though they are mapped between them, yet nothing is coming from the host into the docker machine or container.

And just so you have all the information, here is the output from docker inspect:

bash$ docker inspect somecompanystack_test_nginx_php_lk_1
[
    {
        "Id": "609ac694e545da0fcc3863a807cb377f4d295e78f18782238a637245a14dd340",
        "Created": "2016-09-26T17:24:37.672173232Z",
        "Path": "nginx",
        "Args": [
            "-g",
            "daemon off;"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 2757,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2016-09-26T17:24:38.355524468Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:cb5b7cbbf7b621ed3b9c3f0adfd241ea264c24696eae58fe33a56288d726acdc",
        "ResolvConfPath": "/mnt/sda1/var/lib/docker/containers/609ac694e545da0fcc3863a807cb377f4d295e78f18782238a637245a14dd340/resolv.conf",
        "HostnamePath": "/mnt/sda1/var/lib/docker/containers/609ac694e545da0fcc3863a807cb377f4d295e78f18782238a637245a14dd340/hostname",
        "HostsPath": "/mnt/sda1/var/lib/docker/containers/609ac694e545da0fcc3863a807cb377f4d295e78f18782238a637245a14dd340/hosts",
        "LogPath": "/mnt/sda1/var/lib/docker/containers/609ac694e545da0fcc3863a807cb377f4d295e78f18782238a637245a14dd340/609ac694e545da0fcc3863a807cb377f4d295e78f18782238a637245a14dd340-json.log",
        "Name": "/somecompanystack_test_nginx_php_lk_1",
        "RestartCount": 0,
        "Driver": "aufs",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/Users/somecompany-20150103/projects/somecompany-stack/projects:/var/www:rw"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "somecompanystack_default",
            "PortBindings": {
                "80/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "10000"
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": [],
            "CapAdd": null,
            "CapDrop": null,
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": -1,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        },
        "GraphDriver": {
            "Name": "aufs",
            "Data": null
        },
        "Mounts": [
            {
                "Source": "/Users/somecompany-20150103/projects/somecompany-stack/projects",
                "Destination": "/var/www",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "609ac694e545",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "443/tcp": {},
                "80/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "NGINX_VERSION=1.11.4-1~jessie"
            ],
            "Cmd": [
                "nginx",
                "-g",
                "daemon off;"
            ],
            "Image": "somecompanystack_test_nginx_php_lk",
            "Volumes": {
                "/var/www": {}
            },
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {
                "com.docker.compose.config-hash": "ce4035e73f6da83089c757ab23a4d5e011ce88dbeb14f2bdb061c66eabf95906",
                "com.docker.compose.container-number": "1",
                "com.docker.compose.oneoff": "False",
                "com.docker.compose.project": "somecompanystack",
                "com.docker.compose.service": "test_nginx_php_lk",
                "com.docker.compose.version": "1.8.0"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "2653468e9838ee073dde0cb6e8d4c38e42cf29d37adade82242ace47e779081f",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "443/tcp": null,
                "80/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "10000"
                    }
                ]
            },
            "SandboxKey": "/var/run/docker/netns/2653468e9838",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "somecompanystack_default": {
                    "IPAMConfig": null,
                    "Links": [
                        "somecompanystack_test_php_fpm_lk_1:test_php_fpm_lk_1",
                        "somecompanystack_test_php_fpm_lk_1:somecompanystack_test_php_fpm_lk_1",
                        "somecompanystack_test_php_fpm_lk_1:php"
                    ],
                    "Aliases": [
                        "609ac694e545",
                        "test_nginx_php_lk"
                    ],
                    "NetworkID": "c5de3c27bb8351d2f12a776aee82aafad62a14c039becfa7bdbacb53791195a3",
                    "EndpointID": "74aaced2cc10f70a0ddab2639e1ea2b51f2e367eb7cb1b9bdac93e817b727f25",
                    "Gateway": "172.18.0.1",
                    "IPAddress": "172.18.0.3",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:12:00:03"
                }
            }
        }
    }
]

zchee commented 7 years ago

@christhomas Hi :)

Okay, now I'm little active to develop this project (sorry other issue users...) And develop that related the disk image. actually, implements qcow2 image format support with docker/hyperkit. So I also interested for the if disk i/o, or volume problems.

Your posted comment might be helpful to debug. Thanks for telling me much information.

Are there problems with docker volumes in general? should I go back to using virtualbox if I need this functionality?

I don't know whether it general. because I had to stop before development this project. but I think like it was a lot. xhyve-driver users posted many issues that disk image, shared folder and volume issue. Basically same layer. You will be seen that the multitude if you look at the other issue.(but I really don't see issue thread until recently. so I don't know it correct.)

Just an off topic. If you want to go back to virtualbox, I want to suggest the Docker for Mac. It almost the basic concept is the same as xhyve-driver. Maybe have the same issue, but faster than virtualbox and modern. There will be worth try. If you want to a stable, please use virtualbox.

Next, mainly topic. My debugging start point is I re-using the docker-compose. I do not use for a long time, because the bugs were many And I read your comment, there is also the possibility of other(not driver) causes. I will try your same environment and understand problem causes point.

Also, I want to know what setting that you did launch the xhyve. Could you post a config json such as ~/.docker/machine/machines/foo/config.json as much as possible? (cut the sensitive data If there)

Strech commented 7 years ago

@christhomas In case if you will enable --xhyve-experimental-nfs-share you can quickly check mounting by

$ docker run -v "$PWD:/pwd" -it ubuntu ls -l /pwd

And in case if you will see empty folder, this will mean that you have some issues with nfs daemon, and you can try to restart it, or destroy the machine (it will stop the daemon) and create new one (it will strat the daemon)

emad-elsaid commented 7 years ago

@christhomas I want to make sure of something, which OS are you using as I have the same problem and I suspect it's because I'm running on OSX Seirra.

christhomas commented 7 years ago

@blazeeboy: I am running Sierra yes @Strech I did not actually try it in the end, I will explain.

I spoke with the devops guy at my last company and they told me to get Docker for Mac, which does not use docker-machine anymore, which is still using the old boot2docker / virtual box combo.

So when I used docker from cask, or downloaded it from docker.com, I found that now volumes work, although one drawback is that I cannot build new machines, I am limited to a single machine it seems, so I cannot try out swarm.

otakustay commented 7 years ago

I'm having the same issue running Sierra, the xhyve docker-machine gives an empty dir for volumes while virtualbox ones work as expected (files in volumes)

docker, docker-machine and docker-machine-driver-xhyve are all installed from brew

Have tried to restart the docker machine but does not solve the problem, the volume folder is still empty

Strech commented 7 years ago

@otakustay sounds crazy, but if you will try to recreate created docker machine it should help, because probably for some reason you NFS daemon wasn't working correctly.

Can you try to rm machine and create new one?

otakustay commented 7 years ago

I'm new to docker (2 days total) so I tried to fresh install every environment to reproduce this issue:

Create environment

brew install docker docker-machine-driver-xhyve
sudo chown root:wheel $(brew --prefix)/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
sudo chmod u+s $(brew --prefix)/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
docker-machine create -d xhyve default
eval $(docker-machine env default)

Ensure docker env is correct:

~/Dev » env | grep -i docker

DOCKER_TLS_VERIFY=1
DOCKER_HOST=tcp://192.168.64.5:2376
DOCKER_CERT_PATH=/Users/otakustay/.docker/machine/machines/default
DOCKER_MACHINE_NAME=default

Make a docker image

cd ~/Dev
mkdir docker-volume
cd docker-volume

docker build -t docker-volume .

The Dockerfile contains simple instructions:

FROM nginx

RUN mkdir -p /usr/app

Run and test

docker run --name docker-volume -v /Users/otakustay/Dev/docker-volume/:/usr/app -d docker-volume

docker exec -it docker-volume ls /usr/app

This outputs nothing (empty dir)

touch foo.txt

docker exec -it docker-volume ls /usr/app

Still nothing.

Docker for Mac and -d virtualbox both work as expected (have files in volume)

I don't have any other module (like docker-machine-nfs) installed, could it be the cause of this issue?

haim commented 7 years ago

Any update about how fix this issue ?

haim commented 7 years ago

Hi , @christhomas the problem of empty folder in Firewall. Just For test, turn off Firewall and restart docker-machine and docker-composer and you will see ;) As this NFS you should have open port for correct executable.

P.S.>let me know if this fix problem. Thanks

christhomas commented 7 years ago

I actually stopped using the project when I started to use docker for mac, sorry I can't help you more with this

dgdosen commented 7 years ago

FYI - I was having this same issue, but using the --xhyve-experimental-nfs-share flag when generating the docker machine fixed the issue...

iBobik commented 6 years ago

I had the same issue, but then I figured out that in default settings sharing is disabled:

$ docker-machine inspect local
…
        "Virtio9p": false,
        "Virtio9pFolder": "/Users",
…

Just change Virtio9p to true and docker-machine restart

xiaket commented 5 years ago

Run into this issue today and as mentioned above, the solution is:

Stop your vm: docker-machine inspect ${machine_name}
Go to ~/.docker/machine/machines/${machine_name}, modify config.json, change that Virtio9p setting to true.
Start your vm: docker-machine inspect ${machine_name}

After these, you should see the flag is changed by running docker-machine inspect ${machine_name}

To many people, the ${machine_name} mentioned here should be default

Thanks @iBobik for this solution, you've saved my day! :D

c4lliope commented 5 years ago

Is this something that we can make a default? Is it something that we would want to be a default?

I'm new to VirtIO, but it seems like a standard that has good support.

If someone can point out where the default config.json template is, I can take a look at a pull request.

A search through this repository only returned one file, xhyve/xhyve.go.

odeann commented 4 years ago

Try to change NFSSharesRoot to / in config.json:

{
...
        "NFSShares": [
            "/private/var/srv"
        ],
        "NFSSharesRoot": "/",
        "Virtio9p": null,
        "Virtio9pRoot": "/xhyve-virtio9p",
        "NFSShare": true,
...
}

Also, at some moment i've changed NFSShare to true. May be it is important as well.

P.S. Be sure, you are not mounting folders in NFSShares that are already exist in docker machine image.

machine-drivers / docker-machine-driver-xhyve

docker volumes empty #136

Create environment

Make a docker image

Run and test