search

rancher / convoy

A Docker volume plugin, managing persistent container volumes.
Apache License 2.0
1.31k stars 135 forks source link

Problem with convoy list / convoy inspect <VOL> #63

Closed pwFoo closed 8 years ago

pwFoo commented 8 years ago

Hi,

a use convoy inside a system-docker container and it seems to be working fine.

Volume is created:

drwxr-xr-x    3 root     root          4096 Jan 23 09:59 htdocs

With data, also updated / later created files are there

ls -lh _data/public_html/
total 12
-rw-r--r--    1 root     root         256 Jan 23 11:24 db.php
-rw-r--r--    1 root     root          15 Jan 23 10:07 index.html
-rw-r--r--    1 root     root          30 Jan 23 10:41 index.php

But...

convoy list

{}

convoy inspect htdocs

time="2016-01-23T11:39:31Z" level=error msg="Cannot find volume with name or id htdocs" 
{
    "Error": "Cannot find volume with name or id htdocs"
}
pwFoo commented 8 years ago

After RancherOS reboot convoy starts, but all containers with convoy volume failed to start. Maybe related...

1/24/2016 8:03:25 AM---
1/24/2016 8:03:25 AM2: StartInitialization
1/24/2016 8:03:25 AMPackage: github.com/opencontainers/runc/libcontainer.(*LinuxFactory)
1/24/2016 8:03:25 AMFile: factory_linux.go@242
1/24/2016 8:03:25 AM---
1/24/2016 8:03:25 AM3: initializer
1/24/2016 8:03:25 AMPackage: github.com/docker/docker/daemon/execdriver/native
1/24/2016 8:03:25 AMFile: init.go@35
1/24/2016 8:03:25 AM---
1/24/2016 8:03:25 AM4: Init
1/24/2016 8:03:25 AMPackage: github.com/docker/docker/pkg/reexec
1/24/2016 8:03:25 AMFile: reexec.go@26
1/24/2016 8:03:25 AM---
1/24/2016 8:03:25 AM5: main
1/24/2016 8:03:25 AMPackage: main
1/24/2016 8:03:25 AMFile: docker.go@18
1/24/2016 8:03:25 AM---
1/24/2016 8:03:25 AM6: main
1/24/2016 8:03:25 AMPackage: runtime
1/24/2016 8:03:25 AMFile: proc.go@63
1/24/2016 8:03:25 AM---
1/24/2016 8:03:25 AM7: goexit
1/24/2016 8:03:25 AMPackage: runtime
1/24/2016 8:03:25 AMFile: asm_amd64.s@2232
1/24/2016 8:03:43 AMTimestamp: 2016-01-24 08:03:43.058317466 +0000 UTC
1/24/2016 8:03:43 AMCode: System error
1/24/2016 8:03:43 AM
1/24/2016 8:03:43 AMMessage: stat /var/lib/docker/volumes/convoy_volume/_data: no such file or directory
1/24/2016 8:03:43 AM
1/24/2016 8:03:43 AMFrames:
1/24/2016 8:03:43 AM---
1/24/2016 8:03:43 AM0: setupRootfs
1/24/2016 8:03:43 AMPackage: github.com/opencontainers/runc/libcontainer
1/24/2016 8:03:43 AMFile: rootfs_linux.go@40
1/24/2016 8:03:43 AM---
1/24/2016 8:03:43 AM1: Init
1/24/2016 8:03:43 AMPackage: github.com/opencontainers/runc/libcontainer.(*linuxStandardInit)
1/24/2016 8:03:43 AMFile: standard_init_linux.go@57
1/24/2016 8:03:43 AM---
1/24/2016 8:03:43 AM2: StartInitialization
1/24/2016 8:03:43 AMPackage: github.com/opencontainers/runc/libcontainer.(*LinuxFactory)
1/24/2016 8:03:43 AMFile: factory_linux.go@242
1/24/2016 8:03:43 AM---
1/24/2016 8:03:43 AM3: initializer
1/24/2016 8:03:43 AMPackage: github.com/docker/docker/daemon/execdriver/native
1/24/2016 8:03:43 AMFile: init.go@35
1/24/2016 8:03:43 AM---
1/24/2016 8:03:43 AM4: Init
1/24/2016 8:03:43 AMPackage: github.com/docker/docker/pkg/reexec
1/24/2016 8:03:43 AMFile: reexec.go@26
1/24/2016 8:03:43 AM---
1/24/2016 8:03:43 AM5: main
1/24/2016 8:03:43 AMPackage: main
1/24/2016 8:03:43 AMFile: docker.go@18
1/24/2016 8:03:43 AM---
1/24/2016 8:03:43 AM6: main
1/24/2016 8:03:43 AMPackage: runtime
1/24/2016 8:03:43 AMFile: proc.go@63
1/24/2016 8:03:43 AM---
1/24/2016 8:03:43 AM7: goexit
1/24/2016 8:03:43 AMPackage: runtime
1/24/2016 8:03:43 AMFile: asm_amd64.s@2232
1/24/2016 8:04:24 AMTimestamp: 2016-01-24 08:04:24.247722641 +0000 UTC
1/24/2016 8:04:24 AMCode: System error
1/24/2016 8:04:24 AM
1/24/2016 8:04:24 AMMessage: stat /var/lib/docker/volumes/convoy_volume/_data: no such file or directory
1/24/2016 8:04:24 AM
1/24/2016 8:04:24 AMFrames:
1/24/2016 8:04:24 AM---
1/24/2016 8:04:24 AM0: setupRootfs
1/24/2016 8:04:24 AMPackage: github.com/opencontainers/runc/libcontainer
1/24/2016 8:04:24 AMFile: rootfs_linux.go@40
1/24/2016 8:04:24 AM---
1/24/2016 8:04:24 AM1: Init
1/24/2016 8:04:24 AMPackage: github.com/opencontainers/runc/libcontainer.(*linuxStandardInit)
1/24/2016 8:04:24 AMFile: standard_init_linux.go@57
1/24/2016 8:04:24 AM---
1/24/2016 8:04:24 AM2: StartInitialization
1/24/2016 8:04:24 AMPackage: github.com/opencontainers/runc/libcontainer.(*LinuxFactory)
1/24/2016 8:04:24 AMFile: factory_linux.go@242
1/24/2016 8:04:24 AM---
1/24/2016 8:04:24 AM3: initializer
1/24/2016 8:04:24 AMPackage: github.com/docker/docker/daemon/execdriver/native
1/24/2016 8:04:24 AMFile: init.go@35
1/24/2016 8:04:24 AM---
1/24/2016 8:04:24 AM4: Init
1/24/2016 8:04:24 AMPackage: github.com/docker/docker/pkg/reexec
1/24/2016 8:04:24 AMFile: reexec.go@26
1/24/2016 8:04:24 AM---
1/24/2016 8:04:24 AM5: main
1/24/2016 8:04:24 AMPackage: main
1/24/2016 8:04:24 AMFile: docker.go@18
1/24/2016 8:04:24 AM---
1/24/2016 8:04:24 AM6: main
1/24/2016 8:04:24 AMPackage: runtime
1/24/2016 8:04:24 AMFile: proc.go@63
1/24/2016 8:04:24 AM---
1/24/2016 8:04:24 AM7: goexit
1/24/2016 8:04:24 AMPackage: runtime
1/24/2016 8:04:24 AMFile: asm_amd64.s@2232
1/24/2016 8:05:27 AMCode: System error
1/24/2016 8:05:27 AM
1/24/2016 8:05:27 AMMessage: stat /var/lib/docker/volumes/convoy_volume/_data: no such file or directory
ls /var/lib/docker/volumes/convoy_volume/     
logs         public_html
pwFoo commented 8 years ago

After reboot volumes all of the type DRIVER local (docker volume ls)?

If I delete all volumes with docker volume rm and recreate the stack there is a volume with DRIVER convoy and it's working again.

Why it loses state after RancherOS reboot? But also the new created volume isn't displayed with convoy list.

yasker commented 8 years ago

Hi @pwFoo

How did you configure Convoy in this environment?

pwFoo commented 8 years ago

dockerfile:

FROM rancher/os-console:v0.4.2

ADD files/* /tmp/

#ADD convoy.tar.gz /tmp/convoy.tar.gz
#ADD convoy-start.sh /convoy-start.sh

RUN cp /tmp/convoy.tar.gz/convoy/convoy* /usr/bin/ \
    && rm -rf /tmp/convoy.tar.gz \
    && mv /tmp/convoy-start.sh /convoy-start.sh \
    && chmod +x /convoy-start.sh

ENTRYPOINT ["/convoy-start.sh"]

/convoy-start.sh

#!/bin/sh

DOCKER_VOLUMES=/var/lib/docker/volumes
DOCKER_PLUGINS=/etc/docker/plugins

if [ -z $@ ]; then
    if [ ! -d $DOCKER_PLUGINS ]; then
        mkdir -p $DOCKER_PLUGINS
    fi

    echo "unix:///var/run/convoy/convoy.sock" > $DOCKER_PLUGINS/convoy.spec

    CMD="daemon --drivers vfs --driver-opts vfs.path=$DOCKER_VOLUMES"
else
    CMD=$@
fi

exec /usr/bin/convoy $CMD

Created volumes work fine, but no list / inspect and after I rebooted the RancherOS, volumes doesn't work anymore. Could it be a problem that the convoy volume directory is /var/lib/volumes?

Have you build a convoy image / container which works? I would use it as RancherOS os-service.

yasker commented 8 years ago

I don't think RancherOS preserve /var/lib/convoy, you can change Convoy's root by convoy daemon --root "/var/lib/rancher/convoy", this may fix the restart issue.

Also the volume path you specified is used by Docker as well, so docker would complain it's not a standand Docker volume. You probably want to choose something like /var/lib/rancher/convoy-volumes.

Though I am not sure what's happened with list, probably a permission issue?

pwFoo commented 8 years ago

Thanks!

After changing volume dir and root I can inspect created volumes: convoy inspect test123

{
    "UUID": "f2a12ac4-19f5-42c0-9d92-6c8194e0e76e",
    "Name": "test123",
    "Driver": "vfs",
    "MountPoint": "",
    "CreatedTime": "Mon Jan 25 20:10:10 +0000 2016",
    "DriverInfo": {
        "Driver": "vfs",
        "MountPoint": "",
        "Path": "/var/lib/rancher/convoy-volumes/test123",
        "PrepareForVM": "false",
        "Size": "0"
    },
    "Snapshots": {}
}

But convoy list is empty:

{}

Looks better now... After reboot volume created by rancher-compose works fine, but one created by convoy create isn't available. But could be another problem. I'll do some more tests!

pwFoo commented 8 years ago

Hi @yasker data is persistent and also reboot save after the changes are done, but list, backup and snapshot features seems broken.

_sudo ls -lh /var/lib/rancher/ _

drwxr-xr-x    3 root     root        4.0K Jan 30 11:09 conf
drwxr-xr-x    3 root     root        4.0K Jan 30 11:09 convoy
drwx------    5 root     root        4.0K Jan 26 18:44 convoy-volumes
drwxr-xr-x    3 root     root        4.0K Jan  3 08:29 state

sudo ls -lh /var/lib/rancher/convoy

-rw-r--r--    1 root     root         100 Jan 30 11:09 convoy.cfg
-rw-r--r--    1 root     root           0 Jan 30 11:09 lock
drwx------    2 root     root        4.0K Jan 30 11:09 vfs
-rw-r--r--    1 root     root         205 Jan 26 16:57 volume_11e4ac37-4a7a-4c49-a8cf-57f14b5d8137.json
-rw-r--r--    1 root     root         205 Jan 29 15:57 volume_3a8ff658-37c7-42eb-9091-3a8f77a5e171.json
-rw-r--r--    1 root     root         205 Jan 29 11:29 volume_4b4ce294-d879-472f-aa8c-75fd06c70ac2.json
-rw-r--r--    1 root     root         205 Jan 25 20:50 volume_50e5fbd6-3526-4255-9cbf-73940f10544d.json
-rw-r--r--    1 root     root         205 Jan 29 17:56 volume_6ebe001f-11d3-4702-9226-c2d750325d0a.json
-rw-r--r--    1 root     root         205 Jan 29 11:16 volume_7501b73d-854f-4195-8a48-d1d8f8f95d7f.json
-rw-r--r--    1 root     root         205 Jan 29 11:12 volume_8d324e53-c798-496f-a512-261ddb42d817.json
-rw-r--r--    1 root     root         205 Jan 29 15:21 volume_ab681ee3-43dd-4768-a84b-604259519528.json
-rw-r--r--    1 root     root         205 Jan 29 17:46 volume_af7ab459-9da8-4a43-bc18-909e1e35e8c8.json
-rw-r--r--    1 root     root         205 Jan 29 17:51 volume_b4114334-597b-4445-b483-87735713ac93.json
-rw-r--r--    1 root     root         205 Jan 29 18:11 volume_c45dac37-af6b-4005-b453-688331f96c12.json
-rw-r--r--    1 root     root         205 Jan 30 11:09 volume_cff9ad9f-2bd1-4cda-a4a9-6c36461712a8.json
-rw-r--r--    1 root     root         205 Jan 29 12:52 volume_e168d7c2-3cab-47fb-8dee-8baacb83bc58.json
-rw-r--r--    1 root     root         205 Jan 29 19:36 volume_ee2ae571-7867-44b8-800b-4c357414dd67.json
-rw-r--r--    1 root     root         187 Jan 25 20:10 volume_f2a12ac4-19f5-42c0-9d92-6c8194e0e76e.json
-rw-r--r--    1 root     root         205 Jan 25 20:33 volume_faca0692-ce7e-4caa-9f6a-86c4f3020bd9.json

sudo ls -lh /var/lib/rancher/convoy-volumes/

drwx------    2 root     root        4.0K Jan 25 20:10 test123
drwx------    5 65534    65534       4.0K Jan 25 20:33 webkreativ_website_htdocs

convoy instect works fine. Also the volume data is mounted to the container(s), but list is empty.

Any idea what could be wrong?

pwFoo commented 8 years ago

Moved image to alpine:3.3, replaced the volumes-from with -v <HOSTDIR>:<CONTAINERDIR> and some other small changes and issue is gone!

So it was a bug with my image as suspected.

dcrystalj commented 7 years ago

I am using ubuntu as base image. have same issue.

convoy list shows {} convoy create a:

ERRO[0000] Error response from server, Volume a already exists 

{
        "Error": "Error response from server, Volume a already exists \n"
}

I am using convoy v.0.5.0