Unable to mount rbd volume

[redcodex]

Hi,

I'm trying to use the latest version (2.0.0 it seems) but I'm having trouble with mounting rbd volumes. The creation of a test volume was successful and it is shown with "docker volume ls". But when I try to start a container with "docker run ... -v : --volume-driver=wetopi/rbd ...." I'm getting the following error:

docker: Error response from daemon: VolumeDriver.Mount: volume-rbd Name= Request=Mount Message= unable to mount: exit status 1.

I've enabled debug logging and got the following results (shortened):

volume-rbd Name=<volume> Request=Mount
volume-rbd Name=<volume> Message=MountRbdImage map and mount
volume-rbd Name=<volume> Message=rbd map
volume-rbd Name=<volume> Message=mount -o noatime /dev/rbd/<pool>/<volume> /mnt/volumes/<volume>
volume-rbd Name=<volume> Message=free up image
volume-rbd Message=umount /mnt/volumes/<volume>
volume-rbd Name=<volume> Message=unable to unmount:%!(EXTRA syscall.Errno=invalid argument)
volume-rbd Name=<volume> Message=rbd unmap
Handler for POST /v1.38/containers/create returned error: VolumeDriver.Mount: volume-rbd Name=<volume> Request=Mount Message= unable to mount: exit status 1

So mapping the volume seems to work (got some libceph & rbd output in kernel log) but the mount command doesn't seem to work. The unmount seems to fail since the volume was never mounted (at least man 2 mount says so for EINVAL). I had a look at the source code and logged into the volume container with docker-runc and tried to reproduce all the steps above on the command line:

mkdir /mnt/volumes/<volume> && chmod 755 /mnt/volumes/<volume>
rbd --pool <pool> --name <keyring_user> map <volume> (works)
ls /dev/rbd/<pool>/<volume> (exists)
mount -o noatime /dev/rbd/<pool>/<volume> /mnt/volumes/<volume> (works)
echo $? (result: 0)
mount (shows the mountpoint)
cd /mnt/volumes/<volume> && ls (works)

Since I'm essentially doing the same as the go program I'm a bit puzzled why the manual execution on the command line works and the program does not. I haven't debugged any go programs yet and before I delve into that maybe you have some other things that I could try. Here is some additional info about the system:

Host OS: Debian 9.5 (stretch) Docker Version: Docker version 18.06.1-ce Ceph Version on the cluster nodes: ceph version 12.2.2 Interesting Ceph options in /etc/ceph/ceph.conf: rbd default features = 3

Edit:

Maybe the problem is just a race condition between udev creating the symlink and mounting the path. Even if this doesn't fix my problem I guess it would be a good idea to wait for the availability of the symlink before trying to mount it. Polling every 200ms for 5s or so should be enough. Or even better just use the value returned by the rbd map command, which doesn't rely on udev.

[sitamet]

Hi, I had this same problem. Defining the MOUNT_OPTIONS in short syntax eg '-o noatime' does not work. I solved this passing "--options=noatime"

Check that your plugin loaded the new config.json the last push to master fixed this default var MOUNT_OPTIONS:

    {
      "name": "MOUNT_OPTIONS",
      "Description": "Options to pass to the mount command",
      "settable": [
        "value"
      ],
      "value": "--options=noatime"
    }

[redcodex]

That fixed it, thanks! Great work on the plugin!