search

wetopi / docker-volume-rbd

Docker Engine managed plugin to manage RBD volumes.
MIT License
69 stars 18 forks source link

Unable to use custom pool name #7

Closed r3pek closed 5 years ago

r3pek commented 5 years ago

I've been using your plugin for some time now without issues with the "rbd" pool as default. Now, I renamed my pool to rbd_docker and while I can mount the devices just fine on the host, the plugin doesn't seem to detect the pool name and is unable to mount any device. I tried passing RBD_CONF_POOL on the install of the plugin and also specifying -o pool=rbd_docker and the volume create command but nothing worked. Just as example:

# docker volume create -d wetopi/rbd -o size=20 my_rbd_volume
Error response from daemon: create my_rbd_volume: VolumeDriver.Create: volume-rbd Name=unable to create ceph connection to cluster(ceph) with user(): rados: Invalid argument Request=Create Message=unable to connect to rbd pool
# rbd showmapped
id pool       image snap device    
0  rbd_docker mysql -    /dev/rbd0
r3pek commented 5 years ago

OK, problem might be something else as now, reverting back to the default "rbd" pool, it doesn't work either with the same error.

r3pek commented 5 years ago

It started listing the devices (docker volume ls | grep rbd) after I forced RBD_CONF_KEYRING_USER=client.admin. Without it, it couldn't even find the volumes. Now the volumes show up but can't be used:

Feb 03 18:55:11 docker dockerd[3507]: time="2019-02-03T18:55:11Z" level=error msg="time=\"2019-02-03T18:55:11Z\" level=debug msg=\"volume-rbd Name=mysql Message=mount -o noatime /dev/rbd/rbd_servers/mysql /mnt/volumes/mysql\"" plugin=8638c2cba45c90375b0c2fa2d8ace99dd3df9bad762cf807f17b30341861f538
Feb 03 18:55:11 docker dockerd[3507]: time="2019-02-03T18:55:11Z" level=error msg="time=\"2019-02-03T18:55:11Z\" level=debug msg=\"volume-rbd Name=mysql Message=free up image\"" plugin=8638c2cba45c90375b0c2fa2d8ace99dd3df9bad762cf807f17b30341861f538
Feb 03 18:55:11 docker dockerd[3507]: time="2019-02-03T18:55:11Z" level=error msg="time=\"2019-02-03T18:55:11Z\" level=debug msg=\"volume-rbd Message=umount /mnt/volumes/mysql\"" plugin=8638c2cba45c90375b0c2fa2d8ace99dd3df9bad762cf807f17b30341861f538
Feb 03 18:55:11 docker dockerd[3507]: time="2019-02-03T18:55:11Z" level=error msg="time=\"2019-02-03T18:55:11Z\" level=warning msg=\"volume-rbd Name=mysql Message=unable to unmount:%!(EXTRA syscall.Errno=invalid argument)\"" plugin=8638c2cba45c90375b0c2fa2d8ace99dd3df9bad762cf807f17b30341861f538

creating also fails:

# docker volume create -d wetopi/rbd -o size=20 my_rbd_volume
Error response from daemon: create my_rbd_volume: VolumeDriver.Create: volume-rbd Name=my_rbd_volume Request=Create Message=unable to create ceph rbd image: exit status 6
r3pek commented 5 years ago

Just a bit more info. I installed version 1.0.1 of the plugin and it worked right of the bat, didn't even had to specify RBD_CONF_KEYRING_USER. Restarted the docker service to try again, bam, doesn't work again with the same error.

sitamet commented 5 years ago

I haven't experienced this exact behaviour.... but kind of (eg during upgrades config.json wasn't upgrading) ... My experience with docker plugin upgrades is that the process is prone to fail (not sure if this affirmation is true with latest releases...) My process to upgrade/update the plugin is: 1.- remove the plugin 2.- restart docker service 3.- install docker plugin

box-daxter commented 5 years ago

@r3pek Contact me boxdaxter@gmail.com. I will try to help you with this issue.