search

wetopi / docker-volume-rbd

Docker Engine managed plugin to manage RBD volumes.
MIT License
72 stars 18 forks source link

Can connect with plugin, but it can't create or map images #26

Closed AlexanderShaw closed 7 months ago

AlexanderShaw commented 7 months ago

I'm trying to create an rbd image using this plugin for a container running on a client host, outside my ceph cluster.

root@compute-1:/ # docker volume create -d rbd test_rbd 
Error response from daemon: create test_rbd: VolumeDriver.Create: volume-rbd Name=test_rbd Request=Create Message=unable to create ceph rbd image: exit status 2

Interestingly, if I make the image using ceph directly, this command succeeds, but then I cannot mount it:

root@compute-1:/ # rbd create --size 100 docker/test_rbd --id compute
root@compute-1:/ # rbd ls --pool docker --id compute
test_rbd
root@compute-1:/ # docker volume create -d rbd test_rbd
test_rbd
root@compute-1:/ # docker volume ls
DRIVER          VOLUME NAME
rbd:latest      test_rbd
root@compute-1:/ # docker run -it -v test_rbd:/mount busybox sh
docker: Error response from daemon: error while mounting volume '/mnt/volumes/test_rbd': VolumeDriver.Mount: volume-rbd Name=test_rbd Request=Mount Message= unable to map: test_rbd%!(EXTRA *exec.ExitError=exit status 2).

I've hit a roadblock with these error messages, and could really use some help. It seems like my plugin configuration is off, but I'm not sure how, since it seems by the log that the connection is being made to my ceph cluster by the plugin. I've tried allowing all mgr, mon and osd privileges to the client's ceph user, but that didn't help.

Running ceph version 18.2.0 and Docker version 24.0.5 on Ubuntu 23.10 x64

Here are the logs of both errors:

error 1

Feb 27 06:55:44 compute-1 dockerd[2774]: time="2024-02-27T06:55:44Z" level=error msg="time=\"2024-02-27T06:55:44Z\" level=debug msg=\"volume-rbd Message=connect to ceph pool(docker)\"" plugin=2ac89e6ce2c6643b23ab33da5d0fb2e42d55af0619312b10a33ea5f2310eb92d
Feb 27 06:55:44 compute-1 dockerd[2774]: time="2024-02-27T06:55:44Z" level=error msg="time=\"2024-02-27T06:55:44Z\" level=debug msg=\"volume-rbd Name=test_rbd Message=checking if exists rbd image in pool(docker)\"" plugin=2ac89e6ce2c6643b23ab33da5d0fb2e42d55af0619312b10a33ea5f2310eb92d
Feb 27 06:55:44 compute-1 dockerd[2774]: time="2024-02-27T06:55:44Z" level=error msg="time=\"2024-02-27T06:55:44Z\" level=debug msg=\"volume-rbd Message=connection shutdown from ceph\"" plugin=2ac89e6ce2c6643b23ab33da5d0fb2e42d55af0619312b10a33ea5f2310eb92d
Feb 27 06:55:44 compute-1 dockerd[2774]: time="2024-02-27T06:55:44Z" level=error msg="2024/02/27 06:55:44 Entering go-plugins-helpers createPath" plugin=2ac89e6ce2c6643b23ab33da5d0fb2e42d55af0619312b10a33ea5f2310eb92d
Feb 27 06:55:44 compute-1 dockerd[2774]: time="2024-02-27T06:55:44Z" level=error msg="time=\"2024-02-27T06:55:44Z\" level=info msg=\"volume-rbd Name=test_rbd Request=Create\"" plugin=2ac89e6ce2c6643b23ab33da5d0fb2e42d55af0619312b10a33ea5f2310eb92d
Feb 27 06:55:44 compute-1 dockerd[2774]: time="2024-02-27T06:55:44Z" level=error msg="time=\"2024-02-27T06:55:44Z\" level=debug msg=\"volume-rbd Message=connect to ceph pool(docker)\"" plugin=2ac89e6ce2c6643b23ab33da5d0fb2e42d55af0619312b10a33ea5f2310eb92d
Feb 27 06:55:44 compute-1 dockerd[2774]: time="2024-02-27T06:55:44Z" level=error msg="time=\"2024-02-27T06:55:44Z\" level=debug msg=\"volume-rbd Name=test_rbd Message=checking if exists rbd image in pool(docker)\"" plugin=2ac89e6ce2c6643b23ab33da5d0fb2e42d55af0619312b10a33ea5f2310eb92d
Feb 27 06:55:44 compute-1 dockerd[2774]: time="2024-02-27T06:55:44Z" level=error msg="time=\"2024-02-27T06:55:44Z\" level=debug msg=\"volume-rbd Name=test_rbd Message=create image in pool(docker) with size(512MB) and fstype(ext4)\"" plugin=2ac89e6ce2c6643b23ab33da5d0fb2e42d55af0619312b10a33ea5f2310eb92d
Feb 27 06:55:44 compute-1 dockerd[2774]: time="2024-02-27T06:55:44Z" level=error msg="time=\"2024-02-27T06:55:44Z\" level=debug msg=\"volume-rbd Name=test_rbd Message=rbd map\"" plugin=2ac89e6ce2c6643b23ab33da5d0fb2e42d55af0619312b10a33ea5f2310eb92d
Feb 27 06:55:44 compute-1 dockerd[2774]: time="2024-02-27T06:55:44Z" level=error msg="time=\"2024-02-27T06:55:44Z\" level=debug msg=\"volume-rbd Name=test_rbd Message=remove rbd image\"" plugin=2ac89e6ce2c6643b23ab33da5d0fb2e42d55af0619312b10a33ea5f2310eb92d
Feb 27 06:55:45 compute-1 dockerd[2774]: time="2024-02-27T06:55:45Z" level=error msg="time=\"2024-02-27T06:55:45Z\" level=debug msg=\"volume-rbd Message=connection shutdown from ceph\"" plugin=2ac89e6ce2c6643b23ab33da5d0fb2e42d55af0619312b10a33ea5f2310eb92d
Feb 27 06:55:45 compute-1 dockerd[2774]: time="2024-02-27T06:55:45Z" level=error msg="2024/02/27 06:55:45 Entering go-plugins-helpers capabilitiesPath" plugin=2ac89e6ce2c6643b23ab33da5d0fb2e42d55af0619312b10a33ea5f2310eb92d
Feb 27 06:55:45 compute-1 dockerd[2774]: time="2024-02-27T06:55:45Z" level=error msg="time=\"2024-02-27T06:55:45Z\" level=info msg=\"volume-rbd Request=Capabilities\"" plugin=2ac89e6ce2c6643b23ab33da5d0fb2e42d55af0619312b10a33ea5f2310eb92d
Feb 27 06:55:45 compute-1 dockerd[2774]: time="2024-02-27T06:55:45.034023524Z" level=error msg="Handler for POST /v1.43/volumes/create returned error: create test_rbd: VolumeDriver.Create: volume-rbd Name=test_rbd Request=Create Message=unable to create ceph rbd image: exit status 2"

error 2

Feb 27 06:52:49 compute-1 dockerd[2774]: time="2024-02-27T06:52:49Z" level=error msg="time=\"2024-02-27T06:52:49Z\" level=debug msg=\"volume-rbd Message=connect to ceph pool(docker)\"" plugin=2ac89e6ce2c6643b23ab33da5d0fb2e42d55af0619312b10a33ea5f2310eb92d
Feb 27 06:52:49 compute-1 dockerd[2774]: time="2024-02-27T06:52:49Z" level=error msg="time=\"2024-02-27T06:52:49Z\" level=debug msg=\"volume-rbd Name=test_rbd Message=checking if exists rbd image in pool(docker)\"" plugin=2ac89e6ce2c6643b23ab33da5d0fb2e42d55af0619312b10a33ea5f2310eb92d
Feb 27 06:52:50 compute-1 dockerd[2774]: time="2024-02-27T06:52:50Z" level=error msg="time=\"2024-02-27T06:52:50Z\" level=debug msg=\"volume-rbd Message=connection shutdown from ceph\"" plugin=2ac89e6ce2c6643b23ab33da5d0fb2e42d55af0619312b10a33ea5f2310eb92d
Feb 27 06:52:50 compute-1 dockerd[2774]: time="2024-02-27T06:52:50Z" level=error msg="2024/02/27 06:52:50 Entering go-plugins-helpers capabilitiesPath" plugin=2ac89e6ce2c6643b23ab33da5d0fb2e42d55af0619312b10a33ea5f2310eb92d
Feb 27 06:52:50 compute-1 dockerd[2774]: time="2024-02-27T06:52:50Z" level=error msg="time=\"2024-02-27T06:52:50Z\" level=info msg=\"volume-rbd Request=Capabilities\"" plugin=2ac89e6ce2c6643b23ab33da5d0fb2e42d55af0619312b10a33ea5f2310eb92d
Feb 27 06:52:50 compute-1 dockerd[2774]: time="2024-02-27T06:52:50Z" level=error msg="2024/02/27 06:52:50 Entering go-plugins-helpers mountPath" plugin=2ac89e6ce2c6643b23ab33da5d0fb2e42d55af0619312b10a33ea5f2310eb92d
Feb 27 06:52:50 compute-1 dockerd[2774]: time="2024-02-27T06:52:50Z" level=error msg="time=\"2024-02-27T06:52:50Z\" level=info msg=\"volume-rbd Name=test_rbd Request=Mount\"" plugin=2ac89e6ce2c6643b23ab33da5d0fb2e42d55af0619312b10a33ea5f2310eb92d
Feb 27 06:52:50 compute-1 dockerd[2774]: time="2024-02-27T06:52:50Z" level=error msg="time=\"2024-02-27T06:52:50Z\" level=debug msg=\"volume-rbd Name=test_rbd Message=MountRbdImage map and mount\"" plugin=2ac89e6ce2c6643b23ab33da5d0fb2e42d55af0619312b10a33ea5f2310eb92d
Feb 27 06:52:50 compute-1 dockerd[2774]: time="2024-02-27T06:52:50Z" level=error msg="time=\"2024-02-27T06:52:50Z\" level=debug msg=\"volume-rbd Name=test_rbd Message=rbd map\"" plugin=2ac89e6ce2c6643b23ab33da5d0fb2e42d55af0619312b10a33ea5f2310eb92d
Feb 27 06:52:50 compute-1 dockerd[2774]: time="2024-02-27T06:52:50.526313129Z" level=error msg="Handler for POST /v1.43/containers/3d9450ade204abb6b62bdd06fabf8a8dcb7b3ee3f9ac219d7cc1ac9d5dbafa45/start returned error: error while mounting volume '/var/lib/docker/plugins/2ac89e6ce2c6643b23ab33da5d0fb2e42d55af0619312b10a33ea5f2310eb92d/rootfs': VolumeDriver.Mount: volume-rbd Name=test_rbd Request=Mount Message= unable to map: test_rbd%!(EXTRA *exec.ExitError=exit status 2)"

ceph.conf

root@compute-1:/mnt# cat /etc/ceph/ceph.conf
[global]
    fsid = a389269c-d35b-11ee-861c-c662cf06d087
    mon_host = [v2:10.124.0.5:3300/0,v1:10.124.0.5:6789/0] [v2:10.124.0.6:3300/0,v1:10.124.0.6:6789/0] [v2:10.124.0.7:3300/0,v1:10.124.0.7:6789/0]
[client.compute]
keyring = /etc/ceph/ceph.client.compute.keyring
AlexanderShaw commented 7 months ago

This was resolved by adding rbd to modules and rebooting: echo "rbd" >> /etc/modules

Request to add this to the prerequisites in the readme

sitamet commented 7 months ago

Thats it!