unable to create volume after install

[miberecz]

I have a proxmox server running a ceph cluster. I just installed the plugin to use with my docker swarm, but the first volume creation fails:

root@nuc1:/etc/ceph# docker plugin install wetopi/rbd \
  --alias=wetopi/rbd \
  LOG_LEVEL=3 \
  RBD_CONF_POOL="banaceph" \
  RBD_CONF_CLUSTER=ceph \
  RBD_CONF_KEYRING_USER=client.admin
Plugin "wetopi/rbd" is requesting the following privileges:
 - network: [host]
 - mount: [/lib/modules]
 - mount: [/dev]
 - mount: [/etc/ceph]
 - mount: [/sys]
 - allow-all-devices: [true]
 - capabilities: [CAP_NET_ADMIN CAP_SYS_ADMIN CAP_SYS_MODULE]
Do you grant the above permissions? [y/N] y
latest: Pulling from wetopi/rbd
Digest: sha256:0d33a9f00a417d30e7e190d12e67c4950cc51a5f924c62c0c2344a4556c27359
360a92ce0ac8: Complete
Installed plugin wetopi/rbd

root@nuc1:/etc/ceph# docker volume create -d wetopi/rbd -o size=206 my_rbd_volume
Error response from daemon: create my_rbd_volume: VolumeDriver.Create: volume-rbd Name=unable to read default config /etc/ceph/ceph.conf: rados: ret=-2, No such file or directory Request=Create Message=unable to connect to rbd pool

root@nuc1:/etc/ceph# cat /etc/ceph/ceph.conf
[global]
         auth_client_required = cephx
         auth_cluster_required = cephx
         auth_service_required = cephx
         cluster_network = 10.10.0.105/24
         fsid = 4992e06c-f5b1-440f-8575-f7c9a6848f03
         mon_allow_pool_delete = true
         mon_host = 192.168.0.105
         ms_bind_ipv4 = true
         ms_bind_ipv6 = false
         osd_pool_default_min_size =3 
         osd_pool_default_size = 2
         public_network = 192.168.0.105/24

[client]
         keyring = /etc/pve/priv/$cluster.$name.keyring

[mds]
         keyring = /var/lib/ceph/mds/ceph-$id/keyring

[mds.nuc1]
         host = nuc1
         mds standby for name = pve

[mon.nuc1]
         public_addr = 192.168.0.105

Otherwise the cluster works, I can create VMs or LXC containers using CEPH as storage.

Any idea what am I missing?

[sitamet]

double checking the code, this error appears when "rados" lib tires to read the config file during the contexto of "connect to rbd pool".

idea: Double check each ceph node has its /etc/ceph/ceph.conf and it is readable (read privileges)

[miberecz]

/etc/ceph/ceph.conf has basically 777 permission. It is currently the whole ceph cluster, only one node with one OSD. Anything else can use it, proxmox VM disks, with CEPHFS any other client in my network...

root@nuc1:~# ls -la /etc/ceph/ceph.conf
lrwxrwxrwx 1 root root 18 Sep 11 20:48 /etc/ceph/ceph.conf -> /etc/pve/ceph.conf

root@nuc1:~# cat /etc/ceph/ceph.conf
[global]
         auth_client_required = cephx
         auth_cluster_required = cephx
         auth_service_required = cephx
         cluster_network = 10.10.0.105/24
         fsid = 4992e06c-f5b1-440f-8575-f7c9a6848f03
         mon_allow_pool_delete = true
         mon_host = 192.168.0.105
         ms_bind_ipv4 = true
         ms_bind_ipv6 = false
         osd_pool_default_min_size = 1
         osd_pool_default_size = 1
         public_network = 192.168.0.105/24

[client]
         keyring = /etc/pve/priv/$cluster.$name.keyring

[mds]
         keyring = /var/lib/ceph/mds/ceph-$id/keyring

[mds.nuc1]
         host = nuc1
         mds standby for name = pve

[mon.nuc1]
         public_addr = 192.168.0.105

[sitamet]

not sure if docker has Capabilities to read the /etc/pve?

just for testing purposes.. try with a copy of conf and key in your /etc/ceph/

[miberecz]

Okay I did that, now I have a slightly different error:

root@nuc1:/etc/ceph# ls -la
total 16
drwxr-xr-x   2 ceph ceph 4096 Sep 22 20:55 .
drwxr-xr-x 101 root root 4096 Sep 22 20:45 ..
lrwxrwxrwx   1 root root   18 Sep 11 20:48 bak_ceph.conf -> /etc/pve/ceph.conf
-rw-r--r--   1 ceph ceph  151 Sep 11 20:48 ceph.client.admin.keyring
-rw-r--r--   1 root root  713 Sep 22 20:55 ceph.conf

root@nuc1:/etc/ceph# docker volume create -d wetopi/rbd -o size=206 my_rbd_volume
Error response from daemon: create my_rbd_volume: VolumeDriver.Create: volume-rbd Name=unable to open the ceph cluster connection: rados: ret=-2, No such file or directory Request=Create Message=unable to connect to rbd pool

[sitamet]

OK, looks part of the same, the docker plugin does not have privileges to access to the volume where Ceph conf and keys are stored.

[Drallas]

@miberecz Did you get it work?

@sitamet I have a Proxmox Cluster with Ceph enabled storage, my Docker Swarm Nodes are Vm's on Proxmox, am I able to use this plugin, and if yes, how?

[sitamet]

Yes, try to follow the readme install steps