Version of ceph library / binary inside csi-rbd for cephcsi connections

[jisnardo]

Hello, I have the following question:

When csi-rbd connects with ceph, what version of the library / binary does it use and in what container of which pod is it located?

NAME                                               READY   STATUS    RESTARTS   AGE
csi-rbd-ceph-csi-rbd-nodeplugin-7jzdq              3/3     Running   0          106d
csi-rbd-ceph-csi-rbd-nodeplugin-c945c              3/3     Running   0          106d
csi-rbd-ceph-csi-rbd-nodeplugin-ckxvr              3/3     Running   0          106d
csi-rbd-ceph-csi-rbd-nodeplugin-w7xlx              3/3     Running   6          106d
csi-rbd-ceph-csi-rbd-provisioner-cb66fd5b8-8pmx4   6/6     Running   1          20d
csi-rbd-ceph-csi-rbd-provisioner-cb66fd5b8-gkck2   6/6     Running   0          19d
csi-rbd-ceph-csi-rbd-provisioner-cb66fd5b8-qpspj   6/6     Running   3          20d

a container name must be specified for pod csi-rbd-ceph-csi-rbd-nodeplugin-7jzdq, choose one of: [driver-registrar csi-rbdplugin liveness-prometheus]

a container name must be specified for pod csi-rbd-ceph-csi-rbd-provisioner-cb66fd5b8-8pmx4, choose one of: [csi-provisioner csi-resizer csi-snapshotter csi-attacher csi-rbdplugin liveness-prometheus]

Is there any way to update it?

[root@host01 someone]$ ceph daemon mon.cephsys01a sessions | grep jewel
"MonSession(unknown.0 192.168.80.2:0/703605752 is open allow r, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.4429955 192.168.80.3:0/3486115821 is open allow *, features 0x27018fb86aa42ada (jewel))",

Thank you.

[Madhu-1]

csi-rbdplugin container is the one that connects to the ceph cluster in both the provisioner and nodeplugin pod.

you can exec into the container and run rpm -qa|grep ceph rpm -qa|grep rbd to find out the release we use in cephcsi.

[🎩︎]mrajanna@localhost rook $]kuberc exec -it csi-rbdplugin-r9kq7 -c csi-rbdplugin sh
sh-4.4# rpm -qa|grep ceph
ceph-prometheus-alerts-15.2.4-0.el8.noarch
cephadm-15.2.4-0.el8.x86_64
ceph-common-15.2.4-0.el8.x86_64
ceph-mgr-dashboard-15.2.4-0.el8.noarch
ceph-mgr-15.2.4-0.el8.x86_64
ceph-mon-15.2.4-0.el8.x86_64
nfs-ganesha-ceph-3.3-0.1.el8.x86_64
ceph-fuse-15.2.4-0.el8.x86_64
python3-ceph-argparse-15.2.4-0.el8.x86_64
ceph-grafana-dashboards-15.2.4-0.el8.noarch
ceph-release-1-1.el8.noarch
ceph-base-15.2.4-0.el8.x86_64
ceph-mgr-cephadm-15.2.4-0.el8.noarch
ceph-mgr-diskprediction-local-15.2.4-0.el8.noarch
ceph-mgr-modules-core-15.2.4-0.el8.noarch
ceph-mgr-rook-15.2.4-0.el8.noarch
ceph-mds-15.2.4-0.el8.x86_64
ceph-osd-15.2.4-0.el8.x86_64
ceph-iscsi-3.4-1.el8.noarch
python3-ceph-common-15.2.4-0.el8.x86_64
libcephfs2-15.2.4-0.el8.x86_64
python3-cephfs-15.2.4-0.el8.x86_64
ceph-selinux-15.2.4-0.el8.x86_64
ceph-mgr-k8sevents-15.2.4-0.el8.noarch
ceph-radosgw-15.2.4-0.el8.x86_64
sh-4.4# rpm -qa|grep rbd 
librbd1-15.2.4-0.el8.x86_64
python3-rbd-15.2.4-0.el8.x86_64
rbd-nbd-15.2.4-0.el8.x86_64
rbd-mirror-15.2.4-0.el8.x86_64

in above ceph version is 15.2.4 which is the base ceph image cephcsi v3.1.0 is built on

[Madhu-1]

only way to update the ceph version is you need to build your own cephcsi container from the base image you want

[jisnardo]

Thanks @Madhu-1

[jisnardo]

I'm confused...

I see jewel connections but inside the pod in the rbdplugin container the ceph version is nautilus.

[root@ceph01a someone]$ ceph daemon mon.ceph01a sessions | grep jewel | grep "192.168.81"
"MonSession(client.4039878 192.168.81.13:0/4121742705 is open allow *, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.3904259 192.168.81.11:0/384834475 is open allow *, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.3884870 192.168.81.12:0/4070399204 is open allow *, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.3884426 192.168.81.14:0/4205492656 is open allow *, features 0x27018fb86aa42ada (jewel))",

[someone@mypc ~]$ k8s -n csi-rbd exec -it csi-rbd-ceph-csi-rbd-nodeplugin-7jzdq -c csi-rbdplugin -- sh
sh-4.2# rpm -qa | grep -i ceph | sort
ceph-base-14.2.9-0.el7.x86_64
ceph-common-14.2.9-0.el7.x86_64
ceph-fuse-14.2.9-0.el7.x86_64
ceph-grafana-dashboards-14.2.9-0.el7.noarch
ceph-iscsi-3.4-1.el7.noarch
ceph-mds-14.2.9-0.el7.x86_64
ceph-mgr-14.2.9-0.el7.x86_64
ceph-mgr-dashboard-14.2.9-0.el7.noarch
ceph-mgr-diskprediction-local-14.2.9-0.el7.noarch
ceph-mgr-k8sevents-14.2.9-0.el7.noarch
ceph-mgr-rook-14.2.9-0.el7.noarch
ceph-mon-14.2.9-0.el7.x86_64
ceph-osd-14.2.9-0.el7.x86_64
ceph-radosgw-14.2.9-0.el7.x86_64
ceph-release-1-1.el7.noarch
ceph-selinux-14.2.9-0.el7.x86_64
libcephfs2-14.2.9-0.el7.x86_64
nfs-ganesha-ceph-2.8.1.2-0.1.el7.x86_64
python-ceph-argparse-14.2.9-0.el7.x86_64
python-cephfs-14.2.9-0.el7.x86_64

sh-4.2# ceph -v
ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)

sh-4.2# cephcsi -version
Cephcsi Version: v2.1.1
Git Commit: 9022d899eb6fd464f0be33701d8160ecd1317467
Go Version: go1.13.8
Compiler: gc
Platform: linux/amd64

How is this explained?

We understand that if https://github.com/ceph/ceph-csi/blob/master/build.env appears:

# Ceph version to use
BASE_IMAGE=ceph/ceph:v15
CEPH_VERSION=nautilus

And it has been used to build Cephcsi Version: v2.1.1 via https://github.com/ceph/ceph-csi/blob/master/deploy/cephcsi/image/Dockerfile the connections are not from nautilus.

Thanks

[Madhu-1]

[root@ceph01a someone]$ ceph daemon mon.ceph01a sessions | grep jewel | grep "192.168.81" "MonSession(client.4039878 192.168.81.13:0/4121742705 is open allow , features 0x27018fb86aa42ada (jewel))", "MonSession(client.3904259 192.168.81.11:0/384834475 is open allow , features 0x27018fb86aa42ada (jewel))", "MonSession(client.3884870 192.168.81.12:0/4070399204 is open allow , features 0x27018fb86aa42ada (jewel))", "MonSession(client.3884426 192.168.81.14:0/4205492656 is open allow , features 0x27018fb86aa42ada (jewel))",

@jisnardo what is the ceph cluster version you are connecting to?

[jisnardo]

[root@ceph01a ~]$ ceph -v
ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic (stable)

[Madhu-1]

@dillaman @idryomov can help us more on this one

[idryomov]

@jisnardo What kernel are you running on those 192.168.81 nodes?

The kernel client used to report itself as jewel up to kernel 5.3, even though all luminous features were implemented in kernel 4.13. I'm pretty sure this has nothing to do with CSI pods -- you are seeing kernel rbd sessions.

[jisnardo]

root@ksys01a:~# uname -a
Linux ksys01a.ksysn.test.es 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

[idryomov]

Yup, 4.15 kernels will be reported as jewel.

[jisnardo]

Ok, i try.

[jisnardo]

Now working as expected after installing https://wiki.ubuntu.com/Kernel/LTSEnablementStack over Ubuntu 18.04 LTS.

root@node00a:~# apt install linux-image-generic-hwe-18.04

root@node00a:~# uname -a
Linux node00a.workspace.test.es 5.4.0-47-generic #51~18.04.1-Ubuntu SMP Sat Sep 5 14:35:50 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

root@ceph01a someone]$ ceph daemon mon.ceph01a sessions | grep "192.168.81.1"
"MonSession(unknown.0 192.168.81.1:0/289741994 is open allow r, features 0x2f018fb86aa42ada (luminous))",
"MonSession(unknown.0 192.168.81.1:0/3511154870 is open allow *, features 0x2f018fb86aa42ada (luminous))",

Thanks.