rook-ceph: csi-rbdplugin pod stuck in CrashLoopBackOff

chrrrles commented 10 months ago

Summary

Unable to provision Ceph volumes in microk8s. The ceph-rbd pod is stuck in CrashLoopBackOff and is unable to connect to the csi.sock for registration (I presume).

chrrles@misscompy:~/ollama$ sudo ceph status
  cluster:
    id:     d9fd93ca-b8e9-47b3-a937-fcb7b37db83c
    health: HEALTH_OK

  services:
    mon: 1 daemons, quorum misscompy (age 3d)
    mgr: misscompy(active, since 48m)
    osd: 5 osds: 5 up (since 3d), 5 in (since 5d)

  data:
    pools:   2 pools, 33 pgs
    objects: 11 objects, 577 KiB
    usage:   64 MiB used, 6.4 TiB / 6.4 TiB avail
    pgs:     33 active+clean

chrrles@misscompy:~$ kubectl get pods -n rook-ceph
NAME                                           READY   STATUS             RESTARTS        AGE
rook-ceph-operator-5886b6b65d-f8jll            1/1     Running            0               23m
csi-cephfsplugin-2xbkv                         2/2     Running            0               23m
csi-rbdplugin-provisioner-5885496bf5-z4j8g     5/5     Running            0               23m
csi-cephfsplugin-provisioner-95fbd8485-75wz5   5/5     Running            0               23m
csi-rbdplugin-vcc2p                            1/2     CrashLoopBackOff   9 (2m16s ago)   23m

chrrles@misscompy:~$ kubectl logs pods/csi-rbdplugin-vcc2p -n rook-ceph                                                                                                                 
Defaulted container "driver-registrar" out of: driver-registrar, csi-rbdplugin           
I0110 12:10:28.160704 4028697 main.go:167] Version: v2.7.0                               
I0110 12:10:28.160761 4028697 main.go:168] Running node-driver-registrar in mode=registration                                                                                           
W0110 12:10:38.253660 4028697 connection.go:173] Still connecting to unix:///csi/csi.sock
W0110 12:10:48.253543 4028697 connection.go:173] Still connecting to unix:///csi/csi.sock
[repeats]

PVCs can be created but pods cannot use them.

chrrles@misscompy:~/ollama$ cat ollama-pvc.yaml 
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ollama-pvc
spec:
  storageClassName: ceph-rbd
  accessModes: [ReadWriteOnce]
  volumeMode: Filesystem
  resources:
    requests:
      storage: 100Gi

chrrles@misscompy:~/ollama$ kubectl apply -f ollama-pvc.yaml
persistentvolumeclaim/ollama-pvc configured                                                                                                                                             

chrrles@misscompy:~/ollama$ cat ollama-copy-pod.yaml 
---
apiVersion: v1
kind: Pod
metadata:
  name: ollama-copy-pod
spec:
  volumes:
    - name: ollama-copy-storage
      persistentVolumeClaim:
        claimName: ollama-pvc
  containers:
    - name: ollama-copy-container
      image: alpine
      volumeMounts:
        - name: ollama-copy-storage
          mountPath: "/ollama"
      lifecycle:
        postStart:
          exec:
            command: ["/bin/mkdir", "/ollama"]

chrrles@misscompy:~/ollama$ kubectl apply -f ollama-copy-pod.yaml -n ollama
pod/ollama-copy-pod created

chrrles@misscompy:~/ollama$ kubectl events -n ollama ollama-copy-pod
LAST SEEN            TYPE      REASON        OBJECT                MESSAGE
92s (x13 over 11m)   Warning   FailedMount   Pod/ollama-copy-pod   MountVolume.MountDevice failed for volume "pvc-50f8bda6-750e-459c-b65d-922f8ccb6d0a" : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name rook-ceph.rbd.csi.ceph.com not found in the list of registered CSI drivers
44s                  Normal    Scheduled     Pod/ollama-copy-pod   Successfully assigned ollama/ollama-copy-pod to misscompy
13s (x7 over 44s)    Warning   FailedMount   Pod/ollama-copy-pod   MountVolume.MountDevice failed for volume "pvc-50f8bda6-750e-459c-b65d-922f8ccb6d0a" : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name rook-ceph.rbd.csi.ceph.com not found in the list of registered CSI drivers

What Should Happen Instead?

Pods should be able to utilize PVCs provisioned by ceph RBD.

Reproduction Steps

Install microceph:

sudo snap install microceph --channel=latest/edge
sudo microceph cluster bootstrap
# add disks
sudo microceph /dev/disk/by-id/$disk --wipe
# configure for standalone cluster (via microk8s howto)
sudo microceph.ceph config set global osd_pool_default_size 2                               
sudo microceph.ceph config set mgr mgr_standby_modules false                                                                                                                                                      
sudo microceph.ceph config set osd osd_crush_chooseleaf_type 0

Connect microk8s to local microceph:

sudo microk8s enable rook-ceph
sudo microk8s connect-external-ceph

Provision PVC and pod.

Introspection Report

MicroK8s v1.28.3 revision 6089

inspection-report-20240110_130742.tar.gz

Can you suggest a fix?

:man_shrugging: I wish I could suggest a fix

Are you interested in contributing with a fix?

Yes, with some guidance. :+1:

neoaggelos commented 9 months ago

Hi @chrrrles

Looks like Rook is having trouble registering the CSI drivers and nodes? Can you check the output of:

sudo microk8s kubectl get csidrivers
sudo microk8s kubectl get csinodes

Also, retrieving logs from --all-containers on the rook pods might shed some more light as to what is happening. Can you also check whether the ceph rbd pools are created? Though I imagine the issues are related to CSI instead.

chrrrles commented 9 months ago

Thanks @neoaggelos - the --all-containers log flag identified the problem source. The rbd plugin is unable to load the rbd kernel module:

chrrles@misscompy:~$ kubectl logs csi-rbdplugin-5nvp9 -n rook-ceph --all-containers
...snip...
E0112 01:51:42.259532   42440 rbd_util.go:303] modprobe failed (an error (exit status 1) occurred while running modprobe args: [rbd]): "modprobe: ERROR: could not insert 'rbd': Exec format error\n"

Manually loading the rbd module and reloading then results in a warning that the nbd module cannot be loaded:

chrrles@misscompy:~$ kubectl delete pods csi-rbdplugin-5nvp9 -n rook-ceph
pod "csi-rbdplugin-5nvp9" deleted
chrrles@misscompy:~$ kubectl logs ds/csi-rbdplugin -n rook-ceph --all-containers
W0112 01:57:48.892866   59135 rbd_attach.go:226] nbd modprobe failed (an error (exit status 1) occurred while running modprobe args: [nbd]): "modprobe: ERROR: could not insert 'nbd': Exec format error\n"
chrrles@misscompy:~$ sudo modprobe nbd
chrrles@misscompy:~$ kubectl delete pods/csi-rbdplugin-5n5n9 -n rook-ceph
pod "csi-rbdplugin-5n5n9" deleted
chrrles@misscompy:~$ kubectl logs ds/csi-rbdplugin -n rook-ceph --all-containers
I0112 01:59:58.670031   65930 main.go:167] Version: v2.7.0
I0112 01:59:58.670086   65930 main.go:168] Running node-driver-registrar in mode=registration
I0112 01:59:59.677013   65930 node_register.go:53] Starting Registration Server at: /registration/rook-ceph.rbd.csi.ceph.com-reg.sock
I0112 01:59:59.677312   65930 node_register.go:62] Registration Server started at: /registration/rook-ceph.rbd.csi.ceph.com-reg.sock
I0112 01:59:59.677437   65930 node_register.go:92] Skipping HTTP server because endpoint is set to: ""
I0112 02:00:00.572303   65930 main.go:102] Received GetInfo call: &InfoRequest{}
I0112 02:00:00.573246   65930 main.go:109] "Kubelet registration probe created" path="/var/snap/microk8s/common/var/lib/kubelet/plugins/rook-ceph.rbd.csi.ceph.com/registration"
I0112 02:00:00.633474   65930 main.go:121] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:true,Error:,}

Adding rbd and nbd to /etc/modules fixes the issue with mounting the PVC volume. Further research shows this to be a kernel incompatibility preventing the rbd module from being loaded by the rook agent... Which is odd because this kernel should be compatible (https://documentation.suse.com/es-es/ses/7/html/ses-all/admin-caasp-ceph-common-issues.html#solution-7):

chrrles@misscompy:~/ollama$ uname -a
Linux misscompy 6.5.0-14-generic #14-Ubuntu SMP PREEMPT_DYNAMIC Tue Nov 14 14:59:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Regardless, this does not feel related to microk8s so I will close this issue. Thanks again for helping @neoaggelos !

:100: :fireworks: :raised_hands:

serafdev commented 4 months ago

Thank you @chrrrles , this is not loaded by default on the latest ubuntu cloud image (noble) either, kernel 6.8+;

Don't use the /etc/modules, your solution works. Just to recap I had to run on each Kubernetes node:

sudo modprobe rbd
sudo modprobe nbd

Edit: To make it persistent you need to add to the file as said @chrrrles, the recommended linux way is to have one file per module for some reason:

echo rbd | sudo tee -a /etc/modules-load.d/rbd.conf
echo nbd | sudo tee -a /etc/modules-load.d/nbd.conf
sudo chmod 777 /etc/modules-load.d/rbd.conf
sudo chmod 777 /etc/modules-load.d/nbd.conf

Not sure if you need all 777 permissions, but the deprecated /etc/modules file has those permissions so I just moved it over, without that the modules were not loaded at boot

canonical / microk8s