Cephfs based pvc causes KubePersistentVolumeInodesFillingUp alert

[adux6991]

Describe the bug

Pod with cephfs based pvc shows limited inode information, causing prometheus to raise KubePersistentVolumeInodesFillingUp alert.

Environment details

• Image/version of Ceph CSI driver : cephcsi v3.4.1
• Helm chart version :
• Kernel version :

  uname -a
  Linux io1 5.4.0-100-generic #113-Ubuntu SMP Thu Feb 3 18:43:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

• Mounter used for mounting PVC (for cephFS its fuse or kernel. for rbd its krbd or rbd-nbd) : kernel
• Kubernetes cluster version :

  kubectl version
  Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.0", GitCommit:"a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2", GitTreeState:"clean", BuildDate:"2022-08-23T17:44:59Z", GoVersion:"go1.19", Compiler:"gc", Platform:"darwin/amd64"}
  Kustomize Version: v4.5.7
  Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.10", GitCommit:"5c1d2d4295f9b4eb12bfbf6429fdf989f2ca8a02", GitTreeState:"clean", BuildDate:"2023-01-18T19:08:10Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}

• Ceph cluster version :

  ceph -v
  ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)

  Steps to reproduce

1. create cephfs in ceph cluster
2. create storage class and other related resources in k8s cluster
3. create pvc and pod

   cat <<EOF | kubectl apply -f -
   apiVersion: v1
   kind: Pod
   metadata:
   name: cephfs-demo-pod
   spec:
   containers:
   - name: web-server
     image: nginx:latest
     resources:
       limits:
         cpu: 100m
         memory: 100Mi
     volumeMounts:
       - name: mypvc
         mountPath: /var/lib/www
   volumes:
   - name: mypvc
     persistentVolumeClaim:
       claimName: cephfs-pvc
   ---
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
   name: cephfs-pvc
   spec:
   accessModes:
   - ReadWriteMany
   resources:
   requests:
     storage: 1Gi
   storageClassName: csi-cephfs-sc
   EOF

4. kubectl exec into pod

   kubectl exec -it cephfs-demo-pod -- /bin/bash
   root@cephfs-demo-pod:/# df -i
   Filesystem                                                                                                        Inodes  IUsed    IFree IUse% Mounted on
   overlay                                                                                                         36462592 735679 35726913    3% /
   tmpfs                                                                                                           32964258     17 32964241    1% /dev
   tmpfs                                                                                                           32964258     17 32964241    1% /sys/fs/cgroup
   /dev/mapper/ubuntu--vg-lv--0                                                                                    36462592 735679 35726913    3% /etc/hosts
   shm                                                                                                             32964258      1 32964257    1% /dev/shm
   10.20.65.32:6789:/volumes/csi/csi-vol-1765fe5e-bc90-11ed-b91a-962c376839e9/6bb86d4d-a3ab-4dc2-80c8-b75b91489edd    11521      -        -     - /var/lib/www
   tmpfs                                                                                                           32964258      9 32964249    1% /run/secrets/kubernetes.io/serviceaccount
   tmpfs                                                                                                           32964258      1 32964257    1% /proc/acpi
   tmpfs                                                                                                           32964258      1 32964257    1% /proc/scsi
   tmpfs                                                                                                           32964258      1 32964257    1% /sys/firmware

Actual results

df -i reports - for IUsed / IFree / IUse%, which is regarded as zero by kubelet and causes KubePersistentVolumeInodesFillingUp alert.

Expected behavior

report actual inode usage

[humblec]

cc @nixpanic

[nixpanic]

Commit b7703faf37f5905f8e1c83f0c15a01df6cdbb181 should have addressed this. It was merged through PR #3407 and is part of v3.8.0.

Please update to a more recent Ceph-CSI version and let us know if the problem still persists.

[adux6991]

Thanks a lot! Deploying a new version resolves the problem.