Pods with fsGroup can't attach OpenEBS cStor volumes

[mariusleu]

Hello,

I am running microk8s v1.21.3-3+90fd5f3d2aea0a in a single-node setup. I installed OpenEBS with cStor using their helm-charts.

I can successfully create cStor volumes and attach it to pods, but once the pod gets a securityContext.fsGroup set, this are breaking.

I have a dummy deployment with this securityContext:

    spec:
      securityContext:
        fsGroup: 999

With that set, I get the follwing error:

  Warning  FailedMount  10s (x4 over 24s)  kubelet            MountVolume.SetUp failed for volume "pvc-301be6ac-c375-4fdd-a285-31cc4dd267e6" : applyFSGroup failed for vol pvc-301be6ac-c375-4fdd-a285-31cc4dd267e6: lstat /var/snap/microk8s/common/var/lib/kubelet/pods/0baf8c2a-6c2c-4d25-bb95-7901868bdb87/volumes/kubernetes.io~csi/pvc-301be6ac-c375-4fdd-a285-31cc4dd267e6/mount: no such file or directory

Removing the fsGroup makes the pod running again.

I cannot reproduce this issue with a kubernetes deployed by kubespray.

inspection-report-20210902_163233.tar.gz

[balchua]

You may want to check with the openebs folks. @niladrih sorry for the direct mention, maybe able to provide more insights.

[niladrih]

@mariusleu -- Make sure to have iscsid.service systemd service in active(running) state in all of your nodes. Make sure to have the number of replicas <= number of nodes.

I was able to mount and use a volume with a statefulset whose Pod spec had

      securityContext:
        fsGroup: 999

Mount succeeded and the Pod was able to come to running state.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[mogindi]

Hello @niladrih,

I'm experiencing what seems to be the same issue (though using cinder-csi). It works when I remove fsGroup.

kubectl apply -f - << EOF
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  serviceName: "nginx"
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      #nodeName: k8s-master-0
      securityContext:
        fsGroup: 2000
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: www
          mountPath: /var/lib/www/data
  volumeClaimTemplates:
  - metadata:
      name: www
    spec:
      accessModes: [ "ReadWriteOnce" ]
      #storageClassName: "cinder-csi-2"
      resources:
        requests:
          storage: 2Gi
EOF

and the result is a Pod stuck in ContainerCreating:

Events:
  Type     Reason                  Age               From                     Message
  ----     ------                  ----              ----                     -------
  Warning  FailedMount             49s               kubelet                  Unable to attach or mount volumes: unmounted volumes=[www], unattached volumes=[kube-api-access-lqhfg www]: error processing PVC default/www-web-0: PVC is not bound
  Normal   SuccessfulAttachVolume  45s               attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-0bc42b6a-8490-431e-b8fb-2b6faa15b6a5"
  Warning  FailedMount             4s (x7 over 36s)  kubelet                  MountVolume.SetUp failed for volume "pvc-0bc42b6a-8490-431e-b8fb-2b6faa15b6a5" : applyFSGroup failed for vol fdc3e139-c5be-4628-96a4-922f2e655fb7: lstat /var/snap/microk8s/common/var/lib/kubelet/pods/c5b3da12-ab62-4382-b7a5-760874e15558/volumes/kubernetes.io~csi/pvc-0bc42b6a-8490-431e-b8fb-2b6faa15b6a5/mount: no such file or directory

I did make sure that iscsid.service is running on all the nodes - didn't really help

When you did your tests, you did mount and use an external volume, right?

[astromechza]

I've hit the same thing with topolvm CSI driver.

May 05 18:22:53 hensteeth microk8s.daemon-kubelite[2816]: E0505 18:22:53.567503    2816 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/topolvm.io^2801e0bb-1fe5-4d03-9685-08f5b900b25e podName: nodeName:}" failed. No retries permitted until 2023-05-05 18:24:55.567455089 +0000 UTC m=+77822.342295757 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "pvc-71853ee4-e5e9-45bd-a59c-b5927bc597df" (UniqueName: "kubernetes.io/csi/topolvm.io^2801e0bb-1fe5-4d03-9685-08f5b900b25e") pod "thanos-storegateway-0" (UID: "28081198-b041-40e7-875a-33db4a8b9467") :
applyFSGroup failed for vol 2801e0bb-1fe5-4d03-9685-08f5b900b25e: lstat /var/snap/microk8s/common/var/lib/kubelet/pods/28081198-b041-40e7-875a-33db4a8b9467/volumes/kubernetes.io~csi/pvc-71853ee4-e5e9-45bd-a59c-b5927bc597df/mount: no such file or directory

Again with an fsGroup for the target pod.

[neoaggelos]

Hi @astromechza how are you deploying the topolvm CSI driver? Make sure to change the manifest and replace /var/lib/kubelet with /var/snap/microk8s/common/var/lib/kubelet. What you are describing is a common issue when deploying CSI drivers without setting the proper path to the microk8s kubelet directory.

If you are deploying with a Helm chart, it should be as simple as setting the appropriate config option, e.g. --set kubeletDir=/var/snap/microk8s/common/var/lib/kubelet, or if deploying with a simple manifest, just edit/replace the kubelet paths manually in the hostpath volume mounts.

Happy to help if you encounter any issue!

[astromechza]

@neoaggelos Oh man, I can't believe I didn't spot that in the docs! That's working as expected now for me. Thanks!

[fbrier]

In my snap microk8s, there is a symbolic link "/var/lib/kubelet -> /var/snap/microk8s/common/var/lib/kubelet". You shouldn't have to change the manifest at that point, correct? I am also seeing the same error as @mariusleu using OpenEBS with the openebs-lvmpv storageClass. What is strange is that the Jellyfin Helm chart installed and runs properly with a 1Ti PV, but a Bitnami Tomcat Helm chart fails with a 8Gi PV.

Actually, the error is slightly different, so this may be unrelated.

MountVolume.SetUp failed for volume "pvc-a7da2d41-a371-4ee1-b98b-8e9c778725d3" : rpc error: code = Internal desc = lvmvolumes.local.openebs.io "pvc-a7da2d41-a371-4ee1-b98b-8e9c778725d3" not found

[fbrier]

Correction - it is the same error. After a few minutes the error changes to:

MountVolume.SetUp failed for volume "pvc-c2e063d9-8b92-4f91-8e12-e53b81eabd7e" : applyFSGroup failed for vol pvc-c2e063d9-8b92-4f91-8e12-e53b81eabd7e: lstat /var/snap/microk8s/common/var/lib/kubelet/pods/e6f9181c-9742-4f41-a5eb-7c5f5685eb24/volumes/kubernetes.io~csi/pvc-c2e063d9-8b92-4f91-8e12-e53b81eabd7e/mount: no such file or directory

And the same error as @astromechza in syslog (multiple errors, just snagged one of them)

syslog.1:Jun 27 23:59:17 solar microk8s.daemon-kubelite[2807]: E0627 23:59:17.605095    2807 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/local.csi.openebs.io^pvc-a7da2d41-a371-4ee1-b98b-8e9c778725d3 podName: nodeName:}" failed. No retries permitted until 2023-06-28 00:01:19.60507542 -0600 MDT m=+729139.861732219 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "pvc-a7da2d41-a371-4ee1-b98b-8e9c778725d3" (UniqueName: "kubernetes.io/csi/local.csi.openebs.io^pvc-a7da2d41-a371-4ee1-b98b-8e9c778725d3") pod "redmine-7c944f468d-wd5kz" (UID: "728c341d-d7d7-4f01-8299-50f641d358e7") : rpc error: code = Internal desc = lvmvolumes.local.openebs.io "pvc-a7da2d41-a371-4ee1-b98b-8e9c778725d3" not found

[neoaggelos]

In my snap microk8s, there is a symbolic link "/var/lib/kubelet -> /var/snap/microk8s/common/var/lib/kubelet". You shouldn't have to change the manifest at that point, correct?

This is a hacky fix for some common issues, but will typically not work for most CSI drivers, as shown above.