Microk8s - MountVolume.SetUp failed

[fornof]

What steps did you take and what happened: [A clear and concise description of what the bug is, and what commands you ran.]

https://github.com/openebs/zfs-localpv/issues/235 this is about what I'm running into, but I didn't see how to solve it in the ticket:

   Warning  FailedMount  86s (x12 over 9m40s)  kubelet  MountVolume.SetUp failed for volume "pvc-3da01b7f-3eb3-427d-943d-42758527c8f2" : applyFSGroup failed for vol pvc-3da01b7f-3eb3-427d-943d-42758527c8f2: lstat /var/snap/microk8s/common/var/lib/kubelet/pods/b803ceff-d82e-4fe2-be34-bb05f6bee84d/volumes/kubernetes.io~csi/pvc-3da01b7f-3eb3-427d-943d-42758527c8f2/mount: no such file or directory                                               

I tried again using a deployment mount and I get:

persistent volume:

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM
            STORAGECLASS           VOLUMEATTRIBUTESCLASS   REASON   AGE
pvc-2e49cd25-4b11-4618-817f-3095de775145   20Gi       RWO            Delete           Released   minio-operator/0-microk8s-ss-0-0   microk8s-hostpath      <unset>                          46d
pvc-3a97c089-e659-4d9e-91a6-62bae21f89f4   76Gi       RWO            Delete           Bound      postgres/postgres-fornof-zfs-pvc   openebs-zfspv-fornof   <unset>                          11h
pvc-3da01b7f-3eb3-427d-943d-42758527c8f2   76Gi       RWO            Delete           Bound      db/postgres-fornof-zfs-pvc         openebs-zfspv-fornof   <unset>                          29m
persistent volume claims : 

{ "apiVersion": "v1", "items": [ { "apiVersion": "v1", "kind": "PersistentVolumeClaim", "metadata": { "creationTimestamp": "2024-06-22T16:59:42Z", "finalizers": [ "kubernetes.io/pvc-protection" ], "labels": { "app.kubernetes.io/component": "primary", "app.kubernetes.io/instance": "postgres-bitnami", "app.kubernetes.io/name": "postgresql" }, "name": "data-postgres-bitnami-postgresql-0", "namespace": "db", "resourceVersion": "17057302", "uid": "9a5bb71b-76d0-4a91-a84b-78eb5f9790ee" }, "spec": { "accessModes": [ "ReadWriteOnce" ], "resources": { "requests": { "storage": "8Gi" } }, "volumeMode": "Filesystem" }, "status": { "phase": "Pending" } }, { "apiVersion": "v1", "kind": "PersistentVolumeClaim", "metadata": { "annotations": { "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"PersistentVolumeClaim\",\"metadata\":{\"annotations\":{},\"name\":\"postgres-fornof-zfs-pvc\",\"namespace\":\"db\"},\"spec\":{\"accessModes\":[\"ReadWriteOnce\"],\"resources\":{\"requests\":{\"storage\":\"76Gi\"}},\"storageClassName\":\"openebs-zfspv-fornof\"}}\n", "pv.kubernetes.io/bind-completed": "yes", "pv.kubernetes.io/bound-by-controller": "yes", "volume.beta.kubernetes.io/storage-provisioner": "zfs.csi.openebs.io", "volume.kubernetes.io/storage-provisioner": "zfs.csi.openebs.io" }, "creationTimestamp": "2024-06-22T17:14:01Z", "finalizers": [ "kubernetes.io/pvc-protection" ], "name": "postgres-fornof-zfs-pvc", "namespace": "db", "resourceVersion": "17059608", "uid": "3da01b7f-3eb3-427d-943d-42758527c8f2" }, "spec": { "accessModes": [ "ReadWriteOnce" ], "resources": { "requests": { "storage": "76Gi" } }, "storageClassName": "openebs-zfspv-fornof", "volumeMode": "Filesystem", "volumeName": "pvc-3da01b7f-3eb3-427d-943d-42758527c8f2" }, "status": { "accessModes": [ "ReadWriteOnce" ], "capacity": { "storage": "76Gi" }, "phase": "Bound" } } ], "kind": "List", "metadata": { "resourceVersion": "" } }

NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE data-postgres-bitnami-postgresql-0 Pending

44m postgres-fornof-zfs-pvc Bound pvc-3da01b7f-3eb3-427d-943d-42758527c8f2 76Gi RWO openebs-zfspv-fornof 30m ``` storage classes: ``` NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE openebs-hostpath openebs.io/local Delete WaitForFirstConsumer false 77m openebs-zfspv-fornof zfs.csi.openebs.io Delete Immediate false 15h ``` openebs pods: ``` NAME READY STATUS RESTARTS AGE openebs-localpv-provisioner-7cd9f85f8f-6d7cp 1/1 Running 0 76m openebs-zfs-localpv-controller-7fdcd7f65-74l55 5/5 Running 0 76m openebs-zfs-localpv-node-6dn8t 2/2 Running 0 76m ``` rob@media:/mnt$ zfs list NAME USED AVAIL REFER MOUNTPOINT robin-makeup 299M 1.76T 283M /mnt/robin-makeup-zfs robin-makeup/pvc-3da01b7f-3eb3-427d-943d-42758527c8f2 24K 76.0G 24K legacy **What did you expect to happen:** expected volume to mount **The output of the following commands will help us better understand what's going on**: (Pasting long output into a [GitHub gist](https://gist.github.com) or other [Pastebin](https://pastebin.com/) is fine.) * `kubectl logs -f openebs-zfs-controller-f78f7467c-blr7q -n openebs -c openebs-zfs-plugin` https://nopaste.net/B3aBBGpU1q * `kubectl logs -f openebs-zfs-node-[xxxx] -n openebs -c openebs-zfs-plugin` * https://nopaste.net/yjl29wBl0F * `kubectl get pods -n openebs` ``` NAME READY STATUS RESTARTS AGE openebs-localpv-provisioner-7cd9f85f8f-6d7cp 1/1 Running 0 88m openebs-zfs-localpv-controller-7fdcd7f65-74l55 5/5 Running 0 88m openebs-zfs-localpv-node-6dn8t 2/2 Running 0 88m ``` * `kubectl get zv -A -o yaml` ``` kubectl get zv -A -o yaml apiVersion: v1 items: - apiVersion: zfs.openebs.io/v1 kind: ZFSVolume metadata: creationTimestamp: "2024-06-22T17:14:01Z" finalizers: - zfs.openebs.io/finalizer generation: 2 labels: kubernetes.io/nodename: media name: pvc-3da01b7f-3eb3-427d-943d-42758527c8f2 namespace: openebs resourceVersion: "17059601" uid: 685593db-552a-4daa-838b-4b8137a30c1a spec: capacity: "81604378624" compression: "off" dedup: "off" fsType: zfs ownerNodeID: media poolName: robin-makeup recordsize: 128k volumeType: DATASET status: state: Ready kind: List metadata: resourceVersion: "" ``` **Anything else you would like to add:** I'm trying to get postgresql up and going on kubernetes and having a hard time mounting the persistent volume. [Miscellaneous information that will assist in solving the issue.] I'm using bitnami postgresql with persistence set to true and storage set to true as the same volume claim in the helm chart values. **Environment:** - LocalPV-ZFS version - openebs.io/version=4.0.0 - Kubernetes version (use `kubectl version`): ``` Client Version: v1.30.1 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.30.1 ``` - Kubernetes installer & version: - MicroK8s v1.30.1 revision 6876 - Cloud provider or hardware configuration: Ubuntu 2TB mirrored storage ZFS pool. - OS (e.g. from `/etc/os-release`): ```PRETTY_NAME="Debian GNU/Linux 11 (bullseye)" NAME="Debian GNU/Linux" VERSION_ID="11" VERSION="11 (bullseye)" VERSION_CODENAME=bullseye ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/" ```

[Abhinandan-Purkait]

@fornof Did you change the kubelet dir for the microk8s while installing zfs?

--set zfs-localpv.zfsNode.kubeletDir=/var/snap/microk8s/common/var/lib/kubelet/

[avishnu]

@fornof can you provide the information sought by @Abhinandan-Purkait here? https://github.com/openebs/zfs-localpv/issues/547#issuecomment-2202089035

[avishnu]

Meanwhile, we could fix documentation to cover these steps: https://github.com/openebs/zfs-localpv?tab=readme-ov-file#installation

[fornof]

This can be closed as I am no longer working on getting zfs mounts for microk8s, I ended up just using docker compose for persistent storage/ host storage . But no , i did not change the mount point

[Abhinandan-Purkait]

Sure @fornof Feel free to reopen this if you come by this again. Thanks

[avishnu]

Can we keep this opened to track the doc changes?

[Abhinandan-Purkait]

@avishnu We already have this documented in the repo readme. https://github.com/openebs/zfs-localpv?tab=readme-ov-file#installation

We can reopen this if we feel it's needed to be added in the docs website. Probably an issue to track this on the website repo would be better place wdyt?

[avishnu]

Yes, it's not updated in the documentation clearly.

[Abhinandan-Purkait]

Have reopened this. cc @balaharish7

[balaharish7]

This issue is documented in #499.