search

nutanix / helm

Nutanix Helm Charts repository
https://nutanix.github.io/helm/
MIT License
17 stars 30 forks source link

Microk8s: Nutanix CSI fails to mount NFS share (pvc directory not created) #48

Closed vfrans closed 2 years ago

vfrans commented 2 years ago

When we try to use the Nutanix CSI on out Kubernetes cluster, we have the following error while trying to create a pod that uses a 

PVC bound to Nutanix Files:
2022-03-01T15:16:43.272Z nodeserver.go:108: [INFO] NutanixFiles: successfully whitelisted 10.128.114.115 on NFS share pvc-fd14e158-1524-45a8-ae1e-b7d97df2ed16
E0301 15:16:43.363377       1 mount_linux.go:175] Mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t nfs bas-k8s-storage.mydom.tld:pvc-fd14e158-1524-45a8-ae1e-b7d97df2ed16 /var/snap/microk8s/common/var/lib/kubelet/pods/c9e749d5-cce8-42ad-84d7-a9273966c6cc/volumes/kubernetes.io~csi/pvc-fd14e158-1524-45a8-ae1e-b7d97df2ed16/mount
Output: mount.nfs: mount point /var/snap/microk8s/common/var/lib/kubelet/pods/c9e749d5-cce8-42ad-84d7-a9273966c6cc/volumes/kubernetes.io~csi/pvc-fd14e158-1524-45a8-ae1e-b7d97df2ed16/mount does not exist

2022-03-01T15:16:43.363Z node.go:62: [INFO] NodePublishVolume returned error rpc error: code = Internal desc = rpc error: code = Internal desc = mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t nfs bas-k8s-storage.mydom.tld:pvc-fd14e158-1524-45a8-ae1e-b7d97df2ed16 /var/snap/microk8s/common/var/lib/kubelet/pods/c9e749d5-cce8-42ad-84d7-a9273966c6cc/volumes/kubernetes.io~csi/pvc-fd14e158-1524-45a8-ae1e-b7d97df2ed16/mount
Output: mount.nfs: mount point /var/snap/microk8s/common/var/lib/kubelet/pods/c9e749d5-cce8-42ad-84d7-a9273966c6cc/volumes/kubernetes.io~csi/pvc-fd14e158-1524-45a8-ae1e-b7d97df2ed16/mount does not exist
E0301 15:16:43.363464       1 utils.go:31] GRPC error: rpc error: code = Internal desc = rpc error: code = Internal desc = mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t nfs bas-k8s-storage.mydom.tld:pvc-fd14e158-1524-45a8-ae1e-b7d97df2ed16 /var/snap/microk8s/common/var/lib/kubelet/pods/c9e749d5-cce8-42ad-84d7-a9273966c6cc/volumes/kubernetes.io~csi/pvc-fd14e158-1524-45a8-ae1e-b7d97df2ed16/mount
Output: mount.nfs: mount point /var/snap/microk8s/common/var/lib/kubelet/pods/c9e749d5-cce8-42ad-84d7-a9273966c6cc/volumes/kubernetes.io~csi/pvc-fd14e158-1524-45a8-ae1e-b7d97df2ed16/mount does not exist

We are using the helm chart provided at https://github.com/nutanix/helm/tree/nutanix-csi-storage-2.5.1/charts/nutanix-csi-storage#installing-the-chart, using the version 2.5.1 with the following value file:

# disable unused pv mechanisms
volumeClass: false
fileClass: false

# dynfile mechanism
dynamicFileClass: true
defaultStorageClass: dynfile

# what file server to use
fileServerName: bas-k8s-storage

# prism element ip, username and password
prismEndPoint: 10.128.4.201
username: mycoolworkingusernameinprism
password: some_cool_password
secretName: ntxn-secret
createSecret: true

CSI can create the share, we can see it in the files’s explorer. Also as you can see in the logs, whitelisting works (and if I try to manually mount the share on that server, it works fine).

Here are the versions we are using:

Ubuntu 20.04 MicroK8S v1.22.6-3+7ab10db7034594 Nutanix Files v 4.0.2

Having troubleshooted that a bit, it appears that the folder named after the PVC is not created in the pod’s directory.

tuxtof commented 2 years ago

Hello @vfrans

microk8s don't use default kubelet path so you need to change it during csi install

just add --set kubeletDir=/var/snap/microk8s/common/var/lib/kubelet during your csi-storage helm chart install and it need to solve your issue

vfrans commented 2 years ago

Thank you @tuxtof, that was definitely it, tried it and now it works!

Totally forgot to change that kubeletPath, thanks again!