NFS Provisioner

[jaypaddy]

Following the instructions in Azure Stack HCI Quickstart I have a cluster deployed and AKS Hybrid installed. AKS Hybrid is connected to Arc for Kubernetes. Generally the cluster is operating fine.

Wanted to try out NFS as a Persistent Storage. So, i setup NFS on a Ubuntu machine outside the Cluster. Able to mount the nfs drive on the Host. Validated the nfs server is reachable from within a Pod in the Cluster.

So overall networking and access to NFS is fine. However the provisioner is unable to provision the volume...

Log from csi-nfs-controller-xxxxxxx-xxxx I0727 00:51:02.833479 1 controller.go:1337] provision "default/hci-nfs-pvc" class "nfs-csi": started I0727 00:51:02.834322 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"hci-nfs-pvc", UID:"f05ee6d2-97a6-47b9-91d2-807374b971c1", APIVersion:"v1", ResourceVersion:"553997", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/hci-nfs-pvc" I0727 00:51:02.906455 1 controller.go:1075] Final error received, removing PVC f05ee6d2-97a6-47b9-91d2-807374b971c1 from claims in progress W0727 00:51:02.906495 1 controller.go:934] Retrying syncing claim "f05ee6d2-97a6-47b9-91d2-807374b971c1", failure 28 E0727 00:51:02.906528 1 controller.go:957] error syncing claim "f05ee6d2-97a6-47b9-91d2-807374b971c1": failed to provision volume with StorageClass "nfs-csi": rpc error: code = Internal desc = failed to mount nfs server: rpc error: code = Internal desc = mount failed: exit status 32 Mounting command: mount Mounting arguments: -t nfs -o hard,nfsvers=4.1 10.0.0.5:/srv/nfs /tmp/pvc-f05ee6d2-97a6-47b9-91d2-807374b971c1 Output: mount.nfs: Operation not permitted I0727 00:51:02.906657 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"hci-nfs-pvc", UID:"f05ee6d2-97a6-47b9-91d2-807374b971c1", APIVersion:"v1", ResourceVersion:"553997", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "nfs-csi": rpc error: code = Internal desc = failed to mount nfs server: rpc error: code = Internal desc = mount failed: exit status 32 Mounting command: mount Mounting arguments: -t nfs -o hard,nfsvers=4.1 10.0.0.5:/srv/nfs /tmp/pvc-f05ee6d2-97a6-47b9-91d2-807374b971c1 Output: mount.nfs: Operation not permitted

To setup the NFS provisioner and Storage Class followed these instructions:

Install-AksHciCsiNfs -clusterName

apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: nfs-csi provisioner: nfs.csi.akshci.com parameters: server: 10.0.0.5 # NFS server endpoint share: /srv/nfs # NFS share path reclaimPolicy: Retain volumeBindingMode: Immediate mountOptions:

• hard

• nfsvers=4.1

  PVC apiVersion: v1 kind: PersistentVolumeClaim metadata: name: hci-nfs-pvc spec: accessModes:

• ReadWriteOnce storageClassName: nfs-csi resources: requests: storage: 1Mi

  kubectl describe pvc Name: hci-nfs-pvc Namespace: default StorageClass: nfs-csi Status: Pending Volume: Labels: Annotations: volume.beta.kubernetes.io/storage-provisioner: nfs.csi.akshci.com volume.kubernetes.io/storage-provisioner: nfs.csi.akshci.com Finalizers: [kubernetes.io/pvc-protection] Capacity: Access Modes: VolumeMode: Filesystem Used By: Events: Type Reason Age From Message

  ---

  Normal Provisioning 6s (x5 over 22s) nfs.csi.akshci.com_moc-lika3uh6pyg_2901a927-84d5-455f-864a-a2412a741e9f External provisioner is provisioning volume for claim "default/hci-nfs-pvc" Warning ProvisioningFailed 6s (x5 over 22s) nfs.csi.akshci.com_moc-lika3uh6pyg_2901a927-84d5-455f-864a-a2412a741e9f failed to provision volume with StorageClass "nfs-csi": rpc error: code = Internal desc = failed to mount nfs server: rpc error: code = Internal desc = mount failed: exit status 32 Mounting command: mount Mounting arguments: -t nfs -o hard,nfsvers=4.1 10.0.0.5:/srv/nfs /tmp/pvc-6880efc7-e44e-43ae-a257-181aac64f146 Output: mount.nfs: Operation not permitted Normal ExternalProvisioning 2s (x3 over 22s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "nfs.csi.akshci.com" or manually created by system administrator

  Again validated the Windows Explorer on the Host is able to mount and able to add a file and view it on the NFS server.

[jaypaddy]

As i further dug into the problem, i found the issue is with the SRV records. I tried the upstream CSI driver and still was running into the same issues...

So played with SRV records for the NFS - updated as follow for now.... (need ot learn more about NFS configurations) will work to make this secure... /srv/nfs *(rw,sync,no_root_squash,insecure)

Storage Class apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: nfs-csi provisioner: nfs.csi.akshci.com parameters: server: x.x.x.x # NFS server endpoint share: /srv/nfs # NFS share path reclaimPolicy: Retain volumeBindingMode: Immediate mountOptions:

• nfsvers=4