search

kubernetes-sigs / azurefile-csi-driver

Azure File CSI Driver
Apache License 2.0
147 stars 136 forks source link

Volume ID already exists issue with statically provisioned Azure NFS File Shares #1955

Open ashwajce opened 5 days ago

ashwajce commented 5 days ago

What happened: In an AKS environment, mounting a large NFS file share (12Ti, 11.45M files) to a pod causes significant delays, with the pod staying in "ContainerCreating" status for about 3 days. This occurs when syncing data via a linux vm on the nfs share, after sync the nfs share does mount in the pv and pvc correctly, pod does not start up due to operation on current pv already exists

What you expected to happen: Pod starts with the volume attached immediately

How to reproduce it:

  1. have a storage count
  2. expose a file share under that storage account approx 1Ti with data with NFS
  3. mount the disk under a linux vm
  4. provision data via the linux vm (have considerable amount of files in the storage account eg 2.4M)
  5. on aks provision the pv and attach the pvc in a namespace eg:

kubectl apply -f persistent-volumes.yaml --namespace <masked>

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: file.csi.azure.com
  name: pv-<masked>-shared-home
spec:
  capacity:
    storage: 1Ti
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: <masked>-shared
  mountOptions:
    - rsize=1048576 # for recommended values see https://docs.microsoft.com/en-us/azure/storage/files/storage-troubleshoot-linux-file-connection-problems#troubleshoot-mount-issuescheck https://learn.microsoft.com/en-us/azure/storage/files/storage-files-how-to-mount-nfs-shares?tabs=portal#mount-options
    - wsize=1048576
  csi:
    driver: file.csi.azure.com
    readOnly: false
    volumeHandle: pv-<masked>-shared-home
    volumeAttributes:
      resourceGroup: rg-app
      storageAccount: <masked>
      shareName: shared-home
      server: <masked>.privatelink.file.core.windows.net
      protocol: nfs
      skuName: Premium_LRS

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: <masked>-shared-home
spec:
  storageClassName: <masked>-shared
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Ti

Run the helm chart with pod

Actual result: describe pod msg:

Warning  FailedMount  2m24s (x857 over 28h)  kubelet  MountVolume.MountDevice failed for volume "pv-shared-home" : rpc error: code = Aborted desc = An operation with the given Volume ID pv-shared-home already exists

Anything else we need to know?: Issue is occurs also after we have synced data (rsync to linux vm in azure, with nfs share mounted) into this NFS share.

Once the PV has been mounted and no operations are ongoing the PV and PVC can be removed. 2nd run the pvc is immediately mounted and available. Even if we switch from cluster its fine.

Environment:

andyzhangx commented 20 hours ago

@ashwajce could you provide the kubelet logs on that node in problem? have you set any securityContext in pod? this issue could be related to slow chown operation if you set fsGroup in securityContext in pod, one workaround is set fsGroupChangePolicy: None in pv

fsGroupChangePolicy: indicates how volume's ownership will be changed by the driver, pod securityContext.fsGroupChangePolicy is ignored

https://github.com/kubernetes-sigs/azurefile-csi-driver/blob/master/docs/driver-parameters.md