no such file or directory on StatefulSet with volumeClaimTemplates and fsGroup securityContext

What happened: While creating a StatefulSet with a volumeClaimTemplates targeting an azuredisk volume and with fsGroup securityContext set, the pod remain in ContainerCreating because of

  Normal   SuccessfulAttachVolume  18s              attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-f9316692-5adb-4ae8-8afb-e7a0b98341da"
  Warning  FailedMount             1s (x2 over 2s)  kubelet                  MountVolume.SetUp failed for volume "pvc-f9316692-5adb-4ae8-8afb-e7a0b98341da" : applyFSGroup failed for vol /subscriptions/96452422-d6ae-48b7-869d-cd061dbe58f5/resourceGroups/k8saneci/providers/Microsoft.Compute/disks/pvc-f9316692-5adb-4ae8-8afb-e7a0b98341da: lstat /mnt/resource/k8s/kubelet/pods/f4a61c03-4009-427e-a225-7517151924df/volumes/kubernetes.io~csi/pvc-f9316692-5adb-4ae8-8afb-e7a0b98341da/mount: no such file or directory

StatefulSet spec is as follows:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app: thanos-compact
  name: thanos-compact
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: thanos-compact
  serviceName: thanos-compact
  template:
    metadata:
      labels:
        app: thanos-compact
    spec:
      containers:
      - args:
        - compact
        - --wait
        - --log.level=info
        - --log.format=logfmt
        - --objstore.config=$(OBJSTORE_CONFIG)
        - --data-dir=/var/thanos/compact
        - --debug.accept-malformed-index
        - --retention.resolution-raw=28d
        - --retention.resolution-5m=28d
        - --retention.resolution-1h=28d
        - --delete-delay=48h
        - --downsampling.disable
        - --deduplication.replica-label=prometheus_replica
        - --deduplication.replica-label=rule_replica
        env:
        - name: OBJSTORE_CONFIG
          valueFrom:
            secretKeyRef:
              key: thanos.yaml
              name: thanos-objstore-config
        image: thanosio/thanos:v0.23.1
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 8
          httpGet:
            path: /-/healthy
            port: 10902
            scheme: HTTP
          periodSeconds: 30
          successThreshold: 1
          timeoutSeconds: 1
        name: thanos-compact
        ports:
        - containerPort: 10902
          name: http
          protocol: TCP
        readinessProbe:
          failureThreshold: 20
          httpGet:
            path: /-/ready
            port: 10902
            scheme: HTTP
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            cpu: "4"
            memory: 8000Mi
          requests:
            cpu: "2"
            memory: 4000Mi
        volumeMounts:
        - mountPath: /var/thanos/compact
          name: thanos-compact-data
      securityContext:
        fsGroup: 65534
        runAsUser: 65534
      serviceAccountName: thanos-compact
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: thanos-compact-data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 350Gi
      volumeMode: Filesystem

What you expected to happen: Pod associated with statefulset should get up and running and the pvc correctly mounted even when a securityContext like

securityContext:
        fsGroup: 65534
        runAsUser: 65534

is configured at StatefulSet level.

How to reproduce it: Apply above mentioned StatefulSet spec

Anything else we need to know?:

Added fsGroupPolicy to azuredisk CSIDriver in order to avoid permission denied issues like

level=error ts=2021-12-03T10:43:25.323216903Z caller=main.go:131 err="mkdir /var/thanos/compact/compact: permission denied\ncreate working compact directory\nmain.runCompact\n\t/home/circleci/project/cmd/thanos/compact.go:338\nmain.registerCompact.func1\n\t/home/circleci/project/cmd/thanos/compact.go:92\nmain.main\n\t/home/circleci/project/cmd/thanos/main.go:129\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371\npreparing compact command failed\nmain.main\n\t/home/circleci/project/cmd/thanos/main.go:131\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371"

Default storage class configured at cluster level:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  name: default-csi
parameters:
  cachingmode: None
  fsType: ext4
  kind: Managed
  skuName: Standard_LRS
provisioner: disk.csi.azure.com
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true

Environment:

CSI Driver version: v1.9.0

apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
name: disk.csi.azure.com
annotations:
csiDriver: v1.9.0
snapshot: v3.0.3
spec:
attachRequired: true
fsGroupPolicy: File
podInfoOnMount: false
volumeLifecycleModes:
- Persistent

Kubernetes version (use kubectl version):

❯ k version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"b695d79d4f967c403a96986f1750a35eb75e75f1", GitTreeState:"clean", BuildDate:"2021-11-17T15:41:42Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.11", GitCommit:"27522a29febbcc4badac257763044d0d90c11abd", GitTreeState:"clean", BuildDate:"2021-09-15T19:16:25Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}

OS (e.g. from /etc/os-release): FlatcarOS
Kernel (e.g. uname -a): 5.4.92-flatcar
Install tools:
Others:

The volume is properly mounted at node level, and if I manually create the mount directory(e.g. cd /mnt/resource/k8s/kubelet/pods/f4a61c03-4009-427e-a225-7517151924df/volumes/kubernetes.io~csi/pvc-f9316692-5adb-4ae8-8afb-e7a0b98341da && mkdir mount) inside mounted volume location, the pod starts fine.

In azuredisk logs from azuredisk-node pod associated with the node where my StatefulSet pod was scheduled I'm seeing errors like:

I1203 13:17:32.854139       1 utils.go:95] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I1203 13:17:32.854179       1 utils.go:96] GRPC request: {"target_path":"/mnt/resource/k8s/kubelet/pods/93c4a8f4-81f6-4211-b1ff-d66177310394/volumes/kubernetes.io~csi/pvc-1ad4dbfd-f4c3-4af8-80e7-7956f671608e/mount","volume_id":"/subscriptions/96452422-d6ae-48b7-869d-cd061dbe5812/resourceGroups/k8saneci/providers/Microsoft.Compute/disks/pvc-1ad4dbfd-f4c3-4af8-80e7-7956f671608e"}
I1203 13:17:32.854254       1 nodeserver.go:285] NodeUnpublishVolume: unmounting volume /subscriptions/96452422-d6ae-48b7-869d-cd061dbe5812/resourceGroups/k8saneci/providers/Microsoft.Compute/disks/pvc-1ad4dbfd-f4c3-4af8-80e7-7956f671608e on /mnt/resource/k8s/kubelet/pods/93c4a8f4-81f6-4211-b1ff-d66177310394/volumes/kubernetes.io~csi/pvc-1ad4dbfd-f4c3-4af8-80e7-7956f671608e/mount
W1203 13:17:32.854292       1 mount_helper_common.go:34] Warning: Unmount skipped because path does not exist: /mnt/resource/k8s/kubelet/pods/93c4a8f4-81f6-4211-b1ff-d66177310394/volumes/kubernetes.io~csi/pvc-1ad4dbfd-f4c3-4af8-80e7-7956f671608e/mount
I1203 13:17:32.854308       1 nodeserver.go:291] NodeUnpublishVolume: unmount volume /subscriptions/96452422-d6ae-48b7-869d-cd061dbe5812/resourceGroups/k8saneci/providers/Microsoft.Compute/disks/pvc-1ad4dbfd-f4c3-4af8-80e7-7956f671608e on /mnt/resource/k8s/kubelet/pods/93c4a8f4-81f6-4211-b1ff-d66177310394/volumes/kubernetes.io~csi/pvc-1ad4dbfd-f4c3-4af8-80e7-7956f671608e/mount successfully
I1203 13:17:32.854321       1 utils.go:102] GRPC response: {}
I1203 13:18:23.430559       1 utils.go:95] GRPC call: /csi.v1.Node/NodePublishVolume
I1203 13:18:23.430597       1 utils.go:96] GRPC request: {"publish_context":{"LUN":"1"},"staging_target_path":"/mnt/resource/k8s/kubelet/plugins/kubernetes.io/csi/pv/pvc-f9316692-5adb-4ae8-8afb-e7a0b98341da/globalmount","target_path":"/mnt/resource/k8s/kubelet/pods/f4a61c03-4009-427e-a225-7517151924df/volumes/kubernetes.io~csi/pvc-f9316692-5adb-4ae8-8afb-e7a0b98341da/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-f9316692-5adb-4ae8-8afb-e7a0b98341da","csi.storage.k8s.io/pvc/name":"thanos-compact-data-thanos-compact-test-0","csi.storage.k8s.io/pvc/namespace":"monitoring","fsType":"ext4","kind":"Managed","requestedsizegib":"350","skuName":"Standard_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1638519116133-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/96452422-d6ae-48b7-869d-cd061dbe5812/resourceGroups/k8saneci/providers/Microsoft.Compute/disks/pvc-f9316692-5adb-4ae8-8afb-e7a0b98341da"}
I1203 13:18:23.430833       1 nodeserver.go:263] NodePublishVolume: mounting /mnt/resource/k8s/kubelet/plugins/kubernetes.io/csi/pv/pvc-f9316692-5adb-4ae8-8afb-e7a0b98341da/globalmount at /mnt/resource/k8s/kubelet/pods/f4a61c03-4009-427e-a225-7517151924df/volumes/kubernetes.io~csi/pvc-f9316692-5adb-4ae8-8afb-e7a0b98341da/mount
I1203 13:18:23.430859       1 mount_linux.go:175] Mounting cmd (mount) with arguments ( -o bind /mnt/resource/k8s/kubelet/plugins/kubernetes.io/csi/pv/pvc-f9316692-5adb-4ae8-8afb-e7a0b98341da/globalmount /mnt/resource/k8s/kubelet/pods/f4a61c03-4009-427e-a225-7517151924df/volumes/kubernetes.io~csi/pvc-f9316692-5adb-4ae8-8afb-e7a0b98341da/mount)
E1203 13:18:23.433206       1 mount_linux.go:179] Mount failed: exit status 32
Mounting command: mount
Mounting arguments:  -o bind /mnt/resource/k8s/kubelet/plugins/kubernetes.io/csi/pv/pvc-f9316692-5adb-4ae8-8afb-e7a0b98341da/globalmount /mnt/resource/k8s/kubelet/pods/f4a61c03-4009-427e-a225-7517151924df/volumes/kubernetes.io~csi/pvc-f9316692-5adb-4ae8-8afb-e7a0b98341da/mount
Output: mount: /mnt/resource/k8s/kubelet/pods/f4a61c03-4009-427e-a225-7517151924df/volumes/kubernetes.io~csi/pvc-f9316692-5adb-4ae8-8afb-e7a0b98341da/mount: special device /mnt/resource/k8s/kubelet/plugins/kubernetes.io/csi/pv/pvc-f9316692-5adb-4ae8-8afb-e7a0b98341da/globalmount does not exist.

E1203 13:18:23.433249       1 utils.go:100] GRPC error: rpc error: code = Internal desc = Could not mount "/mnt/resource/k8s/kubelet/plugins/kubernetes.io/csi/pv/pvc-f9316692-5adb-4ae8-8afb-e7a0b98341da/globalmount" at "/mnt/resource/k8s/kubelet/pods/f4a61c03-4009-427e-a225-7517151924df/volumes/kubernetes.io~csi/pvc-f9316692-5adb-4ae8-8afb-e7a0b98341da/mount": mount failed: exit status 32
Mounting command: mount
Mounting arguments:  -o bind /mnt/resource/k8s/kubelet/plugins/kubernetes.io/csi/pv/pvc-f9316692-5adb-4ae8-8afb-e7a0b98341da/globalmount /mnt/resource/k8s/kubelet/pods/f4a61c03-4009-427e-a225-7517151924df/volumes/kubernetes.io~csi/pvc-f9316692-5adb-4ae8-8afb-e7a0b98341da/mount
Output: mount: /mnt/resource/k8s/kubelet/pods/f4a61c03-4009-427e-a225-7517151924df/volumes/kubernetes.io~csi/pvc-f9316692-5adb-4ae8-8afb-e7a0b98341da/mount: special device /mnt/resource/k8s/kubelet/plugins/kubernetes.io/csi/pv/pvc-f9316692-5adb-4ae8-8afb-e7a0b98341da/globalmount does not exist.

The disk(sdd) seems to be successfully attached to the node based on

NAME    MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda       8:0    0   200G  0 disk
|-sda1    8:1    0   128M  0 part  /boot
|-sda2    8:2    0     2M  0 part
|-sda3    8:3    0     1G  0 part
| `-usr 254:0    0  1016M  1 crypt /usr
|-sda4    8:4    0     1G  0 part
|-sda6    8:6    0   128M  0 part  /usr/share/oem
|-sda7    8:7    0    64M  0 part
`-sda9    8:9    0 197.7G  0 part  /
sdb       8:16   0    64G  0 disk
`-sdb1    8:17   0    64G  0 part  /mnt/resource
sdd       8:48   0   350G  0 disk
sr0      11:0    1   634K  0 rom

Also, globalmount directory referenced in logs exists too:

vmss-agent-workery-test-nhkpp000002 # ls /mnt/resource/k8s/kubelet/plugins/kubernetes.io/csi/pv/pvc-f9316692-5adb-4ae8-8afb-e7a0b98341da
globalmount  vol_data.json

what's your node driver config, by default it's https://github.com/kubernetes-sigs/azuredisk-csi-driver/blob/c0173ce3ffbcf53f0eee398fe34cc0ecca500f60/deploy/csi-azuredisk-node.yaml#L124-L128

and you need to config like following:

/bin/mount --make-shared /mnt/resource/k8s/kubelet

Node driver pod has:

volumes:
- hostPath:
    path: /mnt/resource/kubelet/plugins/disk.csi.azure.com
    type: DirectoryOrCreate
  name: socket-dir
- hostPath:
    path: /mnt/resource/kubelet/
    type: DirectoryOrCreate
  name: mountpoint-dir
- hostPath:
    path: /mnt/resource/kubelet/plugins_registry/
    type: DirectoryOrCreate
  name: registration-dir
- hostPath:
    path: /etc/kubernetes/
    type: DirectoryOrCreate
  name: azure-cred
- hostPath:
    path: /dev
    type: Directory
  name: device-dir
- hostPath:
    path: /sys/bus/scsi/devices
    type: Directory
  name: sys-devices-dir
- hostPath:
    path: /sys/class/scsi_host/
    type: Directory
  name: scsi-host-dir

And the azuredisk-csi container from node driver pod has:

volumeMounts:
- mountPath: /csi
  name: socket-dir
- mountPath: /var/lib/kubelet/
  mountPropagation: Bidirectional
  name: mountpoint-dir
- mountPath: /etc/kubernetes/
  name: azure-cred
- mountPath: /dev
  name: device-dir
- mountPath: /sys/bus/scsi/devices
  name: sys-devices-dir
- mountPath: /sys/class/scsi_host/
  name: scsi-host-dir

I'm using the temporary disk available in Azure VMs. Afaik this temporary disk is automatically configured in the Microsoft Azure Linux Guest agent configuration

# df -h /mnt/resource
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1        63G  124M   60G   1% /mnt/resource

And based on the propagation config it seems to be already mounted as shared:

# findmnt -o TARGET,PROPAGATION /mnt/resource
TARGET        PROPAGATION
/mnt/resource shared

your kubelet path is /mnt/resource/k8s/kubelet, not /mnt/resource/kubelet/?

Kubelet is configured with --root-dir=/mnt/resource/kubelet

I'm getting the same error even with a simple deployment like

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
  labels:
    app: standard-csi-storage-class
  name: standard-csi-storage-class
  namespace: menagerie
spec:
  replicas: 1
  selector:
    matchLabels:
      app: standard-csi-storage-class
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: standard-csi-storage-class
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                - linux
      containers:
      - command:
        - tail
        - -f
        - /dev/null
        image: alpine:3.10.2
        name: app
        volumeMounts:
        - mountPath: /mnt/pv
          name: pv
      volumes:
      - name: pv
        persistentVolumeClaim:
          claimName: standard-csi-storage-class

With PVC spec as:

 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
   labels:
     app: standard-csi-storage-class
   name: standard-csi-storage-class
   namespace: menagerie
 spec:
   accessModes:
   - ReadWriteOnce
   resources:
     requests:
       storage: 1Gi
   storageClassName: standard-csi

And in pod events I can see

  Warning  FailedMount             5m4s (x12 over 26m)  kubelet                  MountVolume.SetUp failed for volume "pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c" : rpc error: code = Internal desc = Could not mount "/mnt/resource/kubelet/plugins/kubernetes.io/csi/pv/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/globalmount" at "/mnt/resource/kubelet/pods/899e3795-58a3-4f10-83da-8875717918de/volumes/kubernetes.io~csi/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/mount": mount failed: exit status 32
Mounting command: mount
Mounting arguments:  -o bind /mnt/resource/kubelet/plugins/kubernetes.io/csi/pv/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/globalmount /mnt/resource/kubelet/pods/899e3795-58a3-4f10-83da-8875717918de/volumes/kubernetes.io~csi/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/mount
Output: mount: /mnt/resource/kubelet/pods/899e3795-58a3-4f10-83da-8875717918de/volumes/kubernetes.io~csi/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/mount: special device /mnt/resource/kubelet/plugins/kubernetes.io/csi/pv/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/globalmount does not exist.
  Warning  FailedMount  68s (x5 over 21m)  kubelet  Unable to attach or mount volumes: unmounted volumes=[pv], unattached volumes=[pv default-token-fx6ms]: timed out waiting for the condition

Kubelet logs:

Dec 05 09:58:53 vmss-agent-worker-test-zuonx000000 kubelet[3120]: I1205 09:58:53.982394    3120 operation_generator.go:565] MountVolume.WaitForAttach succeeded for volume "pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c" (UniqueName: "kubernetes.io/csi/disk.csi.azure.com^/subscriptions/96452422-d6ae-48b7-869d-cd061dbe58f5/resourceGroups/k8saneci/providers/Microsoft.Compute/disks/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c") pod "standard-csi-storage-class-67b98f5d9b-pd95m" (UID: "899e3795-58a3-4f10-83da-8875717918de") DevicePath "csi-91426811cb8fe86e88e7b4f158a1c978271cbcde405831fc3c51dadaa393fcb6"
Dec 05 09:58:53 vmss-agent-worker-test-zuonx000000 kubelet[3120]: I1205 09:58:53.982445    3120 clientconn.go:106] parsed scheme: ""
Dec 05 09:58:53 vmss-agent-worker-test-zuonx000000 kubelet[3120]: I1205 09:58:53.982455    3120 clientconn.go:106] scheme "" not registered, fallback to default scheme
Dec 05 09:58:53 vmss-agent-worker-test-zuonx000000 kubelet[3120]: I1205 09:58:53.982473    3120 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/mnt/resource/kubelet/plugins/disk.csi.azure.com/csi.sock  <nil> 0 <nil>}] <nil> <nil>}
Dec 05 09:58:53 vmss-agent-worker-test-zuonx000000 kubelet[3120]: I1205 09:58:53.982482    3120 clientconn.go:948] ClientConn switching balancer to "pick_first"
Dec 05 09:58:53 vmss-agent-worker-test-zuonx000000 kubelet[3120]: I1205 09:58:53.982524    3120 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
Dec 05 09:58:53 vmss-agent-worker-test-zuonx000000 kubelet[3120]: I1205 09:58:53.982598    3120 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc00304adb0, {CONNECTING <nil>}
Dec 05 09:58:53 vmss-agent-worker-test-zuonx000000 kubelet[3120]: I1205 09:58:53.982976    3120 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc00304adb0, {READY <nil>}
Dec 05 09:58:53 vmss-agent-worker-test-zuonx000000 kubelet[3120]: I1205 09:58:53.983591    3120 controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
Dec 05 09:58:53 vmss-agent-worker-test-zuonx000000 kubelet[3120]: I1205 09:58:53.987773    3120 clientconn.go:106] parsed scheme: ""
Dec 05 09:58:53 vmss-agent-worker-test-zuonx000000 kubelet[3120]: I1205 09:58:53.987791    3120 clientconn.go:106] scheme "" not registered, fallback to default scheme
Dec 05 09:58:53 vmss-agent-worker-test-zuonx000000 kubelet[3120]: I1205 09:58:53.987806    3120 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/mnt/resource/kubelet/plugins/disk.csi.azure.com/csi.sock  <nil> 0 <nil>}] <nil> <nil>}
Dec 05 09:58:53 vmss-agent-worker-test-zuonx000000 kubelet[3120]: I1205 09:58:53.987813    3120 clientconn.go:948] ClientConn switching balancer to "pick_first"
Dec 05 09:58:53 vmss-agent-worker-test-zuonx000000 kubelet[3120]: I1205 09:58:53.987843    3120 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick
Dec 05 09:58:53 vmss-agent-worker-test-zuonx000000 kubelet[3120]: I1205 09:58:53.987934    3120 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc0047a8ae0, {CONNECTING <nil>}
Dec 05 09:58:53 vmss-agent-worker-test-zuonx000000 kubelet[3120]: I1205 09:58:53.988119    3120 balancer_conn_wrappers.go:78] pickfirstBalancer: HandleSubConnStateChange: 0xc0047a8ae0, {READY <nil>}
Dec 05 09:58:53 vmss-agent-worker-test-zuonx000000 kubelet[3120]: I1205 09:58:53.990948    3120 controlbuf.go:508] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
Dec 05 09:58:53 vmss-agent-worker-test-zuonx000000 kubelet[3120]: E1205 09:58:53.991116    3120 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/disk.csi.azure.com^/subscriptions/96452422-d6ae-48b7-869d-cd061dbe58f5/resourceGroups/k8saneci/providers/Microsoft.Compute/disks/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c podName: nodeName:}" failed. No retries permitted until 2021-12-05 10:00:55.991075865 +0000 UTC m=+3331.467705298 (durationBeforeRetry 2m2s). Error: "MountVolume.SetUp failed for volume \"pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c\" (UniqueName: \"kubernetes.io/csi/disk.csi.azure.com^/subscriptions/96452422-d6ae-48b7-869d-cd061dbe58f5/resourceGroups/k8saneci/providers/Microsoft.Compute/disks/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c\") pod \"standard-csi-storage-class-67b98f5d9b-pd95m\" (UID: \"899e3795-58a3-4f10-83da-8875717918de\") : rpc error: code = Internal desc = Could not mount \"/mnt/resource/kubelet/plugins/kubernetes.io/csi/pv/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/globalmount\" at \"/mnt/resource/kubelet/pods/899e3795-58a3-4f10-83da-8875717918de/volumes/kubernetes.io~csi/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/mount\": mount failed: exit status 32\nMounting command: mount\nMounting arguments:  -o bind /mnt/resource/kubelet/plugins/kubernetes.io/csi/pv/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/globalmount /mnt/resource/kubelet/pods/899e3795-58a3-4f10-83da-8875717918de/volumes/kubernetes.io~csi/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/mount\nOutput: mount: /mnt/resource/kubelet/pods/899e3795-58a3-4f10-83da-8875717918de/volumes/kubernetes.io~csi/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/mount: special device /mnt/resource/kubelet/plugins/kubernetes.io/csi/pv/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/globalmount does not exist.\n"

Not sure why it is complaining that

special device /mnt/resource/kubelet/plugins/kubernetes.io/csi/pv/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/globalmount does not exist

when is fact that directory already exists:

# ls /mnt/resource/kubelet/plugins/kubernetes.io/csi/pv/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c
globalmount  vol_data.json

I've tried to manually mount it based on the mount command found in error log and I got:

# mount -o bind /mnt/resource/kubelet/plugins/kubernetes.io/csi/pv/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/globalmount /mnt/resource/kubelet/pods/899e3795-58a3-4f10-83da-8875717918de/volumes/kubernetes.io~csi/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/mount
mount: /mnt/resource/kubelet/pods/899e3795-58a3-4f10-83da-8875717918de/volumes/kubernetes.io~csi/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/mount: mount point does not exist.

most probably because /mnt/resource/kubelet/pods/899e3795-58a3-4f10-83da-8875717918de/volumes/kubernetes.io~csi/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c doesn't exist.

Hmm, I just spun up a new VMSS and configured kubelet with default location, /var/lib/kubelet, and deployed a second azuredisk-csi node daemonset with volumes configured according to default kubelet location. Once I moved the pod to run on a VM from this new VMS, kubelet and node driver were able to successfully mount the volume and start the pod.

Then I moved the pod on a new node form VMSS that has the kubelet configured with--root-dir=/mnt/resource/kubelet, and now I'm getting:

Dec 05 10:51:50 vmss-agent-worker-test-zuonx000003 kubelet[3127]: E1205 10:51:50.675846    3127 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/disk.csi.azure.com^/subscriptions/96452422-d6ae-48b7-869d-cd061dbe58f5/resourceGroups/k8saneci/providers/Microsoft.Compute/disks/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c podName: nodeName:}" failed. No retries permitted until 2021-12-05 10:53:52.675816016 +0000 UTC m=+863.093302177 (durationBeforeRetry 2m2s). Error: "MountVolume.SetUp failed for volume \"pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c\" (UniqueName: \"kubernetes.io/csi/disk.csi.azure.com^/subscriptions/96452422-d6ae-48b7-869d-cd061dbe58f5/resourceGroups/k8saneci/providers/Microsoft.Compute/disks/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c\") pod \"standard-csi-storage-class-5f858686c6-7rtpv\" (UID: \"b2cd5181-a707-4ba4-befe-e1c594317a04\") : applyFSGroup failed for vol /subscriptions/96452422-d6ae-48b7-869d-cd061dbe58f5/resourceGroups/k8saneci/providers/Microsoft.Compute/disks/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c: lstat /mnt/resource/kubelet/pods/b2cd5181-a707-4ba4-befe-e1c594317a04/volumes/kubernetes.io~csi/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/mount: no such file or directory"

The location exists, but there is no mount directory:

# ls /mnt/resource/kubelet/pods/b2cd5181-a707-4ba4-befe-e1c594317a04/volumes/kubernetes.io~csi/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/
vol_data.json

I suspect it has something to do with the custom kubelet configured location, /mnt/resource/kubelet.

Not sure if it helps, but here are some logs from node driver azuredisk-csi container:

I1205 10:41:32.719633       1 utils.go:95] GRPC call: /csi.v1.Node/NodeStageVolume
I1205 10:41:32.719661       1 utils.go:96] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/mnt/resource/kubelet/plugins/kubernetes.io/csi/pv/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":1}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c","csi.storage.k8s.io/pvc/name":"standard-csi-storage-class","csi.storage.k8s.io/pvc/namespace":"menagerie","kind":"Managed","requestedsizegib":"1","skuName":"Standard_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1638695746326-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/96452422-d6ae-48b7-869d-cd061dbe58f5/resourceGroups/k8saneci/providers/Microsoft.Compute/disks/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c"}
I1205 10:41:33.916068       1 azure_common_linux.go:177] azureDisk - found /dev/disk/azure/scsi1/lun0 by sdc under /dev/disk/azure/scsi1/
I1205 10:41:33.916122       1 nodeserver.go:112] NodeStageVolume: perf optimization is disabled for /dev/disk/azure/scsi1/lun0. perfProfile none accountType Standard_LRS
I1205 10:41:33.916661       1 nodeserver.go:153] NodeStageVolume: formatting /dev/disk/azure/scsi1/lun0 and mounting at /mnt/resource/kubelet/plugins/kubernetes.io/csi/pv/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/globalmount with mount options([])
I1205 10:41:33.916714       1 mount_linux.go:446] Attempting to determine if disk "/dev/disk/azure/scsi1/lun0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/disk/azure/scsi1/lun0])
I1205 10:41:34.079912       1 mount_linux.go:449] Output: "DEVNAME=/dev/disk/azure/scsi1/lun0\nTYPE=ext4\n"
I1205 10:41:34.079945       1 mount_linux.go:340] Checking for issues with fsck on disk: /dev/disk/azure/scsi1/lun0
I1205 10:41:34.232519       1 mount_linux.go:436] Attempting to mount disk /dev/disk/azure/scsi1/lun0 in ext4 format at /mnt/resource/kubelet/plugins/kubernetes.io/csi/pv/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/globalmount
I1205 10:41:34.232566       1 mount_linux.go:175] Mounting cmd (mount) with arguments (-t ext4 -o defaults /dev/disk/azure/scsi1/lun0 /mnt/resource/kubelet/plugins/kubernetes.io/csi/pv/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/globalmount)
I1205 10:41:34.263615       1 nodeserver.go:159] NodeStageVolume: format /dev/disk/azure/scsi1/lun0 and mounting at /mnt/resource/kubelet/plugins/kubernetes.io/csi/pv/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/globalmount successfully.
I1205 10:41:34.263651       1 utils.go:102] GRPC response: {}
I1205 10:41:34.275329       1 utils.go:95] GRPC call: /csi.v1.Node/NodePublishVolume
I1205 10:41:34.275356       1 utils.go:96] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/mnt/resource/kubelet/plugins/kubernetes.io/csi/pv/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/globalmount","target_path":"/mnt/resource/kubelet/pods/b2cd5181-a707-4ba4-befe-e1c594317a04/volumes/kubernetes.io~csi/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":1}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c","csi.storage.k8s.io/pvc/name":"standard-csi-storage-class","csi.storage.k8s.io/pvc/namespace":"menagerie","kind":"Managed","requestedsizegib":"1","skuName":"Standard_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1638695746326-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/96452422-d6ae-48b7-869d-cd061dbe58f5/resourceGroups/k8saneci/providers/Microsoft.Compute/disks/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c"}
I1205 10:41:34.275863       1 nodeserver.go:263] NodePublishVolume: mounting /mnt/resource/kubelet/plugins/kubernetes.io/csi/pv/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/globalmount at /mnt/resource/kubelet/pods/b2cd5181-a707-4ba4-befe-e1c594317a04/volumes/kubernetes.io~csi/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/mount
I1205 10:41:34.275904       1 mount_linux.go:175] Mounting cmd (mount) with arguments ( -o bind /mnt/resource/kubelet/plugins/kubernetes.io/csi/pv/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/globalmount /mnt/resource/kubelet/pods/b2cd5181-a707-4ba4-befe-e1c594317a04/volumes/kubernetes.io~csi/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/mount)
I1205 10:41:34.278311       1 mount_linux.go:175] Mounting cmd (mount) with arguments ( -o bind,remount /mnt/resource/kubelet/plugins/kubernetes.io/csi/pv/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/globalmount /mnt/resource/kubelet/pods/b2cd5181-a707-4ba4-befe-e1c594317a04/volumes/kubernetes.io~csi/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/mount)
I1205 10:41:34.280485       1 nodeserver.go:268] NodePublishVolume: mount /mnt/resource/kubelet/plugins/kubernetes.io/csi/pv/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/globalmount at /mnt/resource/kubelet/pods/b2cd5181-a707-4ba4-befe-e1c594317a04/volumes/kubernetes.io~csi/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/mount successfully
I1205 10:41:34.280508       1 utils.go:102] GRPC response: {}
I1205 10:41:34.824564       1 utils.go:95] GRPC call: /csi.v1.Node/NodePublishVolume
I1205 10:41:34.824587       1 utils.go:96] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/mnt/resource/kubelet/plugins/kubernetes.io/csi/pv/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/globalmount","target_path":"/mnt/resource/kubelet/pods/b2cd5181-a707-4ba4-befe-e1c594317a04/volumes/kubernetes.io~csi/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":1}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c","csi.storage.k8s.io/pvc/name":"standard-csi-storage-class","csi.storage.k8s.io/pvc/namespace":"menagerie","kind":"Managed","requestedsizegib":"1","skuName":"Standard_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1638695746326-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/96452422-d6ae-48b7-869d-cd061dbe58f5/resourceGroups/k8saneci/providers/Microsoft.Compute/disks/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c"}
I1205 10:41:34.828736       1 nodeserver.go:551] already mounted to target /mnt/resource/kubelet/pods/b2cd5181-a707-4ba4-befe-e1c594317a04/volumes/kubernetes.io~csi/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/mount
I1205 10:41:34.828774       1 nodeserver.go:258] NodePublishVolume: already mounted on target /mnt/resource/kubelet/pods/b2cd5181-a707-4ba4-befe-e1c594317a04/volumes/kubernetes.io~csi/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/mount
I1205 10:41:34.828787       1 utils.go:102] GRPC response: {}

Below logs are logged periodically(based on the timestamps maybe using an exponential algorithm):

I1205 10:43:42.321078       1 utils.go:95] GRPC call: /csi.v1.Node/NodePublishVolume
I1205 10:43:42.321099       1 utils.go:96] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/mnt/resource/kubelet/plugins/kubernetes.io/csi/pv/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/globalmount","target_path":"/mnt/resource/kubelet/pods/b2cd5181-a707-4ba4-befe-e1c594317a04/volumes/kubernetes.io~csi/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":1}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c","csi.storage.k8s.io/pvc/name":"standard-csi-storage-class","csi.storage.k8s.io/pvc/namespace":"menagerie","kind":"Managed","requestedsizegib":"1","skuName":"Standard_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1638695746326-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/96452422-d6ae-48b7-869d-cd061dbe58f5/resourceGroups/k8saneci/providers/Microsoft.Compute/disks/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c"}
I1205 10:43:42.321395       1 nodeserver.go:551] already mounted to target /mnt/resource/kubelet/pods/b2cd5181-a707-4ba4-befe-e1c594317a04/volumes/kubernetes.io~csi/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/mount
I1205 10:43:42.321414       1 nodeserver.go:258] NodePublishVolume: already mounted on target /mnt/resource/kubelet/pods/b2cd5181-a707-4ba4-befe-e1c594317a04/volumes/kubernetes.io~csi/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/mount
I1205 10:43:42.321421       1 utils.go:102] GRPC response: {}

Even if the node driver keeps saying that /mnt/resource/kubelet/pods/b2cd5181-a707-4ba4-befe-e1c594317a04/volumes/kubernetes.io~csi/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/mount is already mounted, the /mount directory doesn't exist

# ls /mnt/resource/kubelet/pods/b2cd5181-a707-4ba4-befe-e1c594317a04/volumes/kubernetes.io~csi/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/
vol_data.json

I restarted the node driver pod and after restart I'm getting other strange logs like:

I1205 11:44:45.932806       1 utils.go:95] GRPC call: /csi.v1.Node/NodePublishVolume
I1205 11:44:45.932835       1 utils.go:96] GRPC request: {"publish_context":{"LUN":"0"},"staging_target_path":"/mnt/resource/kubelet/plugins/kubernetes.io/csi/pv/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/globalmount","target_path":"/mnt/resource/kubelet/pods/b2cd5181-a707-4ba4-befe-e1c594317a04/volumes/kubernetes.io~csi/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/mount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":1}},"volume_context":{"cachingmode":"None","csi.storage.k8s.io/pv/name":"pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c","csi.storage.k8s.io/pvc/name":"standard-csi-storage-class","csi.storage.k8s.io/pvc/namespace":"menagerie","kind":"Managed","requestedsizegib":"1","skuName":"Standard_LRS","storage.kubernetes.io/csiProvisionerIdentity":"1638695746326-8081-disk.csi.azure.com"},"volume_id":"/subscriptions/96452422-d6ae-48b7-869d-cd061dbe58f5/resourceGroups/k8saneci/providers/Microsoft.Compute/disks/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c"}
I1205 11:44:45.933707       1 nodeserver.go:263] NodePublishVolume: mounting /mnt/resource/kubelet/plugins/kubernetes.io/csi/pv/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/globalmount at /mnt/resource/kubelet/pods/b2cd5181-a707-4ba4-befe-e1c594317a04/volumes/kubernetes.io~csi/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/mount
I1205 11:44:45.933763       1 mount_linux.go:175] Mounting cmd (mount) with arguments ( -o bind /mnt/resource/kubelet/plugins/kubernetes.io/csi/pv/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/globalmount /mnt/resource/kubelet/pods/b2cd5181-a707-4ba4-befe-e1c594317a04/volumes/kubernetes.io~csi/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/mount)
E1205 11:44:45.936281       1 mount_linux.go:179] Mount failed: exit status 32
Mounting command: mount
Mounting arguments:  -o bind /mnt/resource/kubelet/plugins/kubernetes.io/csi/pv/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/globalmount /mnt/resource/kubelet/pods/b2cd5181-a707-4ba4-befe-e1c594317a04/volumes/kubernetes.io~csi/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/mount
Output: mount: /mnt/resource/kubelet/pods/b2cd5181-a707-4ba4-befe-e1c594317a04/volumes/kubernetes.io~csi/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/mount: special device /mnt/resource/kubelet/plugins/kubernetes.io/csi/pv/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/globalmount does not exist.

E1205 11:44:45.936331       1 utils.go:100] GRPC error: rpc error: code = Internal desc = Could not mount "/mnt/resource/kubelet/plugins/kubernetes.io/csi/pv/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/globalmount" at "/mnt/resource/kubelet/pods/b2cd5181-a707-4ba4-befe-e1c594317a04/volumes/kubernetes.io~csi/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/mount": mount failed: exit status 32
Mounting command: mount
Mounting arguments:  -o bind /mnt/resource/kubelet/plugins/kubernetes.io/csi/pv/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/globalmount /mnt/resource/kubelet/pods/b2cd5181-a707-4ba4-befe-e1c594317a04/volumes/kubernetes.io~csi/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/mount
Output: mount: /mnt/resource/kubelet/pods/b2cd5181-a707-4ba4-befe-e1c594317a04/volumes/kubernetes.io~csi/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/mount: special device /mnt/resource/kubelet/plugins/kubernetes.io/csi/pv/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/globalmount does not exist.

This time it cannot find /mnt/resource/kubelet/plugins/kubernetes.io/csi/pv/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/globalmount, but I manually checked and this directory exists on the VM:

# ls /mnt/resource/kubelet/plugins/kubernetes.io/csi/pv/pvc-3c2607e6-9221-4dba-8ed3-f392e32cb78c/
globalmount  vol_data.json

Also, I was able to successfully deploy the above mentioned deployment on the same VM that has the kubelet root-dir location configured to /mnt/resource/kubelet using a PVC created from a storageclass that is using the in-tree plugin:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: standard
  labels:
    kubernetes.io/cluster-service: "true"
provisioner: kubernetes.io/azure-disk
parameters:
  kind: Managed
  storageaccounttype: Standard_LRS
  cachingmode: None
allowVolumeExpansion: true
volumeBindingMode: Immediate

PVC spec:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  labels:
    app: standard-storage-class
  name: standard-storage-class
  namespace: menagerie
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: standard

❯ k get po -n menagerie
standard-csi-storage-class-5f858686c6-7rtpv       0/1     ContainerCreating   0          48m
standard-storage-class-fcb56d984-tmq8j            1/1     Running             0          124m

I think I found the problem, mountPath for mountpoint-dir volume mount in azuredisk node driver container should also be adjusted to be in sync with host kubelet location(from /var/lib/kubelet/ to /mnt/resource/kubelet/)

- mountPath: /mnt/resource/kubelet/
  mountPropagation: Bidirectional
  name: mountpoint-dir

I'll close the issue since case is solved now. Thank you for your help!

kubernetes-sigs / azuredisk-csi-driver

no such file or directory on StatefulSet with volumeClaimTemplates and fsGroup securityContext #1104