can't mount flex volume to pod: timeout expired waiting for volumes to attach or mount for pod

[bhardwahnitish19]

Describe the bug I am trying to use flexvolume with pod identity. Even through kubelet tries multiple times, I am getting same time(tried for almost 7-8 times) and the pod status is stuck at ContainerCreating.

Steps To Reproduce

    ---
    apiVersion: v1
    kind: Pod
    metadata:
      name: mysqldb
      labels:
        k8s-app: mysql-label
        aadpodidbinding: mysql-azure-identity-binding
    spec:
      containers:
      - name: mysqldb
        image: mysql:5.7.26
        env:
          - { name: "MYSQL_ROOT_PASSWORD", value: "ansible@1234" }
          - { name: "MYSQL_ROOT_HOST", value: "%" }
        ports:
          - containerPort: 3306
        volumeMounts:
          - { mountPath: "/var/lib/mysql", name: "mysql-persistent-storage" }
          - { mountPath: "/kvmnt", name: "keyvault-store-volume", readOnly: true }
      restartPolicy: OnFailure
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pv-claims
      - name: keyvault-store-volume
        flexVolume:
          driver: "azure/kv"
          options:
            usepodidentity: "true"   
            keyvaultname: "my-keyvault-name"              
            keyvaultobjectnames: "pod-cert;pod-cert"       
            keyvaultobjecttypes: key;secret    
            resourcegroup: "my-resourcegroup"             
            subscriptionid: "subId"           
            tenantid: "tenantid"

Expected behavior Volume must be mounted after couple of re-tries Key Vault FlexVolume version

Access mode: service principal or pod identity pod identity Kubernetes version 1.13.10 Additional context

Events:
  Type     Reason       Age                 From                               Message
  ----     ------       ----                ----                               -------
  Normal   Scheduled    45m                 default-scheduler                  Successfully assigned default/mysqldb to aks-agentpool-50967501-7
  Warning  FailedMount  38m                 kubelet, aks-agentpool-50967501-7  MountVolume.SetUp failed for volume "keyvault-store-volume" : mount command failed, status: Failure, reason: /etc/kubernetes/volumeplugins/azure~kv/azurekeyvault-flexvolume failed, F0416 06:16:27.679613 59110 main.go:82] [error] : failed to get keyvaultClient: failed to get key vault token: failed to get service principal token: nmi response failed with status code: 403
  Warning  FailedMount  31m                 kubelet, aks-agentpool-50967501-7  MountVolume.SetUp failed for volume "keyvault-store-volume" : mount command failed, status: Failure, reason: /etc/kubernetes/volumeplugins/azure~kv/azurekeyvault-flexvolume failed, F0416 06:23:43.399975 63185 main.go:82] [error] : failed to get keyvaultClient: failed to get key vault token: failed to get service principal token: nmi response failed with status code: 403
  Warning  FailedMount  24m                 kubelet, aks-agentpool-50967501-7  MountVolume.SetUp failed for volume "keyvault-store-volume" : mount command failed, status: Failure, reason: /etc/kubernetes/volumeplugins/azure~kv/azurekeyvault-flexvolume failed, F0416 06:30:59.521355 67385 main.go:82] [error] : failed to get keyvaultClient: failed to get key vault token: failed to get service principal token: nmi response failed with status code: 403
  Warning  FailedMount  16m                 kubelet, aks-agentpool-50967501-7  MountVolume.SetUp failed for volume "keyvault-store-volume" : mount command failed, status: Failure, reason: /etc/kubernetes/volumeplugins/azure~kv/azurekeyvault-flexvolume failed, F0416 06:38:16.627236 71671 main.go:82] [error] : failed to get keyvaultClient: failed to get key vault token: failed to get service principal token: nmi response failed with status code: 403
  Warning  FailedMount  9m27s               kubelet, aks-agentpool-50967501-7  MountVolume.SetUp failed for volume "keyvault-store-volume" : mount command failed, status: Failure, reason: /etc/kubernetes/volumeplugins/azure~kv/azurekeyvault-flexvolume failed, F0416 06:45:35.767178 85127 main.go:82] [error] : failed to get keyvaultClient: failed to get key vault token: failed to get service principal token: nmi response failed with status code: 403
  Warning  FailedMount  2m4s                kubelet, aks-agentpool-50967501-7  MountVolume.SetUp failed for volume "keyvault-store-volume" : mount command failed, status: Failure, reason: /etc/kubernetes/volumeplugins/azure~kv/azurekeyvault-flexvolume failed, F0416 06:52:58.873409 89420 main.go:82] [error] : failed to get keyvaultClient: failed to get key vault token: failed to get service principal token: nmi response failed with status code: 403
  Warning  FailedMount  30s (x20 over 43m)  kubelet, aks-agentpool-50967501-7  Unable to mount volumes for pod "mysqldb_default(id)": timeout expired waiting for volumes to attach or mount for pod "default"/"mysqldb". list of unmounted volumes=[keyvault-store-volume]. list of unattached volumes=[mysql-persistent-storage keyvault-store-volume default-token-xh4x6]

[aramase]

@bhardwahnitish19 Looks like NMI is returning an error while requested for a token. Can you please check the NMI and MIC logs for pod-identity to see why the identity assignment is failing?

[bhardwahnitish19]

@aramase Thanks for your quick response. I am not sure how to check those logs. It would be great if you can point me towards it.

Appreciate your help

[aramase]

@bhardwahnitish19 For the MIC logs -

kubectl get pods --all-namespaces | grep mic

There will be 2 pods for MIC. Check the logs for each pod to see if there are any errors. MIC is the component in pod-identity that assigns the identity the underlying VM/VMSS node. NMI is the component that checks if the identity has been assigned, if pod has access to the identity, fetches and returns the token.

If you don't see any errors in MIC, you can check the NMI pod running on the same node as your application pod to see why it's unable to fetch a token.

kubectl get pods --all-namespaces -o wide | grep nmi. Pick the NMI pod running on the same node as the application.

[bhardwahnitish19]

@aramase Thanks, I will check and update the logs for reference.