unable to mount volume-- (Duplicate issue but other resolutions are not working)

[vivekj11]

I am facing a similar issue related to volume mount. my pod definition --

apiVersion: v1
kind: Pod 
metadata:
    name: sample-pod
    labels:
       aadpodidbinding: "dev_app"
spec:
   restartPolicy: Never
   containers:
   - name: nginx 
     image: nginx:latest
     volumeMounts:
      - name: kv-dev
        mountPath: /kvmount
        readOnly: true
   volumes:
   - name: kv-dev
     flexVolume:
        driver: "azure/kv"
        options:
          usepodidentity: "true"                 
          keyvaultname: "kv-name"        
          keyvaultobjectnames: "key1"                                
          keyvaultobjecttypes: "secret"                                
          keyvaultobjectversions: "version" 
          resourcegroup: "mydevrg"                      
          subscriptionid: "subscription id"     
          tenantid: "tenand id"

Logs from /var/log/kv-driver.log

Wed Apr 10 06:51:10 UTC 2019 ismounted | not mounted
Wed Apr 10 06:51:10 UTC 2019 PODNAME: sample-pod
Wed Apr 10 06:51:10 UTC 2019 mount
Wed Apr 10 06:51:10 UTC 2019 /etc/kubernetes/volumeplugins/azure~kv/azurekeyvault-flexvolume -logtostderr=1 -vaultName=env-dev-secrets -vaultObjectNames=key1 -vaultObjectAliases= -resourceGroup=env-development-keys -dir=/var/lib/kubelet/pods/04759d6f-5b5b-11e9-a910-ea94745bd5eb/volumes/azure~kv/kv-dev -subscriptionId=2aefe---de-a2f7-700b4dd06ad1 -cloudName= -tenantId=fbfbfbnf---03-a8a1e287fa9d -aADClientSecret= -aADClientID= -usePodIdentity=true -podNamespace=default -podName=sample-pod -vaultObjectVersions=96efefefefw20411d8b2712cc8 -vaultObjectTypes=secret
I0410 06:51:10.800266   19396 keyvaultFlexvolumeAdapter.go:32] azurekeyvault-flexvolume 0.0.10
I0410 06:51:10.800365   19396 keyvaultFlexvolumeAdapter.go:41] starting the azurekeyvault-flexvolume, 0.0.10
I0410 06:51:10.800426   19396 oauth.go:135] azure: using pod identity to retrieve token
I0410 06:51:10.810047   19396 oauth.go:135] azure: using pod identity to retrieve token
F0410 06:51:10.812298   19396 main.go:82] [error] : failed to get keyvaultClient: failed to get key vault token: failed to get service principal token: nmi response failed with status code: 403
Wed Apr 10 06:51:10 UTC 2019 umount
Wed Apr 10 06:51:10 UTC 2019 ERROR: {"status": "Failure", "message": "/etc/kubernetes/volumeplugins/azure~kv/azurekeyvault-flexvolume failed, F0410 06:51:10.812298 19396 main.go:82] [error] : failed to get keyvaultClient: failed to get key vault token: failed to get service principal token: nmi response failed with status code: 403 "}
Wed Apr 10 06:53:12 UTC 2019 ismounted | not mounted
Wed Apr 10 06:53:12 UTC 2019 PODNAME: sample-pod
Wed Apr 10 06:53:12 UTC 2019 mount
Wed Apr 10 06:53:12 UTC 2019 /etc/kubernetes/volumeplugins/azure~kv/azurekeyvault-flexvolume -logtostderr=1 -vaultName=env-dev-secrets -vaultObjectNames=key1 -vaultObjectAliases= -resourceGroup=env-development-keys -dir=/var/lib/kubelet/pods/04759d6f-5b5b-11e9-a910-ea94745bd5eb/volumes/azure~kv/kv-dev -subscriptionId=25a1ewrewewr8de-a2f7-700b4dd06ad1 -cloudName= -tenantId=c98d2fc9-defewrer43ca-a603-a8a1e287fa9d -aADClientSecret= -aADClientID= -usePodIdentity=true -podNamespace=default -podName=sample-pod -vaultObjectVersions=96e4c4rewrererer11d8b2712cc8 -vaultObjectTypes=secret
I0410 06:53:12.970569   22037 keyvaultFlexvolumeAdapter.go:32] azurekeyvault-flexvolume 0.0.10
I0410 06:53:12.970663   22037 keyvaultFlexvolumeAdapter.go:41] starting the azurekeyvault-flexvolume, 0.0.10
I0410 06:53:12.970703   22037 oauth.go:135] azure: using pod identity to retrieve token
I0410 06:53:12.981358   22037 oauth.go:135] azure: using pod identity to retrieve token
F0410 06:53:12.984216   22037 main.go:82] [error] : failed to get keyvaultClient: failed to get key vault token: failed to get service principal token: nmi response failed with status code: 403
Wed Apr 10 06:53:12 UTC 2019 umount
Wed Apr 10 06:53:13 UTC 2019 ERROR: {"status": "Failure", "message": "/etc/kubernetes/volumeplugins/azure~kv/azurekeyvault-flexvolume failed, F0410 06:53:12.984216 22037 main.go:82] [error] : failed to get keyvaultClient: failed to get key vault token: failed to get service principal token: nmi response failed with status code: 403 "}

As per the suggestions in other issues, I waited for more that one hour but the volume is not mounting.

Azuredentity definition--

apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentity
metadata:
  name: dev-pod-identity
spec:
  type: 0
  ResourceID: /subscriptions/25a1anc-----d06ad1/resourcegroups/env-dev/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dev_pod_identity
  ClientID: cd8b-------abcd ----e8231e1

Azure binding definition--

apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentityBinding
metadata:
 name: dev-pod-identity-binding
spec:
 AzureIdentity: dev-pod-identity
 Selector: dev_app

Looking for urgent help.

[ritazh]

@vivekj11 From the logs, it seems the issue is related to pod identity. The nmi component is not able to get the access token from the azure identity to access your key vault. Few questions:

1. Can you please check to make sure you are using the latest pod identity deployment yaml: https://github.com/Azure/aad-pod-identity/blob/master/deploy/infra/deployment-rbac.yaml
2. What do you get when you run kubectl get azureassignedidentity
3. Does your azure identity have all the necessary access to key vault? Please double check item 3 under https://github.com/Azure/kubernetes-keyvault-flexvol#option-2---pod-identity

[vivekj11]

@ritazh Thank you for the quick response. I guess I had missed something during my last setup. I did the setup once again from scratch and now it is working fine.

quick note - I had provided all required permission as mentioned in point 3. but the missing part was RBAC authorization. Somehow RBAC setup was not correctly configured during my failed attempt.

closing this issue now.