Unable to mount volumes for pod "nginx-flex-kv-podid_default"

[uday31in]

Events:
  Type     Reason                 Age                From                               Message
  ----     ------                 ----               ----                               -------
  Normal   Scheduled              10m                default-scheduler                  Successfully assigned nginx-flex-kv-podid to aks-agentpool-23065106-0
  Normal   SuccessfulMountVolume  10m                kubelet, aks-agentpool-23065106-0  MountVolume.SetUp succeeded for volume "default-token-cqd6w"
  Warning  FailedMount            2m (x10 over 10m)  kubelet, aks-agentpool-23065106-0  MountVolume.SetUp failed for volume "test" : invalid character 's' looking for beginning of value
  Warning  FailedMount            1m (x4 over 8m)    kubelet, aks-agentpool-23065106-0  Unable to mount volumes for pod "nginx-flex-kv-podid_default(be10160c-93bb-11e8-b1ce-aa6af2addf8c)": timeout expired waiting for volumes to attach or mount for pod "default"/"nginx-flex-kv-podid". list of unmounted volumes=[test]. list of unattached volumes=[test default-token-cqd6w]

uday@DESKTOP-RGMO1N7:/mnt/c/git/kubernetes-keyvault-flexvol/deployment$ kubectl get azureassignedidentities
NAME                                         CREATED AT
nginx-flex-kv-podid-default-pod-identity     6m

apiVersion: v1
kind: Pod
metadata:
  labels:
    app: nginx-flex-kv-podid
    aadpodidbinding: "demo"
  name: nginx-flex-kv-podid
spec:
  containers:
  - name: nginx-flex-kv-podid
    image: nginx
    volumeMounts:
    - name: test
      mountPath: /kvmnt
      readOnly: true
  volumes:
  - name: test
    flexVolume:
      driver: "azure/kv"
      options:
        usePodIdentity: "true"
        keyvaultname: "aks-vdc-kv"
        keyvaultobjectname: "tenantID"
        keyvaultobjecttype: "secret"
        keyvaultobjectversion: "33b73696bb9d4b839e0c1a5412089e31"
        resourcegroup: "aks-kv"

[ritazh]

@uday31in can you share logs from /var/log/kv-driver.log on aks-agentpool-23065106-0?

[uday31in]

uday@aks-agentpool-23065106-0:~$ cat /var/log/kv-driver.log
Sun Jul 29 18:09:01 UTC 2018 INFO: {"status": "Success", "capabilities": {"attach": false}}
Mon Jul 30 05:41:42 UTC 2018 ismounted | /var/lib/kubelet/pods/2dc1e1b3-93bb-11e8-b1ce-aa6af2addf8c/volumes/azure~kv/test does not exist
Mon Jul 30 05:41:42 UTC 2018 ERROR: {"status": "Failure", "message": "validation failed, keyvaultobjectversion is empty"}
Mon Jul 30 05:42:08 UTC 2018 ismounted | /var/lib/kubelet/pods/2dc1e1b3-93bb-11e8-b1ce-aa6af2addf8c/volumes/azure~kv/test does not exist
Mon Jul 30 05:42:08 UTC 2018 ERROR: {"status": "Failure", "message": "validation failed, keyvaultobjectversion is empty"}
Mon Jul 30 05:42:28 UTC 2018 ismounted | /var/lib/kubelet/pods/2dc1e1b3-93bb-11e8-b1ce-aa6af2addf8c/volumes/azure~kv/test does not exist
Mon Jul 30 05:42:28 UTC 2018 ERROR: {"status": "Failure", "message": "validation failed, keyvaultobjectversion is empty"}
Mon Jul 30 05:42:55 UTC 2018 ismounted | /var/lib/kubelet/pods/2dc1e1b3-93bb-11e8-b1ce-aa6af2addf8c/volumes/azure~kv/test does not exist
Mon Jul 30 05:42:55 UTC 2018 ERROR: {"status": "Failure", "message": "validation failed, keyvaultobjectversion is empty"}
Mon Jul 30 05:43:25 UTC 2018 ismounted | /var/lib/kubelet/pods/2dc1e1b3-93bb-11e8-b1ce-aa6af2addf8c/volumes/azure~kv/test does not exist
Mon Jul 30 05:43:25 UTC 2018 ERROR: {"status": "Failure", "message": "validation failed, keyvaultobjectversion is empty"}
Mon Jul 30 05:43:58 UTC 2018 ismounted | /var/lib/kubelet/pods/2dc1e1b3-93bb-11e8-b1ce-aa6af2addf8c/volumes/azure~kv/test does not exist
Mon Jul 30 05:43:58 UTC 2018 ERROR: {"status": "Failure", "message": "validation failed, keyvaultobjectversion is empty"}
Mon Jul 30 05:44:39 UTC 2018 ismounted | /var/lib/kubelet/pods/2dc1e1b3-93bb-11e8-b1ce-aa6af2addf8c/volumes/azure~kv/test does not exist

[ritazh]

Seems validation for the flexvolume properties have failed.

Make sure all the properties are all lowercase.usepodidentity and redeploy the pod.

[ritazh]

@uday31in Can you pls confirm if this issue has been resolved and if we can close it?

[uday31in]

@ritazh - yes it worked. thank you for your assistance.