invalid character '/' looking for beginning of value

[sam-cogan]

When I try and deploy a pod with a secret mounted I am getting this error:

invalid character '/' looking for beginning of value

It seems to be related to the "driver" section of the flex volume, but I can;t see anything obvious I am doing wrong.

Type     Reason                 Age               From                               Message
  ----     ------                 ----              ----                               -------
  Normal   Scheduled              14s               default-scheduler                  Successfully assigned keyvault-flex-demo to aks-nodepool1-30108646-0
  Normal   SuccessfulMountVolume  14s               kubelet, aks-nodepool1-30108646-0  MountVolume.SetUp succeeded for volume "default-token-v8hjk"
  Warning  FailedMount            6s (x5 over 14s)  kubelet, aks-nodepool1-30108646-0  MountVolume.SetUp failed for volume "secret1" : invalid character '/' looking for beginning of value

Pod Yaml File:

apiVersion: v1
kind: Pod
metadata:
  name: keyvault-flex-demo
spec:
  containers:
  - name: keyvault-flex-demo
    image: nginx
    volumeMounts:
    - name: secret1
      mountPath: /kvmnt
      readOnly: true
  volumes:
  - name: secret1
    flexVolume:
      driver: "azure/kv"
      secretRef:
        name: kvcreds # mounting point to the pod
      options:
        usepodidentity: "false"
        keyvaultname: "kvFlexVolume"
        keyvaultobjectname: "appsecret1"
        keyvaultobjecttype: secret # OPTIONS: secret, key, cert
        keyvaultobjectversion: "b5b8ecffa0344756be898a7170148e7d"
        resourcegroup: "keyvaultFlexVolume"
        subscriptionid: "xxx"
        tenantid: "xxx"

[sam-cogan]

Looking at the logs on the node, I see the following

Sat Aug  4 18:20:03 UTC 2018 ismounted | /var/lib/kubelet/pods/9e6e8761-9807-11e8-a995-0a58ac1f1fc4/volumes/azure~kv/secret1 does not exist
Sat Aug 4 18:20:03 UTC 2018 ERROR: {"status": "Failure", "message": "validation failed, resourcegroup is empty"}

However, that resource group is the right one and contains the requested key vault.

[ritazh]

Thanks for reporting this @sam-cogan. Just to confirm, were you able to create a secret kvcreds in the same namespace?

kubectl create secret generic kvcreds --from-literal clientid=<CLIENTID> --from-literal clientsecret=<CLIENTSECRET> --type=azure/kv

[sam-cogan]

Yep, secret created and available. Having looked at the code, I think the issue is that us failing a validation on the resourcegroup variable, showing it is empty. So I don't think the values in the "options" section are actually getting passed through properly, I don't think it is getting as far as actually trying to talk to KV.

[ritazh]

Yes looking at the error message, it definitely thinks resourcegroup under options in the yaml is missing. But from the pod yaml you pasted above, everything looks ok. Using your pod yaml, I was not able to reproduce this issue with my own keyvault values. Can you please delete the deployment and redeploy?

[sam-cogan]

I've deleted and re-deployed quite a few times now and see the same result. I've also tried with a fresh yaml file copied from the docs. Just to check, the sub and tenant ID, should these be the actual ID GUIDs, or the names?

[ritazh]

the sub and tenant ID, should these be the actual ID GUIDs, or the names

They should be the actual IDs.

[ritazh]

I copied your pod yaml and created a resourcegroup and a new keyvault similar to yours but still cannot reproduce the issue.

[sam-cogan]

OK, well I fixed it but I'm not entirely sure what resolved it. When looking again this morning I noticed there was a spelling mistake in the resource group parameter name, and when I fixed that it worked. However, the spelling mistake wasn't present in the version I pasted above, so I guess I must have fixed something else along the way.

[ritazh]

spelling mistake in the resource group parameter name

That makes sense.

Glad this has been resolved. Closing this issue. Feel free to followup if you encounter other issues.

[HighwayofLife]

We encountered this error/problem as well, and solved it by the following change in the kv-flexvol-installer:

 ismounted() {
-    count=`sudo ls ${MNTPATH} | wc -l`
+    count=`grep "${MNTPATH}" /proc/mounts | wc -l`

Pull request incoming.