search

Azure / kubernetes-keyvault-flexvol

Azure keyvault integration with Kubernetes via a Flex Volume
MIT License
253 stars 84 forks source link

MountVolume.SetUp failed for volume "test" : invalid character 'F' after object key:value pair #74

Closed shayshahak-zz closed 5 years ago

shayshahak-zz commented 5 years ago

I’m deploying FlexVol on AKS according to the procedure: https://github.com/Azure/kubernetes-keyvault-flexvol

I’m trying option 1 as mentioned “OPTION 1 - Service Principal” and using the sample POD configuration file provided in the same doc.

however the POD doesn’t seem to recognize the flexVolume configuration section and I get the following error: “MountVolume.SetUp failed for volume "test" : invalid character 'F' after object key:value pair”

My AKS version is 1.12 and I have installed the plugin as mentioned in the doc. I can see the installation POD running on my agents and nothing special in its logs:

NAME READY STATUS RESTARTS AGE keyvault-flexvolume-n44sl 1/1 Running 0 10m

can anyone please advise? Thanks,

ritazh commented 5 years ago

@shayshahak can you please share your pod yaml and kubectl describe pod <pod name>? please redact sensitive information.

shayshahak-zz commented 5 years ago

@ritazh yes of course. podyaml.txt describepod.txt

ritazh commented 5 years ago

@shayshahak Thanks for providing these. Few things to check:

  1. I see your pod was created in namespace kv. Please make sure your secret kvcreds was also created in namespaces kv.
  2. ssh into the agent node running the pending pod, get the hostname with kubectl get pod -o wide, or from your describe pod output it's aks-agentpool-94615314-1. then get more detailed logs from the node at: /var/log/kv-driver.log
shayshahak-zz commented 5 years ago

@ritazh

  1. the secret is in fact resides with the same namespace. when it's not i see the error "MountVolume.SetUp failed for volume "test" : Couldn't get secret kv/kvcreds err: secrets "kvcreds" not found".
  2. please see the requested log, there are some failures in it. kv-driverlog_new.txt

(uploaded again this time without my secrets in it.. :-) )

thanks

shayshahak-zz commented 5 years ago

got it! from the log i saw "AccessDenied" so i went back to my key vault only to find out that the ServicePrincipalName was suddenly missing on my Key Vault. fixed that, redeployed and now the sample pod is running:

NAME READY STATUS RESTARTS AGE keyvault-flexvolume-vzjqf 1/1 Running 0 4m nginx-flex-kv 1/1 Running 0 2m

thanks @ritazh for your support :-)