Azure / kubernetes-keyvault-flexvol

Azure keyvault integration with Kubernetes via a Flex Volume
MIT License
253 stars 84 forks source link

Failed to Mount: 'Invalid character 'I' looking for beginning of value' #83

Closed shawnj closed 5 years ago

shawnj commented 5 years ago

We are getting the following error when trying to use the Azure KeyVault Flexvol:

kubectl output using at test job:

  Normal   Scheduled               15s   default-scheduler           Successfully assigned newspace/test-azflexjob-njmcs to test-worker5
  Warning  FailedMount             12s   kubelet, test-worker5  MountVolume.SetUp failed for volume "az-secret" : invalid character 'I' looking for beginning of value
  Normal   Pulling                 11s   kubelet, test-worker5  pulling image "busybox"
  Normal   Pulled                  9s    kubelet, test-worker5  Successfully pulled image "busybox"
  Normal   Created                 9s    kubelet, test-worker5  Created container
  Normal   Started                 8s    kubelet, test-worker5  Started container

Logging output:

E0222 20:31:04.965438    3260 driver-call.go:251] Failed to unmarshal output for command: mount, output: "I0222 20:31:02.904722   14032 keyvaultFlexvolumeAdapter.go:32] azurekeyvault-flexvolume 0.0.8
I0222 20:31:02.904814   14032 keyvaultFlexvolumeAdapter.go:41] starting the azurekeyvault-flexvolume, 0.0.8
I0222 20:31:04.501628   14032 keyvaultFlexvolumeAdapter.go:65] retrieving secret testsecret (version: 7b448f7d55234a76b0068ffwf8hf322)
I0222 20:31:04.962156   14032 keyvaultFlexvolumeAdapter.go:124] azure KeyVault wrote secret testsecret at /opt/rke/var/lib/kubelet/pods/c57e768e-36e0-11e9-b4d6-000d3a36e51b/volumes/azure~kv/az-secret
{\"status\": \"Success\"}
", error: invalid character 'I' looking for beginning of value

The volume is successfully mounted after a 500ms retry attempt.

It appears that the output from the stderr includes more than the status and that data can't be consumed. I have tested removing the -logtostderr=1 from line 158 in the kv file and that fixes the issue.

Env: Kubernetes: v1.11.6 OS: CoreOS-1967.6.0 RKE: v0.1.16