Concurrent mount may result in success with no secret(s) in the volume

[ankrause]

Describe the bug I have a chart that will create 2 deployments at the same time. Both of these deployments mount a secret from Key Vault using AAD Pod Identity (MSI) authentication. The secrets are mounted into the same relative location in each pod.

Using v0.0.10 of the FlexVolume mounts both secrets without issue. However, updating to 0.0.11 of the FlexVolume may result in only one secret being mounted in one pod. The other pod will not receive the secret and will subsequently fail at runtime as the secret is required.

In this scenario, both pods were initially scheduled on the same Linux node. Deleting the pod which did not receive the secret re-scheduled it on a second node which then received the secret correctly and started as expected.

The below log appears to be missing the "write" log message for the first pod, which is the pod that failed. There appear to be 2 messages for every other step as expected (including 2 successes), but only 1 write message is shown and it is for the pod GUID that did successfully start.

Mon Aug 12 04:23:20 UTC 2019 INFO: {"status": "Success", "capabilities": {"attach": false}}
Mon Aug 12 04:27:47 UTC 2019 ismounted | not mounted
Mon Aug 12 04:27:47 UTC 2019 ismounted | not mounted
Mon Aug 12 04:27:47 UTC 2019 PODNAME: <POD-1>
Mon Aug 12 04:27:47 UTC 2019 PODNAME: <POD-2>
Mon Aug 12 04:27:47 UTC 2019 mount
Mon Aug 12 04:27:47 UTC 2019 mount
Mon Aug 12 04:27:47 UTC 2019 /etc/kubernetes/volumeplugins/azure~kv/azurekeyvault-flexvolume -logtostderr=1 -vaultName=<VAULT-NAME> -vaultObjectNames=<SECRET-NAME> -vaultObjectAliases= -resourceGroup=<RESOURCE-GROUP> -dir=/var/lib/kubelet/pods/<POD-1-GUID>/volumes/azure~kv/kv-flexvol-0 -subscriptionId=<SUBSCRIPTION-ID> -cloudName= -tenantId=<TENANT-ID> -aADClientSecret= -aADClientID= -usePodIdentity=true -podNamespace=<POD-NAMESPACE> -podName=<POD-1> -vaultObjectVersions= -vaultObjectTypes=secret
Mon Aug 12 04:27:47 UTC 2019 /etc/kubernetes/volumeplugins/azure~kv/azurekeyvault-flexvolume -logtostderr=1 -vaultName=<VAULT-NAME> -vaultObjectNames=<SECRET-NAME> -vaultObjectAliases= -resourceGroup=<RESOURCE-GROUP> -dir=/var/lib/kubelet/pods/<POD-2-GUID>/volumes/azure~kv/kv-flexvol-0 -subscriptionId=<SUBSCRIPTION-ID> -cloudName= -tenantId=<TENANT-ID> -aADClientSecret= -aADClientID= -usePodIdentity=true -podNamespace=<POD-NAMESPACE> -podName=<POD-2> -vaultObjectVersions= -vaultObjectTypes=secret
I0812 04:27:47.388245   20202 keyvaultFlexvolumeAdapter.go:32] azurekeyvault-flexvolume 0.0.11
I0812 04:27:47.388315   20202 keyvaultFlexvolumeAdapter.go:41] starting the azurekeyvault-flexvolume, 0.0.11
I0812 04:27:47.388355   20202 oauth.go:138] azure: using pod identity to retrieve token
I0812 04:27:47.389811   20203 keyvaultFlexvolumeAdapter.go:32] azurekeyvault-flexvolume 0.0.11
I0812 04:27:47.389879   20203 keyvaultFlexvolumeAdapter.go:41] starting the azurekeyvault-flexvolume, 0.0.11
I0812 04:27:47.389928   20203 oauth.go:138] azure: using pod identity to retrieve token

 accesstoken: <EXTRA REDACTED>

 clientid: <EXTRA REDACTED>
I0812 04:28:02.666908   20203 oauth.go:138] azure: using pod identity to retrieve token

 accesstoken: <EXTRA REDACTED>

 clientid: <EXTRA REDACTED>
I0812 04:28:02.853566   20203 keyvaultFlexvolumeAdapter.go:71] retrieving secret <SECRET-NAME> (version: )
I0812 04:28:02.963212   20203 keyvaultFlexvolumeAdapter.go:102] azure KeyVault wrote secret <SECRET-NAME> at /var/lib/kubelet/pods/<POD-2-GUID>/volumes/azure~kv/kv-flexvol-0/<SECRET-NAME>
Mon Aug 12 04:28:02 UTC 2019 INFO: {"status": "Success"}
Mon Aug 12 04:28:22 UTC 2019 INFO: {"status": "Success"}

Steps To Reproduce

1. Deploy a cluster with AAD Pod Identity and Key Vault FlexVolume 0.0.11
2. Create/Apply 2 deployments which both mount the same secret from the same vault to the same relative location in each pod

Sample volume mount:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: {Name}
  name: {Name}
spec:
  replicas: 1
  selector:
    matchLabels:
      app: {Name}
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 10%
    type: RollingUpdate
  template:
    metadata:
      labels:
        aadpodidbinding: MSI
        app: {Name}
    spec:
      containers:
        image: {Image}
        imagePullPolicy: IfNotPresent
        name: {Name}
        volumeMounts:
        - mountPath: /run/certs
          name: kv-flexvol-0
          readOnly: true
      nodeSelector:
        beta.kubernetes.io/os: linux
      volumes:
      - flexVolume:
          driver: azure/kv
          options:
            keyvaultname: {VaultName}
            keyvaultobjectnames: {SecretName}
            keyvaultobjecttypes: secret
            resourcegroup: {ResourceGroupName}
            subscriptionid: {SubscriptionId}
            tenantid: {TenantId}
            usepodidentity: "true"
        name: kv-flexvol-0

Expected behavior Both pods have their secrets mounted and available by the time the pod's container starts.

Key Vault FlexVolume version 0.0.11

Access mode: service principal or pod identity Pod identity

Kubernetes version 1.14.5

[aramase]

@ankrause Thank you for reporting the issue. I'll take a look.

[aramase]

@ankrause I'm unable to reproduce this. I've 2 pods landing on the same node, requesting the same secret from same keyvault. The mount and the subsequent write seem to work as expected. Are you able to reproduce this consistently or is this behavior intermittent?

I've also tried 3 pods all on same node and it works as expected.

[ankrause]

Thanks for the quick fix!