kubernetes-sigs / secrets-store-csi-driver

Secrets Store CSI driver for Kubernetes secrets - Integrates secrets stores with Kubernetes via a CSI volume.
https://secrets-store-csi-driver.sigs.k8s.io/
Apache License 2.0
1.23k stars 287 forks source link

Unable to set environment variable from Azure Key Vault: Service Account cannot list secrets #670

Closed gperezivo closed 3 years ago

gperezivo commented 3 years ago

What steps did you take and what happened: I created a SecretProviderClass with secretName referencing an Azure Key Vault secret using pod identity:

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
    name: gperez-kv-secret-provider
spec:
  provider: azure
  secretObjects:                                 
  - secretName: kv-secret
    type: Opaque
    data:
      - objectName: testkvalias 
        key:  kvparam 
  parameters:
    usePodIdentity: "true"
    keyvaultName: gperez-kv
    objects: |
      array:
        - |
          objectName: testkv
          objectAlias: testkvalias
          objectType: secret
          objectVersion: ""
        - |
          objectName: MyKeyVaultSecretParameter
          objectType: secret
          objectVersion: ""
    resourceGroup: "XXXXXXXXXXXXXX"
    subscriptionId: "XXXXXXXXXXXXXX"
    tenantId: "XXXXXXXXXXXX"

When I try to refer the secret in a pod env variable, I get an error: secret "kv-secret" not found

apiVersion: v1
kind: Pod
metadata:
  name: gperez-demo-kv-test-pod
  labels:
    name: gperez-demo-kv-test-pod
    aadpodidbinding: gperez-demo-kv-identity-binding-selector
spec:
  containers:
  - name: gperez-demo-kv-test-pod
    image: enriquecatala/fastapi-helloworld
    resources:
      limits:
        memory: "128Mi"
        cpu: "500m"
    env:
    - name: HELLOWORLD_ENV
      valueFrom:
        secretKeyRef:
          name: kv-secret
          key: kvparam

    volumeMounts:
      - name: secrets-store-inline
        mountPath: "/mnt/secrets-store"
        readOnly: true
    ports:
      - containerPort: 5000
  volumes:
  - name: secrets-store-inline
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "gperez-kv-secret-provider"   
Events:
  Type     Reason     Age               From               Message
  ----     ------     ----              ----               -------
  Normal   Scheduled  37s               default-scheduler  Successfully assigned kv-demo/gperez-demo-kv-test-pod to aks-nodepool1-27660810-vmss000000
  Normal   Pulled     19s               kubelet            Successfully pulled image "enriquecatala/fastapi-helloworld" in 759.496933ms
  Normal   Pulled     18s               kubelet            Successfully pulled image "enriquecatala/fastapi-helloworld" in 764.890794ms
  Normal   Pulling    4s (x3 over 20s)  kubelet            Pulling image "enriquecatala/fastapi-helloworld"
  Warning  Failed     3s (x3 over 19s)  kubelet            Error: secret "kv-secret" not found
  Normal   Pulled     3s                kubelet            Successfully pulled image "enriquecatala/fastapi-helloworld" in 895.568611ms

The pod identity it's working and when I remove the environment variable the store inline mounts correctly and fill the values from Key Vault.

 kubectl exec gperez-demo-kv-test-pod -- ls /mnt/secrets-store
MyKeyVaultSecretParameter
testkvalias

I found an error on csi-driver pod:

E0729 09:32:21.321640       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.21.1/tools/cache/reflector.go:167: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:kv-demo:secrets-store-csi-driver" cannot list resource "secrets" in API group "" at the cluster scope

What did you expect to happen: Read an environment variable from Azure Key Vault (?)

Anything else you would like to add: I'm using Azure Kubernetes Services with Kubernetes 1.20.7 I tried to install with parameters --set syncSecret.enabled=true --set rbac.install=true but fails too on pod deploy.

Which provider are you using: Azure Key Vault

Environment:

gperezivo commented 3 years ago

Nevermind, I dont type correctly the secrets-store-csi-driver.syncSecret.enabled=true option... I resolve reading... https://azure.github.io/secrets-store-csi-driver-provider-azure/troubleshooting/#failed-to-create-kubernetes-secret-errsecrets-is-forbidden-user-systemserviceaccountdefaultsecrets-store-csi-driver-cannot-create-resource-secrets-in-api-group--in-the-namespace-default