search

Azure / secrets-store-csi-driver-provider-azure

Azure Key Vault provider for Secret Store CSI driver allows you to get secret contents stored in Azure Key Vault instance and use the Secret Store CSI driver interface to mount them into Kubernetes pods.
https://azure.github.io/secrets-store-csi-driver-provider-azure/
MIT License
436 stars 194 forks source link

The secrets are not getting loaded while using pod identity with CSI secret driver #682

Closed sasarava closed 2 years ago

sasarava commented 3 years ago
The AzureIdentity
    apiVersion: v1
    items:
    - apiVersion: aadpodidentity.k8s.io/v1
      kind: AzureIdentity
      metadata:
        annotations:
          meta.helm.sh/release-name: airflow2
          meta.helm.sh/release-namespace: airflow2
        creationTimestamp: "2021-10-18T07:58:03Z"
        generation: 1
        labels:
          app.kubernetes.io/managed-by: Helm
        name: osdu-identity
        namespace: airflow2
        resourceVersion: "45110335"
        selfLink: /apis/aadpodidentity.k8s.io/v1/namespaces/airflow2/azureidentities/osdu-identity
        uid: 11f844e5-f358-4ad8-ad24-a86de32a3041
      spec:
        clientID: d08868f0-8f53-4f8b-bd2a-fab9d7f5befe
        resourceID: /subscriptions/538f399d-23a8-4880-97d8-0d46020b2bb7/resourcegroups/osdu-mvp-cringestiontestenv-qn63-rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/osdu-mvp-cringestiontestenv-qn63-osdu-identity
        type: 0
    kind: List
    metadata:
      resourceVersion: ""
      selfLink: ""
The Azure Identity Binding
apiVersion: v1
items:
- apiVersion: aadpodidentity.k8s.io/v1
  kind: AzureIdentityBinding
  metadata:
    annotations:
      meta.helm.sh/release-name: airflow2
      meta.helm.sh/release-namespace: airflow2
    creationTimestamp: "2021-10-18T07:58:03Z"
    generation: 1
    labels:
      app.kubernetes.io/managed-by: Helm
    name: osdu-identity-binding
    namespace: airflow2
    resourceVersion: "45110337"
    selfLink: /apis/aadpodidentity.k8s.io/v1/namespaces/airflow2/azureidentitybindings/osdu-identity-binding
    uid: 5f54b545-77c0-4a3e-a3e4-cbde2c8d90b5
  spec:
    azureIdentity: osdu-identity
    selector: osdu-identity
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
The pod definition
apiVersion: v1
kind: Pod
metadata:
  name: busybox
  labels:
    aadpodidbinding: osdu-identity
spec:
  containers:
  - image: busybox
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
    name: busybox
    env:
      - name: AIRFLOW_DATABASE_PASSWORD
        valueFrom:
          secretKeyRef:
            name: postgres
            key: postgres-password
  restartPolicy: Always
The secret provider class
---
kind: SecretProviderClass
spec:
  secretObjects:
  - type: Opaque
    data:
    - objectName: airflow-storage
      key: storage-account
    - objectName: airflow-storage-key
      key: storage-key
    - objectName: airflow-storage-connection
      key: storage-connection
    - objectName: airflow-remote-log-connection
      key: remote-log-connection
    - objectName: airflow-admin-password
      key: admin-password
    - objectName: airflow-fernet-key
      key: fernet-key
    - objectName: airflow-storage
      key: azurestorageaccountname
    - objectName: airflow-storage-key
      key: azurestorageaccountkey
    secretName: airflow
  - type: Opaque
    data:
    - objectName: postgres-password
      key: postgres-password
    secretName: postgres
  - type: Opaque
    data:
    - objectName: redis-queue-password
      key: redis-queue-password
    secretName: redis
  - type: Opaque
    data:
    - objectName: log-workspace-id
      key: workspace-id
    - objectName: log-workspace-key
      key: workspace-key
    secretName: dp-logging
  parameters:
    userAssignedIdentityID: ''
    useVMManagedIdentity: 'false'
    tenantId: 58975fd3-4977-44d0-bea8-37af0baac100
    resourceGroup: osdu-mvp-cringestiontestenv-qn63-rg
    objects: "array:\n  - |\n    objectName: airflow-storage\n    objectType: secret\n\
      \  - |\n    objectName: airflow-storage-connection\n    objectType: secret\n\
      \  - |\n    objectName: airflow-remote-log-connection\n    objectType: secret\n\
      \  - |\n    objectName: airflow-storage-key\n    objectType: secret\n  - |\n\
      \    objectName: airflow-admin-password\n    objectType: secret\n  - |\n   \
      \ objectName: airflow-fernet-key\n    objectType: secret\n  - |\n    objectName:\
      \ postgres-password\n    objectType: secret\n  - |\n    objectName: redis-queue-password\n\
      \    objectType: secret\n  - |\n    objectName: log-workspace-id\n    objectType:\
      \ secret\n  - |\n    objectName: log-workspace-key\n    objectType: secret\n"
    usePodIdentity: 'true'
    keyvaultName: osdu-mvp-cringes-qn63-kv
    subscriptionId: 538f399d-23a8-4880-97d8-0d46020b2bb7
  provider: azure
apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
metadata:
  namespace: airflow2
  name: azure-keyvault
Azure Assigned Identity that got created
apiVersion: aadpodidentity.k8s.io/v1
kind: AzureAssignedIdentity
metadata:
  creationTimestamp: "2021-10-18T09:23:01Z"
  finalizers:
  - azureassignedidentity.finalizers.aadpodidentity.k8s.io
  generation: 8
  labels:
    nodename: aks-default-19538952-vmss000001
  name: busybox-airflow2-osdu-identity
  namespace: default
  resourceVersion: "45146395"
  selfLink: /apis/aadpodidentity.k8s.io/v1/namespaces/default/azureassignedidentities/busybox-airflow2-osdu-identity
  uid: b631d352-34bd-40ea-83b1-109c36cf7aa4
spec:
  azureBindingRef:
    apiVersion: aadpodidentity.k8s.io/v1
    kind: AzureIdentityBinding
    metadata:
      annotations:
        fluxcd.io/sync-checksum: dd6af7b603359832560796d9b2fe43985634b769
        kubectl.kubernetes.io/last-applied-configuration: |
          {"apiVersion":"aadpodidentity.k8s.io/v1","kind":"AzureIdentityBinding","metadata":{"annotations":{"fluxcd.io/sync-checksum":"dd6af7b603359832560796d9b2fe43985634b769"},"labels":{"fluxcd.io/sync-gc-mark":"sha256.pH205iAv4ulVNk3xKXhCvpx7lUCVh4px3LGxs1Y078s"},"name":"osdu-identity-binding","namespace":"osdu"},"spec":{"azureIdentity":"osdu-identity","selector":"osdu-identity"}}
      creationTimestamp: "2021-07-21T06:46:17Z"
      generation: 1
      labels:
        fluxcd.io/sync-gc-mark: sha256.pH205iAv4ulVNk3xKXhCvpx7lUCVh4px3LGxs1Y078s
      managedFields:
      - apiVersion: aadpodidentity.k8s.io/v1
        fieldsType: FieldsV1
        fieldsV1:
          f:metadata:
            f:annotations:
              .: {}
              f:fluxcd.io/sync-checksum: {}
              f:kubectl.kubernetes.io/last-applied-configuration: {}
            f:labels:
              .: {}
              f:fluxcd.io/sync-gc-mark: {}
          f:spec:
            .: {}
            f:azureIdentity: {}
            f:selector: {}
        manager: kubectl
        operation: Update
        time: "2021-07-21T06:46:17Z"
      name: osdu-identity-binding
      namespace: osdu
      resourceVersion: "3052"
      selfLink: /apis/aadpodidentity.k8s.io/v1/namespaces/osdu/azureidentitybindings/osdu-identity-binding
      uid: 0790b9a2-e88c-4935-8519-47a5060f62ad
    spec:
      azureIdentity: osdu-identity
      metadata:
        creationTimestamp: null
      selector: osdu-identity
      weight: 0
    status:
      availableReplicas: 0
      metadata:
        creationTimestamp: null
  azureIdentityRef:
    apiVersion: aadpodidentity.k8s.io/v1
    kind: AzureIdentity
    metadata:
      annotations:
        fluxcd.io/sync-checksum: abac69d92a77b927ec2644333cd9a5e4a1b7dbb7
        kubectl.kubernetes.io/last-applied-configuration: |
          {"apiVersion":"aadpodidentity.k8s.io/v1","kind":"AzureIdentity","metadata":{"annotations":{"fluxcd.io/sync-checksum":"abac69d92a77b927ec2644333cd9a5e4a1b7dbb7"},"labels":{"fluxcd.io/sync-gc-mark":"sha256.ujPeWFHWBfDChw6SHlBYZZmUATtRm0kMuWjaqp6I8hQ"},"name":"osdu-identity","namespace":"osdu"},"spec":{"clientID":"d08868f0-8f53-4f8b-bd2a-fab9d7f5befe","resourceID":"/subscriptions/538f399d-23a8-4880-97d8-0d46020b2bb7/resourcegroups/osdu-mvp-cringestiontestenv-qn63-rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/osdu-mvp-cringestiontestenv-qn63-osdu-identity","type":0}}
      creationTimestamp: "2021-07-21T06:46:17Z"
      generation: 1
      labels:
        fluxcd.io/sync-gc-mark: sha256.ujPeWFHWBfDChw6SHlBYZZmUATtRm0kMuWjaqp6I8hQ
      managedFields:
      - apiVersion: aadpodidentity.k8s.io/v1
        fieldsType: FieldsV1
        fieldsV1:
          f:metadata:
            f:annotations:
              .: {}
              f:fluxcd.io/sync-checksum: {}
              f:kubectl.kubernetes.io/last-applied-configuration: {}
            f:labels:
              .: {}
              f:fluxcd.io/sync-gc-mark: {}
          f:spec:
            .: {}
            f:clientID: {}
            f:resourceID: {}
            f:type: {}
        manager: kubectl
        operation: Update
        time: "2021-07-21T06:46:17Z"
      name: osdu-identity
      namespace: osdu
      resourceVersion: "3051"
      selfLink: /apis/aadpodidentity.k8s.io/v1/namespaces/osdu/azureidentities/osdu-identity
      uid: 061c2d1e-c98a-47e4-be67-194b74e6d7c2
    spec:
      adEndpoint: ""
      adResourceID: ""
      auxiliaryTenantIDs: null
      clientID: d08868f0-8f53-4f8b-bd2a-fab9d7f5befe
      clientPassword: {}
      metadata:
        creationTimestamp: null
      replicas: null
      resourceID: /subscriptions/538f399d-23a8-4880-97d8-0d46020b2bb7/resourcegroups/osdu-mvp-cringestiontestenv-qn63-rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/osdu-mvp-cringestiontestenv-qn63-osdu-identity
      tenantID: ""
      type: 0
    status:
      availableReplicas: 0
      metadata:
        creationTimestamp: null
  metadata:
    creationTimestamp: null
  nodename: aks-default-19538952-vmss000001
  pod: busybox
  podNamespace: airflow2
  replicas: null
status:
  availableReplicas: 1
  metadata:
    creationTimestamp: null
  status: Assigned

Given the above state and given the secrets are present in the Key Vaults, the kubernetes secrets are not getting generated as expected. We have given the necessary permissions to the Key Vault. Please provide some pointers on proceeding further.

aramase commented 3 years ago

@sasarava Could you follow the troubleshooting guide to get logs from the driver and provider pod running on the same node as the workload pod? Also, there should be an error in kubectl describe pod busybox right after you deploy.

github-actions[bot] commented 2 years ago

This issue is stale because it has been open 14 days with no activity. Please comment or this will be closed in 7 days.

github-actions[bot] commented 2 years ago

This issue was closed because it has been stalled for 21 days with no activity. Feel free to re-open if you are experiencing the issue again.