Pod Identity Association not recognised by secrets store CSI driver

[ecole-startupcraft]

Describe the bug Attempting to mount secrets using Pod Identity EKS add-on results in the following error:

Warning FailedMount 103s (x29 over 44m) kubelet MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod default/secrets-store-inline, err: rpc error: code = Unknown desc = us-east-1: An IAM role must be associated with service account demo-app-pod-sa (namespace: default) To Reproduce

Steps to reproduce the behavior:

• Create EKS Cluster
• Install Pod Identity add-on
• Create service account
• Associate service account with IAM role using EKS Pod Identity Association
• Deploy pod:

      - kind: Pod
        apiVersion: v1
        metadata:
          name: secrets-store-inline
        spec:
          serviceAccountName: demo-pod-sa
          containers:
          - image: registry.k8s.io/e2e-test-images/busybox:1.29-4
            name: busybox
            imagePullPolicy: IfNotPresent
            command:
            - "/bin/sleep"
            - "10000"
            volumeMounts:
            - name: secrets-store-inline
              mountPath: "/mnt/secrets-store"
              readOnly: true
          volumes:
            - name: secrets-store-inline
              csi:
                driver: secrets-store.csi.k8s.io
                readOnly: true
                volumeAttributes:
                  secretProviderClass: "aws-secrets"

Do you also notice this bug when using a different secrets store provider (Vault/Azure/GCP...)? Yes/No AWS EKS specific

Expected behavior Secrets Volume properly mounted

Environment: Helm chart secrets-store-csi-driver-provider-aws v0.3.5

[gustavclausen]

This is must likely due to the provider using an older version of the AWS SDK (1.47.10) which doesn't support the container credential provider (supporting the EKS Pod Identity functionality). The minimum required SDK version is v1.47.11 (see https://docs.aws.amazon.com/eks/latest/userguide/pod-id-minimum-sdk.html). Thus, this is also a thing to consider.

[ran2806]

I am also facing the same issue. Is there any solution on it?

[egorksv]

I am also facing the same issue. Is there any solution on it?

As a workaround I'm using good old OIDC-mapped IAM role, but would definitely want to move on to the new way.

[steven-so]

I am also facing the same issue. Is there any solution on it?

As a workaround I'm using good old OIDC-mapped IAM role, but would definitely want to move on to the new way.

What role does the service account need to "MountVolume.SetUp failed for volume 'secrets-store-inline'"?

[steven-so]

I'm also getting: Warning FailedMount 6s (x6 over 22s) kubelet MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod default/user-service-depl-858474494c-5nr6w, err: rpc error: code = Unknown desc = us-west-2: An IAM role must be associated with service account default (namespace: default)

however I'm not using Pod Identity add-on.

I'm trying to spin up a private docker hub image.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-service
  template:
    metadata:
      labels:
        app: my-service
    spec:
      volumes:
        - name: secrets-store-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "aws-secrets"
      containers:
        - name: my-service
          image: myprivate/dockerimage
          env:
            - name: JWT_KEY
              valueFrom:
                secretKeyRef:
                  name: jwt-secret
                  key: JWT_KEY
          volumeMounts:
            - name: secrets-store-inline
              mountPath: "/mnt/secrets-store"
              readOnly: true
      imagePullSecrets:
        - name: "docker-hub"

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aws-secrets
spec:
  provider: aws
  secretObjects:
    - secretName: jwt-secret
      type: Opaque
      data:
        - objectName: jwt-secret
          key: JWT_KEY
    - secretName: docker-hub
      type: kubernetes.io/dockerconfigjson
      data:
        - objectName: "docker-configjson"
          key: ".dockerconfigjson"
  parameters:
    region: us-west-2
    objects: |
      - objectName: "jwt-secret"
        objectType: "secretsmanager"
      - objectName: "docker-configjson"
        objectType: "secretsmanager"

[jbct]

Thank you for the report, we will look into this issue.

[kevarr]

To offer a potential use-case: I'd like to use a cross-account IAM role to centralize permission management to a SecretsManager secret, use EKS Pod Identities to allow a Pod running in one account (Account A) to assume the centralized role in another account (Account B), and mount the secret into the Pod using secrets-store-csi-driver-provider-aws.

With EKS Pod Identities you can only associate roles that are in the same AWS account as the cluster. The documentation states that to achieve cross-account access you should use role chaining[1]. This is in contrast to IRSA which allows you to directly assume a role in another account [2].

If I were to use the provider to mount the secret into my Pod it would need to assume the role in Account B using the credentials of Account A before making the GetSecretValue API request.

[1] https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html [2] https://aws.amazon.com/blogs/containers/cross-account-iam-roles-for-kubernetes-service-accounts/

[dprangnell]

I haven't tested this, but it's almost definitely resolved in the latest release 0.3.6 which deploys v1.49.19 of the SDK.

[dprangnell]

Well I spoke too soon, I tested it and it doesn't work with release 0.3.6:

  Warning  FailedMount  40s (x11 over 6m52s)  kubelet MountVolume.SetUp failed for volume "secret-from-secret-manager" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod default/demo-app, err: rpc error: code = Unknown desc = us-east-1: An IAM role must be associated with service account pod-identity (namespace: default)

helm list -n kube-system      
NAME                            NAMESPACE   REVISION    UPDATED                                 STATUS      CHART                                       APP VERSION
csi-secrets-store               kube-system 1           2024-02-17 10:52:16.835931 -0800 PST    deployed    secrets-store-csi-driver-1.4.1              1.4.1      
secrets-provider-aws            kube-system 1           2024-02-17 10:52:34.524537 -0800 PST    deployed    secrets-store-csi-driver-provider-aws-0.3.6 

    spec:
      serviceAccountName: pod-identity
      volumes:
        - name: secret-from-secret-manager
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "aws-secrets"
      containers:
      - name: demo-app
        image: "example/demo-app:latest"
        volumeMounts:
        - name: secrets-from-secret-manager
          mountPath: "/mnt/secrets-store"
          readOnly: true
        imagePullPolicy: Always

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: aws-secrets
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "arn:aws:secretsmanager:{{.Values.region}}:{{.Values.accountid}}:secret:{{.Values.secretname}}"

[Fatma-J]

@dprangnell Same for me. Have you succeeded in fixing the issue?

[giedriuskilcauskas]

That specific check for annotation is here: https://github.com/aws/secrets-store-csi-driver-provider-aws/blob/main/auth/auth.go#L110-L126 And I think there is no way to workaround it with parameters without code change.

[DanielMcAssey]

Hitting this issue too, unfortunately OIDC is a pain with its limitations on roles

[jbct]

Thank you for the request. The EKS POD Identities page calls out incompatibility with other CSI storage drivers and we are working to get documentation updated to include the Secrets Manager and Config Provider for Secret Store. We have this in our backlog and have marked it as a future enhancement.

[DanielMcAssey]

I have secrets manager manually installed, that page indicates that it should work, but I am hitting the same issue as the above

[yambottle]

System Info

• k8s Version: v1.28.6, created by kOps with EC2 instances(not EKS)
• cilium: quay.io/cilium/operator:v1.13.10
• pod-identity-webhook: amazon/amazon-eks-pod-identity-webhook:v0.4.0
• csi-secrets-store: 1.4.2
• secrets-store-csi-driver-provider-aws: public.ecr.aws/aws-secrets-manager/secrets-store-csi-driver-provider-aws:1.0.r2-58-g4ddce6a-2024.01.31.21.42 from doc

Setup Steps

• kOps created cluster already existis before this test, it has been created with Service Account Issuer Discovery and AWS IAM Roles for Service Accounts (IRSA)
• Installed csi-secrets-store and secrets-store-csi-driver-provider-aws following the doc
• Use kOps to provision IAM role and policy for service account:

  spec:
  iam:
  serviceAccountExternalPermissions:
    - name: default-secret-manager
      namespace: default
      aws:
        inlinePolicy: |-
          [
            {
              "Effect": "Allow",
              "Action": ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"],
              "Resource": ["arn:*:secretsmanager:*:*:secret:*"]
            }
          ]

• Create SA default-secret-manager
• Create SecretProviderClass:

  apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
  kind: SecretProviderClass
  metadata:
  name: aws-secrets
  spec:
  provider: aws
  parameters:                    # provider-specific parameters
  region: xxxx
  objects:  |
    - objectName: "xxxxx"
      objectType: "secretsmanager"

• Create Test Pod:

  apiVersion: v1
  kind: Pod
  metadata:
  name: aws-cli
  spec:
  serviceAccount: default-secret-manager
  containers:
  - name: aws-cli
  image: amazon/aws-cli:latest
  command: ["/bin/bash", "-c", "tail -f /dev/null"]
  imagePullPolicy: IfNotPresent
  volumeMounts:
  - name: test-secret
    mountPath: "/mnt/secrets-store"
    readOnly: true
  volumes:
  - name: test-secret
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "aws-secrets"
  restartPolicy: Always

• Got pod stuck on ContainerCreating with error: MountVolume.SetUp failed for volume "test-secret" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod default/aws-cli, err: rpc error: code = Unknown desc = xxxx: An IAM role must be associated with service account default-secret-manager (namespace: default)
• Tried to attach SecretsManagerReadWrite policy, got the same error
• If I manually attach AWSS3ReadOnly policy to the kOps provisioned role, and comment out the volume mount

  apiVersion: v1
  kind: Pod
  metadata:
  name: aws-cli
  spec:
  serviceAccount: default-secret-manager
  containers:
  - name: aws-cli
  image: amazon/aws-cli:latest
  command: ["/bin/bash", "-c", "tail -f /dev/null"]
  imagePullPolicy: IfNotPresent
  restartPolicy: Always

  I'm able to access the pre-created secret with the same service account

  bash-4.2# aws secretsmanager get-secret-value --secret-id xxxxx
  {
  "ARN": "arn:aws:secretsmanager:xxx:xxx:secret:xxx",
  "Name": "xxxx",
  "VersionId": "xxxxx",
  "SecretString": "xxxxxx,
  "VersionStages": [
      "AWSCURRENT"
  ],
  "CreatedDate": "2024-03-19T21:00:25.681000+00:00"
  }

  Hope this info could help to resolve this issue sooner!

[giedriuskilcauskas]

• Create SA default-secret-manager

@yambottle you need to put annotation on SA with the role created in previous step.

[AnhQKatalon]

I am currently having the exact same issue. This prevents us from migrating to use Pod Identity because all of our services are using Secret CSI to pull the secrets from Secrets Manager

[swt-yoromero]

Hola a todos!!!. Me paso lo mismo, tengo EKS 1.30, y lo solucione agregandole permisos al rol ya que faltaban permisos para los recursos secrets, este era el error que me daba encontrado en los logs del pod: kubectl logs -n kube-system secrets-store-csi-driver-45w8a

secrets-store-csi-driver\" cannot list resource \"secrets\" in API group \"\" at the cluster scope\n"
I0613 20:17:17.679272       1 reflector.go:424] "pkg/mod/k8s.io/client-go@v0.26.4/tools/cache/reflector.go:169: failed to list *v1.Secret: secrets is forbidden: User \"system:serviceaccount:kube-system:secrets-store-csi-driver\" cannot list resource \"secrets\" in API group \"\" at the cluster scope\n"
E0613 20:17:17.679334       1 reflector.go:140] "pkg/mod/k8s.io/client-go@v0.26.4/tools/cache/reflector.go:169: Failed to watch *v1.Secret: failed to list *v1.Secret: secrets is forbidden: User \"system:serviceaccount:kube-system:secrets-store-csi-driver\" cannot list resource \"secrets\" in API group \"\" at the cluster scope\n"
I0613 20:18:00.577364       1 reflector.go:424] "pkg/mod/k8s.io/client-go@v0.26.4/tools/cache/reflector.go:169: failed to list *v1.Secret: secrets is forbidden: User \"system:serviceaccount:kube-system:secrets-store-csi-driver\" cannot list resource \"secrets\" in API group \"\" at the cluster scope\n"

luego de agregar estos permisos [list, watch, get] me dio este error:

I0613 20:46:34.691739       1 secretproviderclasspodstatus_controller.go:342] "The secret operation failed with forbidden error. If you installed the CSI driver using helm, ensure syncSecret.enabled=true is set.\n"
E0613 20:46:34.694549       1 secretproviderclasspodstatus_controller.go:338] "failed to create Kubernetes secret" err="secrets is forbidden: User \"system:serviceaccount:kube-system:secrets-store-csi-driver\" cannot create resource \"secrets\" in API group \"\" in the namespace \"fincuo\"" spc="fincuo/api-funded-natives-secrets" pod="fincuo/api-funded-natives-5df4c46dc7-zgm74" secret="fincuo/api-funded-natives-secrets" spcps="fincuo/api-funded-natives-5df4c46dc7-zgm74-fincuo-api-funded-natives-secrets"

y lo que agregue al ClusterRole quedo fue:

- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch

quedando asi el recurso:

 # Source: secrets-store-csi-driver/templates/role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: secretproviderclasses-role
  labels:
    app.kubernetes.io/instance: "secrets-store-csi-driver"
    app.kubernetes.io/managed-by: "Helm"
    app.kubernetes.io/name: "secrets-store-csi-driver"
    app.kubernetes.io/version: "1.4.3"
    app: secrets-store-csi-driver
    helm.sh/chart: "secrets-store-csi-driver-1.4.3"
rules:
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - create
  - patch
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - secrets-store.csi.x-k8s.io
  resources:
  - secretproviderclasses
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - secrets-store.csi.x-k8s.io
  resources:
  - secretproviderclasspodstatuses
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - secrets-store.csi.x-k8s.io
  resources:
  - secretproviderclasspodstatuses/status
  verbs:
  - get
  - patch
  - update
- apiGroups:
  - storage.k8s.io
  resourceNames:
  - secrets-store.csi.k8s.io
  resources:
  - csidrivers
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch

[mohamed-jawad-etg]

facing the same issue, can't find away around and I don't want to use OIDC

[jgrigg]

I am also very interested in this. We have adopted Pod Identities for general cross-account AWS access but are having to resort to IRSA for mounted secrets (via ESO) which is yuk.

Worth noting that we are also using role chaining (via profile/ini config). Ideally the CSI driver would allow configuration of a role to assume/chain in such an 'external' account.

[psivananda]

modified code and it's working for me for both IRSA and PIA

To make it work for PIA

1. we need to update helm chart to add cluster name as ENV
2. build helm chart, docker image and push to your registries
3. Remove eks.amazonaws.com/role-arn annotations from serviceaccount

https://github.com/psivananda/secrets-store-csi-driver-provider-aws/commit/8eb8b553cbb7a9712c51b0ac27bddb546f7fbf8a