error connecting to provider "aws": provider not found: provider "aws"

[riosje]

Update

ok I've fixed this issue setting the toleration value on secrets-store-csi-driver-provider-aws

tolerations: 
  - operator: Exists

resource "helm_release" "secrets-provider-aws" {
  repository       = "https://aws.github.io/secrets-store-csi-driver-provider-aws"
  name             = "aws-secrets-manager"
  chart            = "secrets-store-csi-driver-provider-aws"
  namespace        = "kube-system"
  create_namespace = true
  version          = "0.3.4"

  values = [<<-EOF
tolerations: 
  - operator: Exists
EOF
  ]
}

CLOSING THIS IN FAVOR OF #267

Describe the bug

I'm not being able to attach a secret on a pod, i get this error.

MY_POD_LOG

MountVolume.SetUp failed for volume "secrets-store-inline" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod ns-ai-service/ns-ai-service-858bf5f554-2jp8c, err: error connecting to provider "aws": provider not found: provider "aws"

ecrets-store-csi-driver LOG

E0831 21:56:53.626573       1 nodeserver.go:242] "failed to mount secrets store object content" err="error connecting to provider \"aws\": provider not found: provider \"aws\"" pod="ns-ai-service/ns-ai-service-858bf5f554-2jp8c"
I0831 21:56:53.626595       1 nodeserver.go:88] "unmounting target path as node publish volume failed" targetPath="/var/lib/kubelet/pods/73dffdf5-a15c-4ece-b330-b627e636094e/volumes/kubernetes.io~csi/secrets-store-inline/mount" pod="ns-ai-service/ns-ai-service-858bf5f554-2jp8c"

aws-secrets-manager-secrets-store-csi-driver-provider-aws LOG

I0831 20:31:56.271368       1 main.go:32] Starting secrets-store-csi-driver-provider-aws version 1.0.r2-50-g5b4aca1-2023.06.09.21.19
I0831 20:31:56.271988       1 main.go:77] Listening for connections on address: /etc/kubernetes/secrets-store-csi-providers/aws.sock

I have 2 nodes, both nodes show the same 2 lines log

Already tried some of the solutions discussed on this issue #91 but found the following:

• the tolerations on secrets-store-csi-driver are already set on [{"operator": "Exists"}]

• the grpcSupportedProviders setting is not available or needed anymore on the secrets-store-csi-driver chart

To Reproduce

Steps to reproduce the behavior:

Deploy the CSI drivers helm charts (I use terraform)

resource "helm_release" "secrets-store-csi-driver" {
  repository       = "https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts"
  name             = "secrets-store-csi-driver"
  chart            = "secrets-store-csi-driver"
  namespace        = "kube-system"
  create_namespace = true
  version          = "1.3.4"

  set {
    name  = "syncSecret.enabled"
    value = "true"
  }
}

resource "helm_release" "secrets-provider-aws" {
  repository       = "https://aws.github.io/secrets-store-csi-driver-provider-aws"
  name             = "aws-secrets-manager"
  chart            = "secrets-store-csi-driver-provider-aws"
  namespace        = "kube-system"
  create_namespace = true
  version          = "0.3.4"

}

SecretProviderClass

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: ns-ai-service-secrets
  namespace: ns-ai-service
spec:
  provider: aws
  secretObjects:
  - data:
    - key: NS_AIS_REDIS_SECURE
      objectName: NS_AIS_REDIS_SECURE
    secretName: "ns-ai-secrets"
    type: Opaque 
  parameters:
    objects: |
        - objectName: "/ns-ai-service/staging/NS_AIS_REDIS_SECURE"
          objectAlias: "NS_AIS_REDIS_SECURE"
          objectType: "ssmparameter"

serviceAccount

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::XXXXXX:role/k8s-ns-ai-role
  name: ns-ai-service
  namespace: ns-ai-service

Deployment

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ns-ai-service
  labels:
    app.kubernetes.io/name: ns-ai-service
  namespace: ns-ai-service
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: ns-ai-service
  replicas: 1
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ns-ai-service
    spec:
      serviceAccountName: ns-ai-service
      volumes:
      - name: secrets-store-inline
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: "ns-ai-service-secrets"
      containers:
        - name: ns-ai-service
          image: debian
          command: ['sh', '-c', "sleep 2000000000000000000"]
          imagePullPolicy: Always
          volumeMounts:
          - name: secrets-store-inline
            mountPath: "/mnt/secrets-store"
            readOnly: true

If yes, the issue is likely with the k8s Secrets Store CSI driver, not the AWS provider. Open an issue in that repo.

Expected behavior I would like to make it work.

Environment: Kubernetes version 1.25

AMI release version 1.25.12-20230825

AMI type AL2_x86_64

Platform version eks.6

Terraform Helm source = "hashicorp/helm" version = ">= 2.9.0"

Thank you guys for any hint you can give to me.

[pcnoic]

@riosje did you ever get this resolved?

[riosje]

Hi @pcnoic, yes take too look to this issue https://github.com/aws/secrets-store-csi-driver-provider-aws/issues/267