AWS provider pod failing with "panic: runtime error: invalid memory address or nil pointer dereference"

[egarbi]

Describe the bug When trying to mount secrets coming from SSM in JSON format the secrets fail to be mounted.

To Reproduce

Steps to reproduce the behaviour: Adding secrets.yaml as follows:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: grafana-db-creds
  namespace: grafana
spec:
  provider: aws
  secretObjects:
    - secretName: grafana-db-creds # pragma: allowlist secret
      type: Opaque
      data:
        - key: username
          objectName: username
        - key: password
          objectName: password
  parameters:
    region: eu-central-1
    objects: |
      - objectName: "arn:aws:ssm:eu-central-1:12345678910:parameter/Shared/Grafana/db-creds"
        objectType: ssmparameter
        jmesPath:
          - path: username
            objectAlias: username
          - path: password
            objectAlias: password

... and then try to use it within a deployment as

      volumes:
        - name: grafana-db-creds
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: grafana-db-creds
      volumeMounts:
        - name: grafana-db-creds
          mountPath: /mnt/secrets
          readOnly: true
      env:
        - name: GF_SECURITY_ADMIN_USER
          valueFrom:
            secretKeyRef:
              name: grafana-db-creds
              key: username
        - name: GF_SECURITY_ADMIN_PASSWORD
          valueFrom:
            secretKeyRef:
              name: grafana-db-creds
              key: password

From describe pod I can see

Warning  FailedMount  7m44s                 kubelet            MountVolume.SetUp failed for volume "grafana-db-creds" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod │
│  grafana/grafana-5f9f594965-5rvlj, err: rpc error: code = Unavailable desc = error reading from server: EOF

From secrets-store-csi-driver-provider-aws* pod I can see

I0425 14:27:09.301611       1 main.go:32] Starting secrets-store-csi-driver-provider-aws version 1.0.r2-50-g5b4aca1-2023.06.09.21.19                                                                     │
│ I0425 14:27:09.399141       1 main.go:77] Listening for connections on address: /etc/kubernetes/secrets-store-csi-providers/aws.sock                                                                     │
│ I0425 14:27:34.144366       1 server.go:124] Servicing mount request for pod grafana-5f9f594965-5rvlj in namespace grafana using service account grafana with region(s) eu-central-1                     │
│ I0425 14:27:34.198954       1 auth.go:123] Role ARN for grafana:grafana is arn:aws:iam::12345678910:role/GrafanaPod-eu-central-1-toolshed20240425130235351900000001                                     │
│ panic: runtime error: invalid memory address or nil pointer dereference                                                                                                                                  │
│ [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x16db39c]                                                                                                                                  │
│                                                                                                                                                                                                          │
│ goroutine 37 [running]:                                                                                                                                                                                  │
│ github.com/aws/secrets-store-csi-driver-provider-aws/provider.(*ParameterStoreProvider).fetchParameterStoreBatch(0xc00068c000?, {0x0, {0xc00059a0a0, 0xc}, {0x1f0d760, 0xc000012320}}, {0x1ed3790, 0xc00 │
│     /Users/martysi/Documents/workspaces/upstream_repos/secrets-store-csi-driver-provider-aws/provider/parameter_store_provider.go:157 +0x77c                                                             │
│ github.com/aws/secrets-store-csi-driver-provider-aws/provider.(*ParameterStoreProvider).fetchParameterStoreValue(0xc000494d50, {0x1ed3790, 0xc000596000}, {0xc000012308, 0x1, 0x1}, 0x16dd5ca?)          │
│     /Users/martysi/Documents/workspaces/upstream_repos/secrets-store-csi-driver-provider-aws/provider/parameter_store_provider.go:85 +0x1bc                                                              │
│ github.com/aws/secrets-store-csi-driver-provider-aws/provider.(*ParameterStoreProvider).GetSecretValues(0x185f480?, {0x1ed3790, 0xc000596000}, {0xc000012308, 0x1, 0x1}, 0xc00078a2b0?)                  │
│     /Users/martysi/Documents/workspaces/upstream_repos/secrets-store-csi-driver-provider-aws/provider/parameter_store_provider.go:63 +0x18e                                                              │
│ github.com/aws/secrets-store-csi-driver-provider-aws/server.(*CSIDriverProviderServer).Mount(0xc000443980, {0x1ed3790, 0xc000596000}, 0xc000598000)                                                      │
│     /Users/martysi/Documents/workspaces/upstream_repos/secrets-store-csi-driver-provider-aws/server/server.go:149 +0xba8                                                                                 │
│ sigs.k8s.io/secrets-store-csi-driver/provider/v1alpha1._CSIDriverProvider_Mount_Handler({0x1a4dc20?, 0xc000443980}, {0x1ed3790, 0xc000596000}, 0xc0001e4070, 0x0)                                        │
│     /Users/martysi/go/pkg/mod/sigs.k8s.io/secrets-store-csi-driver@v1.3.3/provider/v1alpha1/service_grpc.pb.go:132 +0x170                                                                                │
│ google.golang.org/grpc.(*Server).processUnaryRPC(0xc00080e000, {0x1ed83a0, 0xc00051cd00}, 0xc00053a000, 0xc000443a10, 0x2b930f8, 0x0)                                                                    │
│     /Users/martysi/go/pkg/mod/google.golang.org/grpc@v1.53.0/server.go:1336 +0xd33                                                                                                                       │
│ google.golang.org/grpc.(*Server).handleStream(0xc00080e000, {0x1ed83a0, 0xc00051cd00}, 0xc00053a000, 0x0)                                                                                                │
│     /Users/martysi/go/pkg/mod/google.golang.org/grpc@v1.53.0/server.go:1704 +0xa36                                                                                                                       │
│ google.golang.org/grpc.(*Server).serveStreams.func1.2()                                                                                                                                                  │
│     /Users/martysi/go/pkg/mod/google.golang.org/grpc@v1.53.0/server.go:965 +0x98                                                                                                                         │
│ created by google.golang.org/grpc.(*Server).serveStreams.func1                                                                                                                                           │
│     /Users/martysi/go/pkg/mod/google.golang.org/grpc@v1.53.0/server.go:963 +0x28a

Do you also notice this bug when using a different secrets store provider (Vault/Azure/GCP...)? Yes/No

If yes, the issue is likely with the k8s Secrets Store CSI driver, not the AWS provider. Open an issue in that repo.

Expected behavior The secret should be retrieved from SSM and being mounted to be available for the container.

Environment: OS, Go version, etc. EKS, AWS, kubernetes 1.28, driver version can be seen on details above.

Additional context I was trying to deploy grafana using Helm in EKS using kubectl manually. The role associated with the service account exists and has the proper permissions. The SSM path exists and its a JSON with user, password on it

[jbct]

Thank you for the bug report, @egarbi. We will take a look

[YuvalShAz]

Hi @egarbi I see that in your supplied secrets.yaml provider class you seem to be using an ssmparameter objectType and a full arn as the objectName. However, for parameter store objects you must use the name and not the full arn. From the README:

For SSM Parameter Store, this must be the Name of the parameter and can not be a full ARN.

I've reproduced your problem using the full arn:

MountVolume.SetUp failed for volume "grafana-db-creds" : rpc error: code = Unavailable desc = failed to mount secrets store objects for pod default/pod, err: rpc error: code = Unavailable desc = error reading from server: EOF

And I've confirmed that the container starts up and mounts as intended when using just the name.

An additional note: I can't see your full deployment specification file so it's impossible to know, but if you check the examples/ExampleDeployment.yaml you'll see that the

volumeMounts:
  - name: grafana-db-creds
    mountPath: /mnt/secrets
    readOnly: true
env:
  - name: GF_SECURITY_ADMIN_USER
    valueFrom:
      secretKeyRef:
        name: grafana-db-creds
        key: username
  - name: GF_SECURITY_ADMIN_PASSWORD
    valueFrom:
      secretKeyRef:
        name: grafana-db-creds
        key: password

block is included in the containers section and is therefore not at the same indentation as volumes which is what your deployment specification seems to be.