Unable to mount secret, "Failed fetching secret <secretName>: RequestCanceled: request context canceled"

[oddball-jwaclawski]

Describe the bug Attempts to mount a secret inside a pod fail using very standard, very straightforward configurations.

To Reproduce

Steps to reproduce the behavior:

1. Install the secrets-store-csi-driver using Helm:

$ helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver --namespace kube-system --set syncSecret.enabled=true,enableSecretRotation=true

2. Deploy AWS Provider resources using Kubernetes manifests:

$ kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml

3. Create some resources to use during testing

$ export REGION=us-east-1 $ export CLUSTERNAME=this-is-my-cluster $ aws --region "$REGION" secretsmanager create-secret --name MySecret --secret-string '{"username":"memeuser", "password":"hunter2"}' $ export POLICY_ARN=$(aws --region "$REGION" --query Policy.Arn --output text iam create-policy --policy-name my-test-policy-name --path /my/necessary/iampath/ --policy-document '{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "secretsmanager:*", "Resource": "*" } ] }')

$ (skipped manual creation of OIDC provider; created with cluster build IAC)

The following steps were run using EKSCTL in documentation, I could not use EKSCTL due to my environment's requirement for pathing IAM roles and policies and EKSCTL not providing a method for handling this. I broke out the operations I'm assuming EKSCTL performs with "eksctl create iamserviceaccount"

$ aws iam create-role \ --role-name my-test-role \ --assume-role-policy-document '{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "eks.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }' \ --path "/my/necessary/iampath/" \ --permissions-boundary "arn:aws:iam::1234567890:policy/my/necessary-boundary-policy"

$ aws iam attach-role-policy --role-name my-test-role --policy-arn "$POLICY_ARN"

$ kubectl create serviceaccount my-test-sa $ kubectl annotate serviceaccount my-test-sa eks.amazonaws.com/role-arn=arn:aws:iam::1234567890:role/my/necessary/iampath/my-test-role

4. Deploy and monitor test resources

$ kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/examples/ExampleSecretProviderClass.yaml

$ kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/examples/ExampleDeployment.yaml

5. Notice that the test pod is not spinning up, so check the logs in the AWS Provider pods:

I0311 20:21:02.965312 1 server.go:124] Servicing mount request for pod test-deployment-5bbdbff64c-nbcjs in namespace default using service account my-test-sa with region(s) us-east-1 I0311 20:21:02.975217 1 auth.go:123] Role ARN for default:my-test-sa is arn:aws:iam::1234567890:role/my/necessary/iampath/my-test-role W0311 20:23:02.962483 1 secrets_manager_provider.go:84] us-east-1: Failed fetching secret MySecret: RequestCanceled: request context canceled caused by: context canceled E0311 20:23:02.962514 1 server.go:151] Failure getting secret values from provider type secretsmanager: Failed to fetch secret from all regions: MySecret

Do you also notice this bug when using a different secrets store provider (Vault/Azure/GCP...)? Not Tested

Expected behavior Expecting the AWS Provider to catch the SecretProviderClass requested during test pod initialization and mount the requested secret inside the pod as a volume.

[oddball-jwaclawski]

Was missing the necessary Secrets endpoint.