Closed jakesgordon closed 4 years ago
I've found my issue. The problem was that even though the pod used that service account, because you are running the container as a non-root user (node) it didn't have permission to actually read the AWS_WEB_IDENTITY_TOKEN file.
The solution is to add a securityContext to the external secrets deployment template spec
securityContext:
fsGroup: 1337
... as described in https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html#pod-configuration
I'm trying to use IAM roles for service accounts to allow external-secrets to pull from AWS Secrets Manager. I have created an IAM role that allows me to pull secrets and successfully tested it inside a pod that uses that service account to prove I can (using the aws-cli) pull secrets, so I'm fairly confident the role is configured correctly. I've also confirmed that the external-secrets pod itself is running as that service account (the ENV inside the pod has a valid AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE)... however the logs show that it's failing to access secrets manager when trying to pull a secret with a somewhat vague "EC2 Metadata creds request returned error".
I've updated the LOG_LEVEL on the pod to debug but not much additional information.
Is there any way to get deeper insight into exactly what error the EC2 Metadata creds request is returning.
This is the full output of the external-secrets pod log error...
Thanks!