How to get more info on "EC2 Metadata creds request returned error"

[jakesgordon]

I'm trying to use IAM roles for service accounts to allow external-secrets to pull from AWS Secrets Manager. I have created an IAM role that allows me to pull secrets and successfully tested it inside a pod that uses that service account to prove I can (using the aws-cli) pull secrets, so I'm fairly confident the role is configured correctly. I've also confirmed that the external-secrets pod itself is running as that service account (the ENV inside the pod has a valid AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE)... however the logs show that it's failing to access secrets manager when trying to pull a secret with a somewhat vague "EC2 Metadata creds request returned error".

I've updated the LOG_LEVEL on the pod to debug but not much additional information.

Is there any way to get deeper insight into exactly what error the EC2 Metadata creds request is returning.

This is the full output of the external-secrets pod log error...

{
  "level": 50,
  "time": "2020-02-06T17:12:43.503Z",
  "pid": 17,
  "hostname": "platform-external-secrets-85c6f84466-2qs98",
  "message": "Missing credentials in config",
  "statusCode": 500,
  "retryable": true,
  "code": "CredentialsError",
  "originalError": {
    "message": "Could not load credentials from any providers",
    "statusCode": 500,
    "retryable": true,
    "time": "2020-02-06T17:12:43.503Z",
    "code": "CredentialsError",
    "originalError": {
      "message": "Missing credentials in config",
      "statusCode": 500,
      "retryable": true,
      "time": "2020-02-06T17:12:43.503Z",
      "code": "CredentialsError",
      "originalError": {
        "message": "Could not load credentials from any providers",
        "statusCode": 500,
        "retryable": true,
        "time": "2020-02-06T17:12:43.503Z",
        "code": "CredentialsError",
        "originalError": {
          "message": "EC2 Metadata creds request returned error",
          "statusCode": 500,
          "retryable": true,
          "time": "2020-02-06T17:12:43.502Z"
        }
      }
    }
  },
  "msg": "failure while polling the secret platform-infrastructure/platform-example-secret",
  "stack": "CredentialsError: Missing credentials in config\n    at IncomingMessage.<anonymous> (/app/node_modules/aws-sdk/lib/util.js:895:34)\n    at IncomingMessage.emit (events.js:215:7)\n    at IncomingMessage.EventEmitter.emit (domain.js:476:20)\n    at endReadableNT (_stream_readable.js:1183:12)\n    at processTicksAndRejections (internal/process/task_queues.js:80:21)",
  "type": "Error",
  "v": 1
}

Thanks!

[jakesgordon]

I've found my issue. The problem was that even though the pod used that service account, because you are running the container as a non-root user (node) it didn't have permission to actually read the AWS_WEB_IDENTITY_TOKEN file.

The solution is to add a securityContext to the external secrets deployment template spec

      securityContext:
        fsGroup: 1337

... as described in https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html#pod-configuration

[Flydiverny]

https://github.com/godaddy/kubernetes-external-secrets/issues/161#issuecomment-540591653 for future readers