Can't' obtain AWS credentials inside k8s pod in China region

GenrikhFetischev commented 1 year ago

Checkboxes for prior research

[X] I've gone through Developer Guide and API reference
[X] I've checked AWS Forums and StackOverflow.
[X] I've searched for previous similar issues and didn't find any solution.

Describe the bug

Hello! Could you please help me with the following problem: I need to obtain AWS creds from token file inside a pod in k8s cluster. Previously I used aws-sdk v2 for that purpose and it worked, the problem occured after migration to sdk v3. The error is:

InvalidIdentityTokenException: No OpenIDConnect provider found in your account for https://oidc.eks.cn-north-1.amazonaws.com.cn/id/<ID>
    at de_InvalidIdentityTokenExceptionRes (/path/node_modules/@aws-sdk/client-sts/dist-cjs/protocols/Aws_query.js:436:23)
    at de_AssumeRoleWithWebIdentityCommandError (/path/node_modules/@aws-sdk/client-sts/dist-cjs/protocols/Aws_query.js:219:25)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /path/node_modules/@smithy/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
    at async /path/node_modules/@smithy/middleware-retry/dist-cjs/retryMiddleware.js:27:46
    at async /path/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26
    at async /path/node_modules/@aws-sdk/client-sts/dist-cjs/defaultStsRoleAssumers.js:58:33
    at async file:///x/index.js:9:17 {
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 400,
    requestId: '<ID>',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  Error: {
    Type: 'Sender',
    Code: 'InvalidIdentityToken',
    Message: 'No OpenIDConnect provider found in your account for https://oidc.eks.cn-north-1.amazonaws.com.cn/id/<ID>',
    message: 'No OpenIDConnect provider found in your account for https://oidc.eks.cn-north-1.amazonaws.com.cn/id/<ID>'
  },
  RequestId: '<ID>',
  xmlns: 'https://sts.amazonaws.com/doc/2011-06-15/'
}

I went through the following troubleshoot process - https://repost.aws/knowledge-center/eks-troubleshoot-oidc-and-irsa and looks like everything is ok according to the checklist.

The problem is observed only in China region clusters, in the rest regions everything is ok.

SDK version number

@aws-sdk/credential-providers@3.385.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v20.5.1

Reproduction Steps

I'm not sure I can provide a k8s cluster to reproduce the bug. The code which trying to retrieve creds is the following:

const inPodCreds = fromTokenFile()();

Observed Behavior

I'm getting the following error:

InvalidIdentityTokenException: No OpenIDConnect provider found in your account for https://oidc.eks.cn-north-1.amazonaws.com.cn/id/<ID>
    at de_InvalidIdentityTokenExceptionRes (/path/node_modules/@aws-sdk/client-sts/dist-cjs/protocols/Aws_query.js:436:23)
    at de_AssumeRoleWithWebIdentityCommandError (/path/node_modules/@aws-sdk/client-sts/dist-cjs/protocols/Aws_query.js:219:25)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /path/node_modules/@smithy/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
    at async /path/node_modules/@smithy/middleware-retry/dist-cjs/retryMiddleware.js:27:46
    at async /path/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26
    at async /path/node_modules/@aws-sdk/client-sts/dist-cjs/defaultStsRoleAssumers.js:58:33
    at async file:///x/index.js:9:17 {
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 400,
    requestId: '<ID>',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  Error: {
    Type: 'Sender',
    Code: 'InvalidIdentityToken',
    Message: 'No OpenIDConnect provider found in your account for https://oidc.eks.cn-north-1.amazonaws.com.cn/id/<ID>',
    message: 'No OpenIDConnect provider found in your account for https://oidc.eks.cn-north-1.amazonaws.com.cn/id/<ID>'
  },
  RequestId: '<ID>',
  xmlns: 'https://sts.amazonaws.com/doc/2011-06-15/'
}

Expected Behavior

I expect to get AWS credentials from provider.

Possible Solution

No response

Additional Information/Context

The problem is observed only in China region clusters, in the rest regions everything is ok. The same code can retrieve apps in others cluster. Code with aws-sdk v2 works in China region k8s clusters as well, the problem only being observed with aws-sdk-3

yenfryherrerafeliz commented 1 year ago

Hi @GenrikhFetischev, sorry to hear about your issues. There are different reasons why you could be getting this error. But, lets try to figure out the basics first:

Can you please go to AWS Console -> IAM -> Identity providers and do you see the provider we get in the error there?, which is oidc.eks.cn-north-1.amazonaws.com.cn/id/. If you do not see it there is because it was deactivated.
Can you please make sure OIDC is enabled by running the following: aws eks describe-cluster --name "your-cluster-name" --query "cluster.identity.oidc.issuer".

Please let me know the results.

Thanks!

github-actions[bot] commented 1 year ago

This issue has not received a response in 1 week. If you still think there is a problem, please leave a comment to avoid the issue from automatically closing.

GenrikhFetischev commented 1 year ago

Hello @yenfryherrerafeliz! Thanks for your response. As I mentioned I went through the following troubleshoot process - https://repost.aws/knowledge-center/eks-troubleshoot-oidc-and-irsa that includes steps you advised to start with.

In particular:

Check if you have an existing IAM OIDC provider for your cluster
Check if your IAM role has an attached needful IAM policy with required permissions
Verify that the IAM role trust relations are correctly set
Check if you created a service account
Verify that the service account has the correct IAM role annotations
Verify that you correctly specified the serviceAccountName in your pod
Check the environment variables and permissions
Verify that the application uses a supported AWS SDK
Check the pod user and group
Recreate pods
Verify that the audience is correct
Verify that you configured the correct thumbprint
For the AWS China Region, check the AWS_DEFAULT_REGION environment variable

And everything looks as it intended to be. Also, I mentioned that authorization works in aws sdk v2 in the exactly same cluster, OIDC providers exists, active and work.

aBurmeseDev commented 3 weeks ago

Hi @GenrikhFetischev - sorry for the long silence here. Is this still an ongoing issue for you?

The error suggests that the OpenID Connect (OIDC) provider is not set up correctly in your AWS account for the China (Beijing) region (cn-north-1).

The OIDC provider is a crucial component for Kubernetes clusters using IAM Roles for Service Accounts (IRSA) to work with EKS. It allows the Kubernetes service accounts to assume AWS IAM roles and obtain temporary AWS credentials.

Since you mentioned that the problem only occurs in the China region clusters and not in other regions, it could be related to a specific configuration or setup required for the China regions.

I'd be happy to further assist you if you're still working on this project. Best, John

GenrikhFetischev commented 3 weeks ago

Hello! @aBurmeseDev Is it possible that with OIDC provider which is not correctly set up AWS SDK v2 works as expected?

aBurmeseDev commented 3 weeks ago

To better understand and address the issue, we would need to examine your specific configuration and setup, particularly the regions you have configured. I tried to reproduce the problem in the cn-north-1 region but was unable to do so successfully.

GenrikhFetischev commented 2 weeks ago

Ok, let me try to reproduce that one more time. It's been a while since I had that problem, may be in that year the problem was fixed. I don't know current state because we have been using SDK v2 due to the problem

github-actions[bot] commented 1 week ago

This issue has not received a response in 1 week. If you still think there is a problem, please leave a comment to avoid the issue from automatically closing.

aws / aws-sdk-js-v3