Cannot perform SSM or Route53 operations with IRSA

[Velavee]

Checkboxes for prior research

• [X] I've gone through Developer Guide and API reference
• [X] I've checked AWS Forums and StackOverflow.
• [X] I've searched for previous similar issues and didn't find any solution.

Describe the bug

When I try to perform ssm or route53 operations with IRSA, I get the following error (the operations have worked when the same IAM role is assumed through a different method) even though I have tried explicitly passing the region using multiple methods:

if (runtimeConfig.region === void 0) {
      throw new Error("Region is missing from runtimeConfig");
    }
    const region = runtimeConfig.region;
    if (typeof region === "string") {
      return region;
    }
    return region();
  }: InvalidIdentityTokenException: No OpenIDConnect provider found in your account for https://oidc.eks.us-east-1.amazonaws.com/id/<redacted>

At first this looked very similar to this dynamDB issue: https://github.com/aws/aws-sdk-js-v3/issues/5749#issuecomment-1922118058

Elsewhere in our codebase, the region was passed to the inner sts client and fixed the issue for dynamDB (without updating aws-sdk). For route53 and ssm, I attempted to simply update the aws-sdk from 3.496.0 to 3.637.0 because it seemed there was a fix for the inner client to assume the region of the outer client (see repro steps). This did not work, so I explicitly passed the region to the inner client, which also did not work.

SDK version number

@aws-sdk/client-ssm 3.637.0 and route-53 3.637.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

node v20.8.0

Reproduction Steps

1. Have an IRSA set up to assume an IAM role that has full SSM and Route53 permissions.
2. Instantiate the client in one of the following ways:

a.

const {
  SSMClient,
  GetParametersCommand,
  GetParameterCommand,
} = require("@aws-sdk/client-ssm");

const ssm = new SSMClient({
  region: "us-west-1",
});

b.

const {
  SSMClient,
  GetParametersCommand,
  GetParameterCommand,
} = require("@aws-sdk/client-ssm");

const { fromNodeProviderChain } = require("@aws-sdk/credential-providers");

const ssm = new SSMClient({
  region: "us-west-1",
  credentials: fromNodeProviderChain({
    clientConfig: { region: "us-west-1" },
  }),
});

c.

process.env.AWS_REGION = "us-west-1";
const {
  SSMClient,
  GetParametersCommand,
  GetParameterCommand,
} = require("@aws-sdk/client-ssm");

const { fromNodeProviderChain } = require("@aws-sdk/credential-providers");

const ssm = new SSMClient({
  region: "us-west-1",
  credentials: fromNodeProviderChain({
    clientConfig: { region: "us-west-1" },
  }),
});

Observed Behavior

Error:

if (runtimeConfig.region === void 0) {
      throw new Error("Region is missing from runtimeConfig");
    }
    const region = runtimeConfig.region;
    if (typeof region === "string") {
      return region;
    }
    return region();
  }: InvalidIdentityTokenException: No OpenIDConnect provider found in your account for https://oidc.eks.us-east-1.amazonaws.com/id/<redacted>

Expected Behavior

The ssm and route53 clients should be successfully instantiated.

Possible Solution

No response

Additional Information/Context

No response

[RanVaknin]

Hi @Velavee ,

It's not clear what the error is here since you added an internal code snippet from the SDK.

Is the error you are seeing : InvalidIdentityTokenException: No OpenIDConnect provider found in your account for https://oidc.eks.us-east-1.amazonaws.com/id/<redacted>?

If so, you need to correctly configure your IRSA provider.

Also, instead of using fromNodeProviderChain you can use the specific credential provider that uses IRSA:

const client = new Route53Client({
  region: "us-west-1",
  credentials: fromWebToken({
    clientConfig: { region: "us-west-1" },
    logger: console
  })
});

I also added the logger into the credential provider so you can observe what is failing in the implicit AssumeRoleWithWebIdentity that the SDK makes to exchange your token for a set of temporary credentials.

Thanks, Ran~

[Velavee]

@RanVaknin the code snippet was output as a part of the error log, I'm guessing a stacktrace. I am sure my IRSA provider is correctly configured, but I will try your suggestion of using fromWebToken, thank you.

[Velavee]

@RanVaknin This configuration throws a different error (again, code snippet was a part of the error):

if (runtimeConfig.region === void 0) {
      throw new Error("Region is missing from runtimeConfig");
    }
    const region = runtimeConfig.region;
    if (typeof region === "string") {
      return region;
    }
    return region();
  }: ValidationError: 2 validation errors detected: Value null at 'roleArn' failed to satisfy constraint: Member must not be null; Value at 'webIdentityToken' failed to satisfy constraint: Member must not be null
  Error: Failed to get parameters from SSM async () => {
      if (runtimeConfig.region === void 0) {
        throw new Error("Region is missing from runtimeConfig");

One of the reasons I am investigating the sdk is I have checked the creds for the pod and I am sure the creds are present.

[default]
role_arn = arn:aws:iam::<redacted>:role/<redacted>
role_session_name = <redacted>
source_profile = irsa
[irsa]
role_arn = arn:aws:iam::<redacted>:role/<redacted>
web_identity_token_file = /var/run/secrets/eks.amazonaws.com/serviceaccount/token

Can you confirm whether the problem I have been seeing is related to this older issue: https://github.com/aws/aws-sdk-js-v3/issues/5749

If it is, it was my understanding that this fix was supposed to make it so explicitly passing region to the inner sts client was unnecessary: https://github.com/aws/aws-sdk-js-v3/pull/5800

Do you know if it's expected that I would have to pass region to both the inner and outer clients?

[Velavee]

ssm client configuration I tried for clarity:

const {
  SSMClient,
  GetParametersCommand,
  GetParameterCommand,
} = require("@aws-sdk/client-ssm");
const { fromWebToken } = require("@aws-sdk/credential-providers");
const ssm = new SSMClient({
  region: "us-west-1",
  credentials: fromWebToken({
    clientConfig: { region: "us-west-1" },
    logger: console,
  }),
});

[RanVaknin]

Hi @Velavee ,

When you are specifying a specific provider like fromTokenFile() the SDK will not look at your INI file that defines the role and token. Is there a reason you are using an INI file in the first place?

Reading the error you get here:

}: ValidationError: 2 validation errors detected: Value null at 'roleArn' failed to satisfy constraint: Member must not be null; Value at 'webIdentityToken' failed to satisfy constraint: Member must not be null

It seems like the SDK is not able to read the role or the token from the env variables that your pod should be injected with on boot.

If your EKS cluster has an IRSA provider setup correctly, your pod should be created with certain env variables injected into it, like the location of your token file, and the role. https://github.com/aws/aws-sdk-js-v3/blob/main/packages/credential-provider-web-identity/src/fromTokenFile.ts#L8

The SDK will read those from the environment, and make an implicit STS call "under the hood" (the internal client) to exchange the role and the token for a set of temporary credentials (AssumeRoleWithWebIdentity).

Can you confirm whether the problem I have been seeing is related to this older issue: https://github.com/aws/aws-sdk-js-v3/issues/5749

I'm still not clear about what the issue is, so I can't say if its related to this, but I don't think so.

what does the output look like when you log the pod's env variables?

kubectl exec -it <pod-name> -- printenv

And from your nodejs application:

console.log(process.env);

You can also try to exec into the pod, and try to locate the token file at /var/run/secrets/eks.amazonaws.com/serviceaccount/token and see if theres an actual token file being written to the file system.

Let me know what you find. Thanks, Ran~