client-secrets-manager - ERROR: getApplicationSecrets The security token included in the request is invalid.

geoffcorey commented 1 month ago

Checkboxes for prior research

[X] I've gone through Developer Guide and API reference
[X] I've checked AWS Forums and StackOverflow.
[X] I've searched for previous similar issues and didn't find any solution.

Describe the bug

using aws sso login and running the application using @aws-sdk/client-secrets-manager version 3.623.0 we get the error.

If we drop back to 3.441.0 we are able to retrieve the AWS secrets

SDK version number

3.623.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

20.11.1

Reproduction Steps

OS/X Sonoma 14.5 Node 20.11.1


~/src/awsbug is 📦 v1.0.0 via  v20.11.1 on ☁️   (us-west-2)
❯ cat package.json
{
  "name": "awsbug",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "@aws-sdk/client-secrets-manager": "^3.623.0"
  }
}

~/src/awsbug is 📦 v1.0.0 via  v20.11.1 on ☁️   (us-west-2)
❯ cat index.js
const { SecretsManagerClient, GetSecretValueCommand } = require('@aws-sdk/client-secrets-manager');
const config = {
  aws: {
    region: 'us-west-2',
    secretName: 'my-key'
  }
};

const getApplicationSecrets = async () => {
  let client;

  client = new SecretsManagerClient({
    region: config.aws.region,
    logger: null
  });

  const command = new GetSecretValueCommand({
    SecretId: config.aws.secretName
  });

  const response = await client.send(command);
  return response.secretString;
}

getApplicationSecrets().then(r => { console.log('success') }).catch(console.err);

~/src/awsbug is 📦 v1.0.0 via  v20.11.1 on ☁️   (us-west-2)

❯ node index.js
/Users/gcorey/src/awsbug/node_modules/@smithy/smithy-client/dist-cjs/index.js:840
  const response = new exceptionCtor({
                   ^

UnrecognizedClientException: The security token included in the request is invalid.
    at throwDefaultError (/Users/gcorey/src/awsbug/node_modules/@smithy/smithy-client/dist-cjs/index.js:840:20)
    at /Users/gcorey/src/awsbug/node_modules/@smithy/smithy-client/dist-cjs/index.js:849:5
    at de_CommandError (/Users/gcorey/src/awsbug/node_modules/@aws-sdk/client-secrets-manager/dist-cjs/index.js:992:14)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /Users/gcorey/src/awsbug/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
    at async /Users/gcorey/src/awsbug/node_modules/@smithy/core/dist-cjs/index.js:165:18
    at async /Users/gcorey/src/awsbug/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
    at async /Users/gcorey/src/awsbug/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:22
    at async getApplicationSecrets (/Users/gcorey/src/awsbug/index.js:21:20) {
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 400,
    requestId: '38866c6c-e16e-4dbf-bbf5-1a6de4fc4fc0',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  __type: 'UnrecognizedClientException'
}

Change package to 3.441.0


~/src/awsbug is 📦 v1.0.0 via  v20.11.1 on ☁️   (us-west-2)
❯ vi package.json

~/src/awsbug is 📦 v1.0.0 via  v20.11.1 on ☁️   (us-west-2) took 20s
❯ rm package-lock.json

~/src/awsbug is 📦 v1.0.0 via  v20.11.1 on ☁️   (us-west-2)
❯ rm -fr node_modules

~/src/awsbug is 📦 v1.0.0 via  v20.11.1 on ☁️   (us-west-2)
❯ npm i

added 78 packages, and audited 79 packages in 2s

2 packages are looking for funding
  run `npm fund` for details

3 high severity vulnerabilities

To address all issues, run:
  npm audit fix --force

Run `npm audit` for details.

~/src/awsbug is 📦 v1.0.0 via  v20.11.1 on ☁️   (us-west-2) took 2s
❯ cat package.json
{
  "name": "awsbug",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "@aws-sdk/client-secrets-manager": "3.441.0"
  }
}

~/src/awsbug is 📦 v1.0.0 via  v20.11.1 on ☁️   (us-west-2)
❯ node index.js
success

Observed Behavior

upgrading package to latest from 3.441.0 breaks retrieval of secrets when using aws sso login on local machine running OS/X

Expected Behavior

Should work just like 3.441.0

Possible Solution

Don't ever upgrade?

Additional Information/Context

No response

aBurmeseDev commented 1 month ago

Hi @geoffcorey - thanks for reaching out.

The error "UnrecognizedClientException: The security token included in the request is invalid" suggests that the temporary security credentials used to authenticate the request to AWS Secrets Manager are either invalid or have expired.

Since you mentioned that you are using AWS SSO for authentication, it's likely that the temporary credentials obtained through the AWS SSO session have expired.

To help diagnose and resolve the issue, it would be helpful if you could provide more details on how you have configured the AWS SSO session and the AWS profile in your application. Specifically, please share the following information:

How are you initializing and configuring the AWS SSO session in your application?
If you're using the AWS CLI, please share the command(s) you're running to initiate the AWS SSO session and configure the AWS profile.
If you're using the AWS SDK for JavaScript, please share the code snippet where you're configuring the AWS credentials using the fromSSO method from the @aws-sdk/credential-providers package.
Are you setting any environment variables or using any external configuration files to configure the AWS credentials or the AWS SSO session?
What is the expiration time or duration of the AWS SSO session you're using?

Providing these details will help us better understand your setup and configuration, which can assist in identifying the root cause of the issue and suggesting appropriate solutions.

Please refer to the AWS documentation links provided below for guidance on configuring AWS SSO with the AWS CLI and the AWS SDK for JavaScript (v3). AWS CLI: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sso.html JavaScript SDK v3: https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/Package/-aws-sdk-credential-providers/#fromsso

geoffcorey commented 1 month ago

I call aws sso login from the command line. If I use the latest package version the app will not connect. if I use the 441 version it does.

version of the cli client

❯ aws --version aws-cli/2.17.22 Python/3.11.9 Darwin/23.5.0 source/x86_64

We have not moved to v3 aws-cli due to some issue. I will upgrade tomorrow and upgrade the package and see if it will work.

geoffcorey commented 1 month ago

So I am running the latest awscli client already and it is configured for SSO. Am I suppose to do something different for

const { SecretsManagerClient, GetSecretValueCommand } = require('@aws-sdk/client-secrets-manager');
const config = {
  aws: {
    region: 'us-west-2',
    secretName: 'my-key'
  }
};

const getApplicationSecrets = async () => {
  let client;

  client = new SecretsManagerClient({
    region: config.aws.region,
    logger: null
  });

  const command = new GetSecretValueCommand({
    SecretId: config.aws.secretName
  });

  const response = await client.send(command);
  return response.secretString;
}

getApplicationSecrets().then(r => { console.log('success') }).catch(console.err);

Because the latest fails with sso login and version 3.441.0 does not

aBurmeseDev commented 1 month ago

Thanks for sharing additional info.

I call aws sso login from the command line. If I use the latest package version the app will not connect. if I use the 441 version it does

I’m a bit unclear on the details, especially since you also mentioned being unable to connect with aws sso login. Could you clarify what happens when you attempt to log in? Normally, you should see a 'Successfully logged into' message in your command line. You can refer to the documentation for more details.

$ aws sso login --profile my-dev-profile
SSO authorization page has automatically been opened in your default browser. 
Follow the instructions in the browser to complete this authorization request.
Successfully logged into Start URL: https://my-sso-portal.awsapps.com/start

Let’s say you were able to successfully configure and login SSO with CLI, you can leverage these credentials in your JavaScript SDK code. Here's an example using credential providers: can use these credential providers in your JavaScript SDK code. See here.

const client = new FooClient({ credentials: fromSSO({ profile: "my-sso-profile" })});

geoffcorey commented 1 month ago

I successfully login using aws sso login

If I execute the code I posted above using 3.441.0 of @aws-sdk/client-secrets-manager the code will pull the secrets successfully. If I use 3.623.0 and execute the code it throws and error

❯ node index.js
/Users/gcorey/src/awsbug/node_modules/@smithy/smithy-client/dist-cjs/index.js:840
  const response = new exceptionCtor({
                   ^

UnrecognizedClientException: The security token included in the request is invalid.
    at throwDefaultError (/Users/gcorey/src/awsbug/node_modules/@smithy/smithy-client/dist-cjs/index.js:840:20)
    at /Users/gcorey/src/awsbug/node_modules/@smithy/smithy-client/dist-cjs/index.js:849:5
    at de_CommandError (/Users/gcorey/src/awsbug/node_modules/@aws-sdk/client-secrets-manager/dist-cjs/index.js:992:14)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /Users/gcorey/src/awsbug/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
    at async /Users/gcorey/src/awsbug/node_modules/@smithy/core/dist-cjs/index.js:165:18
    at async /Users/gcorey/src/awsbug/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
    at async /Users/gcorey/src/awsbug/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:22
    at async getApplicationSecrets (/Users/gcorey/src/awsbug/index.js:21:20) {
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 400,
    requestId: '38866c6c-e16e-4dbf-bbf5-1a6de4fc4fc0',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  __type: 'UnrecognizedClientException'
}

geoffcorey commented 1 month ago

Any insights?

geoffcorey commented 3 weeks ago

Updates?

aBurmeseDev commented 3 weeks ago

Hi @geoffcorey - apologies for the delay.

It is peculiar that the issue manifests itself exclusively in version 3.623.0. I recently attempted to replicate the error using the same version, but I was unable to reproduce it. Have you had the opportunity to explore the SDK credential providers? If leveraging those providers resolves the problem, we may be able to pinpoint the underlying cause more effectively.

Let’s say you were able to successfully configure and login SSO with CLI, you can leverage these credentials in your JavaScript SDK code. Here's an example using credential providers: can use these credential providers in your JavaScript SDK code. See here.

I'd also like to look at request logs, you can retrieve those by adding middleware stack to your client calls. Please share the logs from both versions.

const client = new DynamoDBClient({ region: "us-west-2" });

client.middlewareStack.add(
  (next, context) => async (args) => {
    console.log("AWS SDK context", context.clientName, context.commandName);
    console.log("AWS SDK request input", args.input);
    const result = await next(args);
    console.log("AWS SDK request output:", result.output);
    return result;
  },
  {
    name: "MyMiddleware",
    step: "build",
    override: true,
  }
);

await client.listTables({});

Repro attempt

configure sso

$ aws configure sso
[profile my-dev-profile]
sso_session = my-sso
sso_account_id = 123456789011
sso_role_name = readOnly
region = us-west-2
output = json

sso login with --profile

$ aws sso login --profile my-dev-profile
SSO authorization page has automatically been opened in your default browser. 
Follow the instructions in the browser to complete this authorization request.
Successfully logged into Start URL: https://my-sso-portal.awsapps.com/start

SDK Code


import { SecretsManagerClient, GetSecretValueCommand } from "@aws-sdk/client-secrets-manager"; 
import { fromSSO } from "@aws-sdk/credential-providers";

geoffcorey commented 3 weeks ago

client.listTables isn't a valid call. Cut-n-paste error? Anyway, here is the two runs. Had someone on the infrastructure team search cloudtrail and did not see that request id in the logs.


~/src/awsbug is 📦 v1.0.0 via  v20.11.1 on ☁️   (us-west-2)
❯ cat index.js
const { SecretsManagerClient, GetSecretValueCommand } = require('@aws-sdk/client-secrets-manager');
const config = {
  aws: {
    region: 'us-west-2',
    secretName: 'mykey'
  }
};

const getApplicationSecrets = async () => {
  let client;

  client = new SecretsManagerClient({
    region: config.aws.region,
    logger: null
  });
  client.middlewareStack.add(
    (next, context) => async (args) => {
      console.log("AWS SDK context", context.clientName, context.commandName);
      console.log("AWS SDK request input", args.input);
      const result = await next(args);
      console.log("AWS SDK request output:", result.output);
      return result;
    },
    {
      name: "MyMiddleware",
      step: "build",
      override: true,
    }
  )
  const command = new GetSecretValueCommand({
    SecretId: config.aws.secretName
  });

  //await client.listTables({});
  const response = await client.send(command);
  return response;
}

getApplicationSecrets().then(r => { console.log('success') }).catch(console.err);

~/src/awsbug is 📦 v1.0.0 via  v20.11.1 on ☁️   (us-west-2)
❯ npm i && node index.js

added 78 packages, and audited 79 packages in 2s

2 packages are looking for funding
  run `npm fund` for details

3 high severity vulnerabilities

To address all issues, run:
  npm audit fix --force

Run `npm audit` for details.
AWS SDK context SecretsManagerClient GetSecretValueCommand
AWS SDK request input { SecretId: 'PA-Dev-key' }
AWS SDK request output: {
  '$metadata': {
    httpStatusCode: 200,
    requestId: '26e91637-e8f2-4da4-bbc7-7fada6b9c934',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  ARN: 'arn:aws:secretsmanager:us-west-2:169560604650:secret:PA-Dev-key-LefBMG',
  CreatedDate: 2024-07-18T16:21:10.776Z,
  Name: 'PA-Dev-key',
  SecretString: `{<REDACTED>}`,
  VersionId: '41bf8a9c-788a-4995-80f7-256377639827',
  VersionStages: [ 'AWSCURRENT' ]
}
success

~/src/awsbug is 📦 v1.0.0 via  v20.11.1 on ☁️   (us-west-2) took 3s
❯ vi package.json

~/src/awsbug is 📦 v1.0.0 via  v20.11.1 on ☁️   (us-west-2) took 10s
❯ rm package-lock.json && rm -fr node_modules && npm i && cat package.json && node index.js

added 77 packages, and audited 78 packages in 2s

2 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
{
  "name": "awsbug",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "@aws-sdk/client-secrets-manager": "3.632.0"
  }
}
AWS SDK context SecretsManagerClient GetSecretValueCommand
AWS SDK request input { SecretId: 'PA-Dev-key' }
/Users/gcorey/src/awsbug/node_modules/@smithy/smithy-client/dist-cjs/index.js:842
  const response = new exceptionCtor({
                   ^

UnrecognizedClientException: The security token included in the request is invalid.
    at throwDefaultError (/Users/gcorey/src/awsbug/node_modules/@smithy/smithy-client/dist-cjs/index.js:842:20)
    at /Users/gcorey/src/awsbug/node_modules/@smithy/smithy-client/dist-cjs/index.js:851:5
    at de_CommandError (/Users/gcorey/src/awsbug/node_modules/@aws-sdk/client-secrets-manager/dist-cjs/index.js:992:14)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /Users/gcorey/src/awsbug/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
    at async /Users/gcorey/src/awsbug/node_modules/@smithy/core/dist-cjs/index.js:165:18
    at async /Users/gcorey/src/awsbug/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
    at async /Users/gcorey/src/awsbug/index.js:20:22
    at async /Users/gcorey/src/awsbug/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:22
    at async getApplicationSecrets (/Users/gcorey/src/awsbug/index.js:35:20) {
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 400,
    requestId: '9a6cc38d-f322-4eac-96e5-2eb7207de45a',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  __type: 'UnrecognizedClientException'
}

Node.js v20.11.1

~/src/awsbug is 📦 v1.0.0 via  v20.11.1 on ☁️   (us-west-2) took 2s

geoffcorey commented 3 weeks ago

Also did a new aws configure and did a new profile. SSO login successful --profile with the new profile. Code still reports the same thing as above.


Successfully logged into Start URL: https://d-9a6716c0d7.awsapps.com/start#/

~/src/awsbug is 📦 v1.0.0 via  v20.11.1 on ☁️   (us-west-2) took 6s
❯ node index.js
AWS SDK context SecretsManagerClient GetSecretValueCommand
AWS SDK request input { SecretId: 'PA-Dev-key' }
/Users/gcorey/src/awsbug/node_modules/@smithy/smithy-client/dist-cjs/index.js:842
  const response = new exceptionCtor({
                   ^

UnrecognizedClientException: The security token included in the request is invalid.
    at throwDefaultError (/Users/gcorey/src/awsbug/node_modules/@smithy/smithy-client/dist-cjs/index.js:842:20)
    at /Users/gcorey/src/awsbug/node_modules/@smithy/smithy-client/dist-cjs/index.js:851:5
    at de_CommandError (/Users/gcorey/src/awsbug/node_modules/@aws-sdk/client-secrets-manager/dist-cjs/index.js:992:14)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /Users/gcorey/src/awsbug/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
    at async /Users/gcorey/src/awsbug/node_modules/@smithy/core/dist-cjs/index.js:165:18
    at async /Users/gcorey/src/awsbug/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38
    at async /Users/gcorey/src/awsbug/index.js:20:22
    at async /Users/gcorey/src/awsbug/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:22
    at async getApplicationSecrets (/Users/gcorey/src/awsbug/index.js:35:20) {
  '$fault': 'client',
  '$metadata': {
    httpStatusCode: 400,
    requestId: 'ca8cf1cd-fafb-408f-b671-f1112419a838',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  __type: 'UnrecognizedClientException'
}

Node.js v20.11.1

~/src/awsbug is 📦 v1.0.0 via  v20.11.1 on ☁️   (us-west-2)

geoffcorey commented 2 weeks ago

@aBurmeseDev ok, so apparently if I am going to use SSO I have to pass it explicitly in the new version. In the 3.441.0 version it would figure it out. So doesn't seem functionally backwards compatible. Here is the work around.

const { SecretsManagerClient, GetSecretValueCommand } = require('@aws-sdk/client-secrets-manager');

const { fromSSO } = require('@aws-sdk/credential-provider-sso');

const config = {
  aws: {
    region: 'us-west-2',
    secretName: 'my-key'
  }
};

const getApplicationSecrets = async () => {
  let client;

  client = new SecretsManagerClient({
    region: config.aws.region,
    credentials: fromSSO(),
    logger: null
  });
  client.middlewareStack.add(
    (next, context) => async (args) => {
      console.log("AWS SDK context", context.clientName, context.commandName);
      console.log("AWS SDK request input", args.input);
      const result = await next(args);
      console.log("AWS SDK request output:", result.output);
      return result;
    },
    {
      name: "MyMiddleware",
      step: "build",
      override: true,
    }
  )
  const command = new GetSecretValueCommand({
    SecretId: config.aws.secretName
  });

  //await client.listTables({});
  const response = await client.send(command);
  return response;
}

getApplicationSecrets().then(r => { console.log('success') }).catch(console.err);

geoffcorey commented 2 weeks ago

Any thoughts on why it isn't backward compatible?

geoffcorey commented 2 weeks ago

Additional notes. Installed a brand new machine with awscli via brew and this still fails without the fromSSO()

geoffcorey commented 4 days ago

No updates?

aws / aws-sdk-js-v3