InvalidSignatureException: Signature expired

dobrynin commented 3 months ago

Checkboxes for prior research

[X] I've gone through Developer Guide and API reference
[X] I've checked AWS Forums and StackOverflow.
[X] I've searched for previous similar issues and didn't find any solution.

Describe the bug

This issue is just re-opening https://github.com/aws/aws-sdk-js-v3/issues/5192 which was never solved. If you initialize a secrets-manager client in a lambda runtime which gets frozen and then later (after 5 minutes) the lambda is triggered and tries to fetch a secret you will hit the signature expired error. The suggested workarounds in #5192 (such as using top-level await) are untenable as they would require extensive code rework. It would be great to identify why exactly expired signatures are being sent. @trivikr explained the process for making signed requests in https://github.com/aws/aws-sdk-js-v3/issues/5192#issuecomment-1836916445 although it does appear as if the constructor is somehow setting a signature which later expires, as described by @deanc in https://github.com/aws/aws-sdk-js-v3/issues/5192#issuecomment-1986264214

SDK version number

"@aws-sdk/client-secrets-manager": "^3.549.0"

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

Node.js 20.x

Reproduction Steps

export const secretsManager = AWSXRay.captureAWSv3Client(
  new SecretsManagerClient(commonConfiguration),
);

import secretsManager into a lambda runtime, but don't request any secrets. Wait 5 minutes. Invoke the lambda and request a secret.

Observed Behavior

Invalid signature error

Expected Behavior

Properly set signature.

Possible Solution

No response

Additional Information/Context

No response

trivikr commented 3 months ago

As you've read already, we've done extensive analysis in https://github.com/aws/aws-sdk-js-v3/issues/5192 This happens only on AWS Lambda, and we're working with them to add a documentation about it.

This is the messaging from AWS Lambda team I can provide for future reference

As stated in our documentation, for functions using on-demand concurrency, Lambda may occasionally initialize execution environments ahead of invocation requests. When this happens, you may also observe a time gap between your function's initialization and invocation phases. In this case, we understand your functions are creating and using the AWS SDK client during the function INIT phase. This creates asynchronous tasks on the Node.js event loop to both sign and send a request to SSM. However, these tasks may be executed during either the INIT or the INVOKE phase. Where a request signature is created during the INIT phase and sent during the INVOKE phase, the occasional delays between INIT and INVOKE can cause the signatures to expire. This explains the intermittent errors you have been seeing.

To resolve this issue, we recommend using Node.js top-level await to ensure asynchronous tasks initiated in the INIT phase are also completed in the INIT phase. This will remove the delay between signing and sending requests, preventing these signature expired errors.

We recognize that updating your code to use top-level await requires switching from CommonJS Modules to ES Modules, which may take some time. As a workaround you can create SDK client in INVOKE phase, but make API call inside the handler.

From the GitHub Issue:

TLA recommendation: https://github.com/aws/aws-sdk-js-v3/issues/5192#issuecomment-2073243617
API call inside handler workaround: https://github.com/aws/aws-sdk-js-v3/issues/5192#issuecomment-1857037911

dobrynin commented 3 months ago

@trivikr thank you for response, it is very helpful. I think I am missing something about how SSM/Secrets Manager works. One of the solutions you linked is

// ...
const client = new SSM();
let secret;

const handler = async (event) => {
  if (secret === undefined) {
    secret = await client.getSecretValue(params);
  }
  // Use the secret.
}

We do more-or-less exactly this, although we are instantiating a secrets manager client instead of SSM. Is there something specific to secrets manager that causes us to run into the async init problem?

aws / aws-sdk-js-v3