"InvalidSignatureException: The request signature we calculated does not match the signature you provided" when calling a command with credentials from the SDK, but the same command and credentials work when called from the CLI

[stanleyb]

Checkboxes for prior research

• [X] I've gone through Developer Guide and API reference
• [X] I've checked AWS Forums and StackOverflow.
• [X] I've searched for previous similar issues and didn't find any solution.

Describe the bug

I have an Angular app written in TypeScript. I am using the developer-authenticated (enhanced) flow and my back-end application performs GetOpenIdTokenForDeveloperIdentity against an Identity Pool to get an identity id and token for an authenticated user. It then passes the Identity Id and Token to the Angular front-end so that the Angular front-end can fetch temporary credentials for the user from the Cognito Identity Pool. I am able to successfully get these credentials back from Cognito.

Unfortunately, when I then try to use these credentials in a Service Request (for example in a DynamoDB Command), I get an exception: InvalidSignatureException: The request signature we calculated does not match the signature you provided.

If I execute the same command from the CLI, using the same temporary credentials that my Angular client fetched, the call works as expected.

SDK version number

@aws-sdk/client-dynamodb@3.515.0

Which JavaScript Runtime is this issue in?

Browser

Details of the browser/Node.js/ReactNative version

Chrome_121.0.0.0

Reproduction Steps

The following is a method illustrating the problem.

import { CognitoIdentityClient } from "@aws-sdk/client-cognito-identity"; import { fromCognitoIdentity } from "@aws-sdk/credential-providers"; import { ListTablesCommand, DynamoDBClient } from "@aws-sdk/client-dynamodb";

async test() {

const cognitoIdentity = new CognitoIdentityClient({
    credentials : fromCognitoIdentity({
        identityId: 'IDENTITY_ID_RETURNED_FROM_MY_DEVELOPER_PROVIDER' //IdentityId returned from calling GetOpenIdTokenForDeveloperIdentityRequest() on my server-side,
        logins: {
        'cognito-identity.amazonaws.com': 'TOKEN_RETURNED_FROM_MY_DEVELOPER_PROVIDER' //Token returned from calling GetOpenIdTokenForDeveloperIdentityRequest() on my server-side
        },
        clientConfig: {region: "eu-west-1"},
    }),
});

const credentials = await cognitoIdentity.config.credentials()
console.log(credentials) //Use these credentials in the CLI to reproduce the issue. These credentials work when calling services from the CLI.

const client = new DynamoDBClient({
    region: 'eu-west-1',
    credentials: credentials,
});

const command = new ListTablesCommand({});

const response = await client.send(command); //Throws exception: InvalidSignatureException: The request signature we calculated does not match the signature you provided.
return response;

}

If I use the same credentials that I get from cognito (and output in the console in the above code) and call the same command in the CLI, it works as expected. The command that works is: aws dynamodb list-tables --region eu-west-1 --profile my_temp_credential_profile (where my_temp_credential_profile is a profile I created from the output credentials just to test with in the CLI)

Observed Behavior

InvalidSignatureException: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

Expected Behavior

I am expecting a response when I execute the command on the service client.

Possible Solution

No response

Additional Information/Context

The only reason I tried this same command in the CLI is because I wanted to verify that my problem doesn't lie in the way that I fetch credentials from the Cognito Identity Pool. And since the credentials work when I call the same command from the CLI it indicates to me that the problem doesn't lie with how I'm fetching the credentials, but rather in how the client calculates the signature.

I have also done a test where I use @aws-sdk/client-location@3.511.0 and I construct a LocationClient and then execute the "GetDevicePositionCommand" command against that client. And it gives the same error as I get on the DynamoDB client. So I think the problem lies in one of the parent classes that all the Clients derive from rather than in the DynamoDB client per se.

The imports to show this on the LocationClient and GetDevicePositionCommand are given below, but I'm not including a code example for that case here for the sake of brevity. (I logged a StackOverflow question for that example a number of days ago that can be viewed here if that helps.) The below import shows the library applicable to that example.

import { LocationClient, GetDevicePositionCommand } from '@aws-sdk/client-location';

[kuhe]

Is there any information on the region of the returned credentials before you feed them to DynamoDBClient? Could they be for the wrong region?

[stanleyb]

The identity id in the credentials are for the correct region: eu-west-1. (Obviously I can't glean much from the Access Key, Secret Access Key and the Session Token as these don't show region). But again, I should mention, that if I use these exact same credentials in the CLI to call the services, it works. So I don't think the problem lies with the credentials. The error happens in the client. I have also double-checked the network request that the sdk makes and the url it is made to is https://dynamodb.eu-west-1.amazonaws.com/ which is also the correct region. (I can't comment about whether the signature calculation in the sdk takes the correct region into consideration or defaults to some other region as I don't have visibility into this. It could indeed be one possible explanation for the bug if the signature calculation takes a different value for one of the fields it is based on than the actual value that is being sent through.)

[RanVaknin]

Hi @stanleyb2 ,

Thanks for the info.

The command that works is: aws dynamodb list-tables --region eu-west-1 --profile my_temp_credential_profile (where my_temp_credential_profile is a profile I created from the output credentials just to test with in the CLI)

That is not really an indication, since running a command from a CLI and from a browser application is not an apples to apples comparison.

A better comparison will be to use the SDK and instruct it to use the profile the same way you did for the CLI:

import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import { fromIni } from "@aws-sdk/credential-provider-ini";

const client = new DynamoDBClient({
  region: "eu-west-1",
  credentials: fromIni({ profile: "my_temp_credential_profile" }),
});

// list tables code goes here

I'm guessing this will work the same way it did with the CLI, indicating that the problem is not the credentials, but rather different headers, extra headers or incorrect request formatting.

I'm not familiar with Angular, but there is a chance that angular / browser you are running this from is tacking on extra headers or changing something about the outgoing request that causes a mismatch. I see this sometimes with proxies, but unsure about Angular.

Here are some self debug steps:

1. Add a middleware log hook and print it to the console to inspect the raw request that goes out:

   
   const client = new DynamoDBClient({})

client.middlewareStack.add(next => async (args) => { console.log(args.request) const response = await next(args); console.log(response); return next(args); }, {step: 'finalizeRequest'})

// rest of your code

2. After you have the request sent by the SDK printed in the console, look at your browser's network tab and inspect the request body and headers. 
Cross reference what is printed by the SDK and what is actually sent by the browser and see if there are any discrepancies. 

3. Comparison with the CLI - you can run the successful CLI command with the `--debug` flag to examine the raw outgoing request: `aws dynamodb list-tables --region eu-west-1 --profile my_temp_credential_profile`. 

In theory all 3 should be the similar (`user-agent` would be different but this is not signed) . My suspicion is that with step 2, you might see something different. Either a value that has changed, or an extra header that didn't exist in the CLI.
This can definitely lead to signature mismatch errors. 

Let us know what you find.
Thanks,
Ran~

[stanleyb]

I have done the comparison. The CLI (which is working) is certainly using far fewer headers in the request than does the SDK and the browser. It is also clear that the browser/network request includes several additional headers to what the SDK does. But all the SignedHeaders listed in the authorization field and their corresponding header values match in the SDK request and network request (except for the user-agent which differs). I do notice that the sdk generates the "host", "content-type" and "content-length" headers in lowercase whereas the browser sends it through as "Host", "Content-Type" and "Content-Length" (but that conforms to ISO standards). Is the signature calculation perhaps breaking based on case-sensitivity? (That's one idea). I can honestly see nothing obviously wrong in the headers.

Below are my headers from the SDK output:

"content-type": "application/x-amz-json-1.0",
"x-amz-target": "DynamoDB_20120810.ListTables",
"content-length": "2",
"host": "dynamodb.eu-west-1.amazonaws.com",
"x-amz-user-agent": "aws-sdk-js/3.521.0 ua/2.0 os/macOS#10.15.7 lang/js md/browser#Chrome_122.0.0.0 api/dynamodb#3.521.0",
"amz-sdk-invocation-id": "99cefd79-3698-4851-83b1-a5230b70770b",
"amz-sdk-request": "attempt=1; max=3",
"x-amz-date": "20240226T093306.631+0000",
"x-amz-security-token": "XXXX",
"x-amz-content-sha256": "xxx",
"authorization": "AWS4-HMAC-SHA256 Credential=ASIAxxx/20240226/eu-west-1/dynamodb/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-security-token;x-amz-target;x-amz-user-agent, Signature=xxx"

And below are my headers from the browser network request:

POST / HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: en-GB,en-US;q=0.9,en;q=0.8
Connection: keep-alive
Content-Length: 2
Host: dynamodb.eu-west-1.amazonaws.com
Origin: http://localhost:4200
Referer: http://localhost:4200/
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: cross-site
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36
amz-sdk-invocation-id: 99cefd79-3698-4851-83b1-a5230b70770b
amz-sdk-request: attempt=1; max=3
authorization: AWS4-HMAC-SHA256 Credential=ASIAXXXX/20240226/eu-west-1/dynamodb/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-security-token;x-amz-target;x-amz-user-agent, Signature=xxx
content-type: application/x-amz-json-1.0
sec-ch-ua: "Chromium";v="122", "Not(A:Brand";v="24", "Google Chrome";v="122"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "macOS"
x-amz-content-sha256: xxx
x-amz-date: 20240226T093306.631+0000
x-amz-security-token: XXXX
x-amz-target: DynamoDB_20120810.ListTables
x-amz-user-agent: aws-sdk-js/3.521.0 ua/2.0 os/macOS#10.15.7 lang/js md/browser#Chrome_122.0.0.0 api/dynamodb#3.521.0

And below are the headers from the CLI request:

"X-Amz-Target":"DynamoDB_20120810.ListTables",
"Content-Type":"application/x-amz-json-1.0",
"User-Agent":"aws-cli/2.14.2 Python/3.11.6 Darwin/23.3.0 exe/x86_64 prompt/off command/dynamodb.list-tables",
"X-Amz-Date":"20240226T092522Z",
"X-Amz-Security-Token":"XXXX",
"Authorization":"AWS4-HMAC-SHA256 Credential=ASIAXXXX/20240226/eu-west-1/dynamodb/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-security-token;x-amz-target, Signature=xxx",
"Content-Length":"2"

In all three cases I used the same credentials. Yet the only request that works is the CLI one (where the signature is calculated based on far fewer headers). (For brevity and security I have replaced the actual values of sensitive fields with XXXX and xxxx in the above).

[RanVaknin]

Hi @stanleyb2 ,

Thanks for the additional context. upon re-examining your code snippet, the method in which you are obtaining credentials is not something I've seen before. Is there a reason you are using the CognitoIdentityClient? In your use case you can use the fromCognitoIdentity credential provider directly.

Can you give this a try?

const client = new DynamoDBClient({
        region: 'eu-west-1',
        credentials: fromCognitoIdentity({
            identityId: 'IDENTITY_ID_RETURNED_FROM_MY_DEVELOPER_PROVIDER' //IdentityId returned from calling GetOpenIdTokenForDeveloperIdentityRequest() on my server-side,
            logins: {
            'cognito-identity.amazonaws.com': 'TOKEN_RETURNED_FROM_MY_DEVELOPER_PROVIDER' //Token returned from calling GetOpenIdTokenForDeveloperIdentityRequest() on my server-side
            },
            clientConfig: {region: "eu-west-1"},
        }),
});

Thanks, Ran~

[stanleyb]

Hi @RanVaknin

I've already tried the exact code you suggest before. But I tried it again now to make sure. I changed my code now to be as you suggest and I get the exact same error.

[deepalig11]

Hi @stanleyb2 I was also facing the same issue in nodejs sdk what worked for me was creating a new client id and secret in aws console and then running. as my old access key id and secret was created for ses it was having this issue

[stanleyb]

Hi @deepalig11 Thank you, but this is not the same issue. We are using the developer-authenticated (enhanced) flow in the client. Not a client id and secret like in your case.

[stanleyb]

Hi @RanVaknin

Do you maybe have any further news on this? This issue is really holding us back in our development and I feel like we've exhausted all options.

[scott-irwin]

@RanVaknin Same error on node.js running on windows, I can submit a redshift-data query but fail to describe the statement on the subsequent call. Windows, and my system clock is 2sec off website time.is. Code works on v2 but not after upgrade to v3.

Signature expired: 20240312T141610Z is now earlier than 20240312T151111Z (20240312T151611Z - 5 min.) 20240312T151611Z - this is the correct UTC time 20240312T141610Z - this is wrong - daylight savings time issue? My windows clock is the correct EST Daylight savings time.

Credentials obtained via .aws/credentials and aws console (logoff, on (Okta), fresh tokens). Also tried using sso obtained creds and the same issue.

Where is it getting this 1 hr off timestamp?

[ryanolson-aumni]

Anything?

[AuxinJeron]

I have seen the same exception when using BedrockRuntimeClient along with fromCognitoIdentityPool

The sample code

let COGNITO_ID = "COGNITO_ID";
let loginData = {
  [COGNITO_ID]: idToken,
};

let credentials = AWS.fromCognitoIdentityPool({
    clientConfig: { region: "us-west-2" },
    identityPoolId: 'IDENTITY_POOL_ID',
    logins: loginData,
});

const client = new AWS.BedrockRuntimeClient({
    region: "us-west-2" ,
    credentials: credentials
});

const params = {...};

const command = new AWS.InvokeModelCommand(params);

client.send(command)

But this exception did not show up when I was using EC2Client along with fromCognitoIdentityPool.

The sample code

let COGNITO_ID = "COGNITO_ID";
let loginData = {
  [COGNITO_ID]: idToken,
};

let credentials = AWS.fromCognitoIdentityPool({
    clientConfig: { region: "us-west-2" },
    identityPoolId: 'IDENTITY_POOL_ID',
    logins: loginData,
});

const client = new AWS.EC2Client({
    region: "us-west-2" ,
    credentials: credentials
});

const input = {
    InstanceIds: ["INSTANCE_ID"],
};
const command = new AWS.DescribeInstancesCommand(input);

client.send(command)

[stanleyb]

Hi @RanVaknin

Any feedback on this?

[RanVaknin]

Hi. everyone,

InvalidSignatureException simply means that the signature calculated for the signedHeaders section in your request headers, did not match the actual signature calculated by the server for these same headers. When you run your application from a web context, your browser / front end environment might be mutating some of the headers after they are signed. It can happen also for clock skew reasons, or really any number of variations between server and client and is not an SDK specific bug, but rather a product of the environment you are running your code from.

@stanleyb2,

I'd like you to do the following thing:

1. share a complete code snippet of your code with the corrected Cognito credential provider so I may run it myself and see if I can gather any other details myself.
2. try to run the same code from a NodeJS environment
3. try to run this from a different browser / disabling any browser extensions you might have.

My best guess is this has nothing to do with the method of credentials, but rather that your browser is acting as a proxy here and is overwriting some of the headers after the signature was calculated therefore it will result in a mismatch.

@scott-irwin, The SDK does not do anything sophisticated with the timestamp, it simply invokes NodeJS's Date.Now() when calculating a signature, in this case it seems like there is a skew between the time the signature was calculated and the time the API call was fired. The fact that this is exactly 1 hour does seem like it has to do with daylight savings. Are you running this from a local NodeJS environment on this machine? A container? A browser? I don't know enough about your setup to identify the reason for the clock skew. If this still persists please open a new issue so we may better help you. You might want to sync your machine with an NTP service.

@AuxinJeron @ryanolson-aumni
Please open separate github issues and fill the intake form. This exception can be raised for many reasons and is very environment dependent.

Thanks, Ran~

[stanleyb]

Hi @RanVaknin Thank you for getting back to me. I followed your advice and tried in a NodeJS environment and it works. I then also started a small clean Angular application to share with you and tested my code there and it also works. (Which makes it difficult to share a small snippet to reproduce the error.) I do notice that in cases where it works, the x-amz-date field follows the ISO 8601 format: YYYYMMDDTHHMMSSZ (eg 20240520T101316Z). And I also notice that in my application's case, where the sdk miscalculates the signature, the x-amz-date field follows the Extended ISO 8601 format (eg. 20240520T144244.124+0000). I notice this consistently in the headers from the CLI too that I mention in my earlier tests. The signing documentation suggests that the format HAS to follow format YYYYMMDDTHHMMSSZ (and without milliseconds). So I am guessing that this is the reason it miscalculates. If I look at the output from the middleware added to the SDK Client (like you suggested in your comment from 23 Feb), then I see that the header value for x-amz-date that the SDK generates is in the extended format already instead of the correct "Z" format. It doesn't only happen when the request gets submitted over the network from the browser. Is there a way to solve this?

[stanleyb]

Hi @RanVaknin

I have resolved the issue. I found that my exact code works on a "vanilla" Angular application and on a "vanilla" NodeJS application. As mentioned in my previous comment, I also saw that the x-amz-date header sent from my application is in extended ISO 8601 format instead of standard ISO 8601 format (like SignatureV4 expects). So that made me pretty sure that it is because of this date that the signature miscalculates.

I then dug through the libraries that our application uses. (It is built on ASP.NET Zero). And I found that both the standard JavaScript .toISOString() methods and .toString() methods are being overridden to use the Extended ISO 8601 format!

So obviously, the AWS SDK libraries expect normal behaviour from calling those methods internally to construct the headers and to sign the request, but it's getting overridden behaviour! And from my end-use perspective, I am calling the SDK libraries (which transparently build up the headers calculates the signature) perfectly like I should. But because the standard function the SDK uses internally is being overridden by some framework the signature calculated internally by the AWS SDK libraries does not behave as it should.

I fixed it by removing that override customization so that the date functions can function like normal! (Because overriding such a standard function will obviously impact on other libraries that one might use in their app as well! It is ridiculous that ASP.NET Zero overrides it in the framework!)

Thank you for your feedback and replies to this! I really appreciate it. But this can be closed. It is not a bug on the JavaScript AWS SDK.

[RanVaknin]

Hey @stanleyb2 ,

I'm really happy to hear that you were able to figure this out 😄 !

All the best, Ran~

[github-actions[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.