Getting FailedToFetch error for DynamoDBClient QueryCommand

[renchix]

Checkboxes for prior research

• [X] I've gone through Developer Guide and API reference
• [X] I've checked AWS Forums and StackOverflow.
• [X] I've searched for previous similar issues and didn't find any solution.

Describe the bug

Fetching data from DynamoDB through QueryCommand ocasionally results in FailedToFetch error. Error persists temporarily and comes and goes periodically. First cases were observed in Oct 31, 2024 and Nov 1, 2024. And again today.

Regression Issue

• [x] Select this option if this issue appears to be a regression.

SDK version number

@aws-sdk/client-dynamodb@3.682

Which JavaScript Runtime is this issue in?

Browser

Details of the browser/Node.js/ReactNative version

Microsoft Edge

Reproduction Steps

        const command = new QueryCommand(input);
        response = await client.send(command);

Observed Behavior

Code has been working for at least a year, no changes have been made recently. First idea was that it some how related to CORS, but I am not sure if this can somehow be configured for DynamoDBClient.

Expected Behavior

DyanmoDb results should be fetched without raising error.

Possible Solution

No response

Additional Information/Context

No response

[renchix]

It is CORS issue, but the question is how comes it hasn't been before and how can it be resolved:

Access to fetch at 'https://dynamodb.eu-north-1.amazonaws.com/' from origin 'https://XXXX.XXXX.XXX has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

[zshzbh]

Hey @renchix ,

There's some potential root causes:

1. Network condition : Intermittent network issues or latency spikes could cause requests to timeout or fail, which might trigger CORS errors in some browsers.
2. AWS Service Availability: While rare, there could be intermittent issues with AWS services in your region if the DDB service or its associated auth services have brief outages or performance issues.
3. Browser Caching: Browsers can cache CORS preflight responses. If a successful request is cached, subsequent requests might work, while new requests or requests after the cache expires might fail. ref: https://repost.aws/questions/QUt1qUofacSDKawoShV1fgKA/api-gateway-method-cors-error-cloudformation. And there's a stack overflow topic regarding this issue. To Implement Retry Logic: Add a retry mechanism for failed requests. This can help mitigate intermittent issues.
4. DNS Resolution
5. Load Balancing: If AWS is using multiple servers or load balancers for the DynamoDB service, some might be configured correctly for CORS while others are not.

To identify the patterns in the failures, you can implement detailed logging to include the timestamps and full request details and response headers.

To help potentially resolve this issue, you can apply retry logic:

const MAX_RETRIES = 3;
const RETRY_DELAY = 1000; 

This can help mitigate intermittent issues.

Thanks! Maggie

[renchix]

Thanks for the detailed response.

To my understanding default in-built retry mechanism () already makes 3 attempts after failed request, so it seems that this doesn't solve the issue as it should already be working.

Caching errors seems to be a possible cause, however, as we have seen cases where even after trying to fully refresh page for several times error still persists, then this also doesn't feel as the likely cause.

I could hope that this ir really some temporary issue - the fact that 99.9% of time everything works correctly also kind of supports this logic. However, I am still puzzled about the fact that in the last week there have been multiple such cases, without anything similar before.

Also, in those cases where DynamoDBClient keeps failing Lambda Client works without issues (I understand that there the logic is a bit different - but still it goes under aws-sdk which then makes causes #1 and #4 as less likely) - this also make me feel that this is some issue specific to DynamoDBClient.

[zshzbh]

Thanks for the updates! I agree, and also it's hard to troubleshoot to find the root cause as it works well in 99.9% of time. I have asked Dynamo DB team and see if there's any known issue happened last week in your region.

[renchix]

Still getting the same error every day or two:

monitor:1 Access to fetch at 'https://dynamodb.eu-north-1.amazonaws.com/' from origin 'https://www.xxxxxxx.com' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled. dynamodb.eu-north-1.amazonaws.com/:1

    Failed to load resource: net::ERR_FAILED

main.535af7c7.js:2

    Uncaught (in promise) TypeError: Failed to fetch
at Vv.handle (main.535af7c7.js:2:1408941)
at async main.535af7c7.js:2:1317124
at async clientCommand.middlewareStack.addRelativeTo.name (main.535af7c7.js:2:1427841)
at async main.535af7c7.js:2:1318306
at async main.535af7c7.js:2:1362663
at async main.535af7c7.js:2:1314299
at async R (main.535af7c7.js:2:3258575)

Is there some way to force those headers for DynamoDBClient request in our React code?

[renchix]

To me it seems that only solution then is to do all requests through API gateway (not directly to DynamoDB as it is now) That obviously would mean that there is one extra intermediary step involved as well as additional costs for API gateway use, and a lot of code rewriting. This doesn't sound good.

[renchix]

To add maybe a little bit more context for DynamoDB team - even if QueryItems fails and then problem persists for some time - at the same time GetItem requests are fulfilled.

[renchix]

After some additional investigation I found that preflight request is made and received successfully, however the real request in those failed cases is missing 'Remote address' in the Headers. This is how failed one looks: and this is successful request:

Preflight requests for both of them seem to be identical.

@zshzbh any suggestions?

[zshzbh]

Hey @renchix ,

I can't reproduce this error. Could you please provide any pre-condition and minimal code reproduction so we can better troubleshoot this issue?

[renchix]

There is not much of code to provide. This basically is all that leads to the error:

const client = new DynamoDBClient({region: region, credentials: credentials, maxRetries: 5 });
const command = new QueryCommand(input);
const response = await DynamoDBDocumentClient.from(client).send(command)

As error usually comes by surprise, I cannot reliably reproduce it as well. When it has happened I have inspected the http requests and missing Remote address field in the Headers, mentioned in my previous post was only thing that seemed out of normal in all of these cases.

[zshzbh]

Hey @renchix,

We are not able to troubleshoot the exact root cause with the existing information. I'd like to suggest -

1. Try to use other browsers, for example Firefox , and see if this also happens on Firefox, or if it's a specific issue on the browser you use.
2. monitor if there’s anything changes when you get the error.
3. try to make the request with lambda.
4. use error handling methods

Use middlewareStack to check the request header and adjust the request - ref.This would be helpful.

[felixschwamm]

I'm also trying to resolve this issue for about 3 weeks now. It also started around Oct 31, and it happens very rarely, but once it happens all requests to DynamoDB fail, and it takes some time (usually a few minutes) until the requests can succeed again.

This image shows the response headers of the failed request on the left side and the response headers of the successful request on the right side. As you can see, the failed request on the left does not include the Access-Control headers in the response and this is also what causes the browser to throw a CORS error.

Another thing: the failed requests always have the following capitalization for the request id header "x-amzn-requestid" and the successful ones "x-amzn-RequestId". I know that the HTTP/1.1 specification states that capitalization should not be relevant for request handling, but this difference suggests that even if both requests have a 200 status (even the failed ones always have a 200 status) and the response is for the most part identical the code that generates this response must be somehow a different one because it sends a slightly different request id header back.

Can you also observe this behavior @renchix ?

Hint: Some browsers have dev tools that mess with the capitalization of the headers to make it more consistent when they display it. In that case you would have to download the HAR file and inspect that, or in chrome there is a checkbox with something like "raw data" that you need to check.

[felixschwamm]

The AWS Console is also affected

[MainzerKaiser]

I can confirm that we struggle with the very same problems for a couple weeks now. Sporadic "TypeError: Failed to fetch" and

"Access to fetch at 'https://dynamodb.eu-central-1.amazonaws.com/' from origin 'https://xxxx.xxxx.xxx/ has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled."

Status code is 200 and AWS console confirms CORS problem.

If one user sees the problems, another user has it at the same time.

After a couple minutes everything works smoothly. Anything we can do to improve the situation?

How is this exactly done, looking from the developer side:

If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled

The only thing that I adapted reading this thread was setting the maxAttempts higher:

import { DynamoDB } from "@aws-sdk/client-dynamodb";

const fetchDataDynamoDB_getItem= async(
    dynamoName: string,
    uuid: string, 
    mycred:Mycred,
    ): Promise<GetItemResult> => {

    const dynamoClient = new DynamoDB({ region: 'eu-central-1', credentials: mycred, maxAttempts: 5 }); 

My dependencies are: "dependencies": { "@aws-sdk/client-athena": "^3.379.1", "@aws-sdk/client-dynamodb": "^3.363.0", "@aws-sdk/client-s3": "^3.363.0", "@aws-sdk/lib-dynamodb": "^3.370.0",