Closed jay-herrera closed 2 weeks ago
Hi @jay-herrera A network error along ERR_CONNECTION_RESET
is mostly caused by your application cannot establish a connection to your S3 bucket endpoint.
Could you follow this article to ensure the connectivity from your development environment to your S3 bucket endpoint?
The region where your bucket was created in may also be a factor if the region is far from your location.
In addition could you also provide a sample of the failed request, including the URL, request headers and response (if any), you can find these information from developer tool, network tab.
@HuiSF Hey! I will follow that article and report back:)
@HuiSF Ok, I have done a few things.
I followed that article and noticed that the server connection closed at around 23 seconds every time. Here is my output: Trying 52.216.34.154... Connected to s3-r-w.us-east-1.amazonaws.com. Escape character is '^]'. Connection closed by foreign host. telnet 443 0.01s user 0.01s system 0% cpu 23.163 total
I patched the aws-amplify library to give more detailed errors for the xhr failure. You can find the logs attached as xhr.log. This did not give much further insight, as it appeared that no response is ever returned to the xhr request. The error was a "Progress Event", which indicates that the problem did not occur in the initial sennding of the request.
I set up a man in the middle proxy to analyze the traffic. You can find the result attached as mitm.txt, however, you may want to use mitmproxy to view it, because it's not encoded in a readable format. The result also indicated that the server had closed the connection before the multipart upload was finished.
Hi @jay-herrera
Thanks for getting back at us with the details. I tried to replay with the mitm proxy but I got no luck. The requests are all timed out now. The xhr log is more helpful, but it indicates the browser emitting net::ERR_CONNECTION_RESET error.
There are various reasons can cause this issue and most of them are related to the network connectivity. Here the AWS SDK provides some step to trouble shoot this error as well.
Did you start to see this error after upgrading to v6? Or did you start to see this error after change of network condition?
@AllanZhengYP After some further investigation and much trial and much trial and error, I think I've found a solution that seems to be working very well. I did have to use s3 through the aws-sdk, then I set all of the native retries to only 1 maxAttempt. Then I wrapped everything in this retry wrapper:
export const wrapRetry = async <T>(
fn: (..._any: any[]) => Promise<T>,
maxAttempts: number,
backoffFunction: (attemptNumber: number) => number
) => {
for (let attempt = 0; attempt < maxAttempts; attempt++) {
try {
await new Promise((resolve) => {
return setTimeout(resolve, backoffFunction(attempt))
})
return await fn()
} catch (err: any) {
if (attempt === maxAttempts - 1) {
throw err
}
console.error(err, typeof err, JSON.stringify(err))
if (err.httpStatusCode === 400 || err.httpStatusCode === 403) {
throw err
}
console.info(`Attempt ${attempt + 1} failed. Retrying...`)
}
}
throw new Error('Maximum number of attempts exceeded.')
}
I've learned a few things from this process: -I am uncertain if uploadData implements a retry strategy directly -If it does, the strategy does not recognize "net::ERR_CONNECTION_RESET", "ERR_NO_RESPONSE", or "503 SLOWDOWN" as retryable errors, although experimentally, these errors are likely to succeed on a retry -Using backoff functions for retries has a huge effect on how likely a retry is to succeed
@jay-herrera, thank you for the additional context here. We're investigating the retry handler further and will mark this as a bug.
Hi @jay-herrera We released v6.6.7 which fixes retry is not occurring as expected, which should mitigate the issue you are facing, please upgrade and test again, thanks!
Closing this issue as it should now be resolved per the above comment! If anyone still experiences this on v6.6.7+, please feel free to reply back and let us know any reproduction steps.
Thank you!
Before opening, please confirm:
JavaScript Framework
React
Amplify APIs
Storage
Amplify Version
v6
Amplify Categories
auth, storage
Backend
Amplify Gen 2 (Preview)
Environment information
Describe the bug
Hey, guys! I have been having some issues with random failures when uploading to S3 using uploadData from a web browser. I have created a POC with amplify that is just auth and an S3 bucket. The request does not seem to actually reach the S3 endpoint, and does not return a response. I assume that the error is caused by a timeout because no response is ever received.
I do not see the failed requests in the S3 Access Logs. On safari, I get a "Network Error - Failed to load resource: The network connection was lost" emitted, and chrome emits "ERR_CONNECTION_RESET". In order to rule out cors errors, and permission errors, I gave the broadest possible cors configuration, and gave all S3 actions access to my auth role.
I have also tried just using the S3 client directly. I have also tried using a bucket not attached to Amplify. I have tried different networks, different devices,, and different file types. The error is emmitted from xhr-http-handler, but there is no more detailed information. I am at my wits' end, looking everywhere for at least a thread to pull on, but everything has been fruitless so far.
Expected behavior
S3 Uploads work reliably
Reproduction steps
Code Snippet
Log output
aws-exports.js
No response
Manual configuration
No response
Additional configuration
No response
Mobile Device
No response
Mobile Operating System
No response
Mobile Browser
No response
Mobile Browser Version
No response
Additional information and screenshots
No response