S3TransferError 400 BadRequest with reason RequestTimeout

[GitHK]

• After release 1.38.0 PR https://github.com/ITISFoundation/osparc-simcore/pull/3448 correctly retries in case of a RequestTimeout code when receiving a 400 BadRequest while uploading a file via multipart upload.

Analysing the log, it was observed that the upload was retried a few times, and there are also different RequestId entries. This happened for a period of ~8 minutes.

See traceback for details https://monitoring.osparc.io/graylog/dashboards/636a4c758211d89dbcb69f25

[sanderegg]

see issue frequency by issuing following: https://monitoring.osparc.io/graylog/search?q=%22Your+socket+connection+to+the+server+was+not+read+from+or+written+to+within+the+timeout+period%22&rangetype=relative&from=1209600

[sanderegg]

added more informative logs with https://github.com/ITISFoundation/osparc-simcore/pull/3717

[sanderegg]

https://monitoring.osparc.io/graylog/search?q=message%3A%22Your+socket+connection+to+the+server+was+not+read+from+or+written+to+within+the+timeout+period%22+AND+%22file_size%3D%22&rangetype=relative&from=2419200

This shows the latest happenning of that issue.

[sanderegg]

https://monitoring.osparc.io/graylog/search?q=message%3A%22Your+socket+connection+to+the+server+was+not+read+from+or+written+to+within+the+timeout+period%22+AND+%22file_size%3D%22&rangetype=relative&from=2419200

This shows the occurences.

[sanderegg]

After checking it can be said:

• the failed part is retried 5 times
• there is a 2minute interval
• after the first failure, the next call is done after everything was uploaded (it goes as the last task in the events loop)
• one thing found is a potential removal of the Content-Length header, but it makes the upload go about 5x slower

[GitHK]

@sanderegg I have no longer seen this issue pop up after merging #3737. Should we maybe close this?

[sanderegg]

@GitHK ok that is very good. feel free to close.

[sanderegg]

new occurences happened. This graylog calls show their occurences: https://monitoring.osparc.io/graylog/search?q=%22last_chunk_size%3D%22&rangetype=relative&from=2592000

[sanderegg]

Issues are still occuring, see https://monitoring.osparc.io/graylog/search?q=%22last_chunk_size%3D%22&rangetype=relative&from=2592000

[mrnicegyu11]

Just as an update: This is still happening.

The issue is known to AWS and was reported at least on the git-repo of the aws java sdk. One of the good leads it seems is that the content-length might be slightly off [https://github.com/aws/aws-sdk-js/issues/281#issuecomment-194026499]. People claim that retrying the PUT call fixes this [https://github.com/aws/aws-sdk-js/issues/281#issuecomment-313800925], but in our case it doesn't.

Let me summarize the state of things from where I am sitting:

• There are sudden sub-second network drops, it seems. This is the starting point. A PUT request to a pre-signed link doesnt reach AWS S3. I read through the syslogs at length and diligently, I dont know how to address this.
• The core of the issue: We retry the PUT request to the pre-signed link, but in these cases, the retrying doesnt ever seem to help. All retries fail
• BUT: If we trigger the whole saving, including the generation of pre-signed links etc., it works always.

[GitHK]

Will close this one for now since https://github.com/ITISFoundation/osparc-simcore/pull/4996 was merged I have not seen any issues regarding this. Feel free to reopen it if it pops up again