Retry for slow upload part request

[TingDaoK]

Issue #, if available:

• S3 upload part request has a very low rate (around 0.1%) result in a very high time to first byte latency (couple seconds)

Description of changes:

• Based on the response-first-byte timeout, we can retry for the slow upload part request
• Updated to have metric enabled without the callback set
• A "fairly reasonable" algorithm to dynamic determine the good timeout value to be used.

With the current main branch, the upload 30GB benchmark on c5n.18xlarge:

Run:1 Secs:12.309 Gb/s:20.9 Mb/s:20935.6 GiB/s:2.4 MiB/s:2495.7
Run:2 Secs:10.043 Gb/s:25.7 Mb/s:25660.6 GiB/s:3.0 MiB/s:3059.0
Run:3 Secs:11.212 Gb/s:23.0 Mb/s:22983.5 GiB/s:2.7 MiB/s:2739.8
Run:4 Secs:9.451 Gb/s:27.3 Mb/s:27266.7 GiB/s:3.2 MiB/s:3250.4
Run:5 Secs:9.466 Gb/s:27.2 Mb/s:27224.2 GiB/s:3.2 MiB/s:3245.4
Run:6 Secs:9.290 Gb/s:27.7 Mb/s:27740.1 GiB/s:3.2 MiB/s:3306.9
Run:7 Secs:10.312 Gb/s:25.0 Mb/s:24990.2 GiB/s:2.9 MiB/s:2979.1
Run:8 Secs:11.001 Gb/s:23.4 Mb/s:23424.0 GiB/s:2.7 MiB/s:2792.4
Run:9 Secs:10.130 Gb/s:25.4 Mb/s:25439.2 GiB/s:3.0 MiB/s:3032.6
Run:10 Secs:10.829 Gb/s:23.8 Mb/s:23796.8 GiB/s:2.8 MiB/s:2836.8
Overall stats; Throughput Mean:24768.4 Mb/s Throughput Variance:319537.6 Mb/s Duration Mean:10.404 s Duration Variance:0.806 s Peak RSS:7330.281 Mb

With this branch:

Run:1 Secs:10.595 Gb/s:24.3 Mb/s:24322.1 GiB/s:2.8 MiB/s:2899.4
Run:2 Secs:7.735 Gb/s:33.3 Mb/s:33316.0 GiB/s:3.9 MiB/s:3971.6
Run:3 Secs:7.641 Gb/s:33.7 Mb/s:33726.9 GiB/s:3.9 MiB/s:4020.6
Run:4 Secs:7.901 Gb/s:32.6 Mb/s:32615.2 GiB/s:3.8 MiB/s:3888.0
Run:5 Secs:7.482 Gb/s:34.4 Mb/s:34442.9 GiB/s:4.0 MiB/s:4105.9
Run:6 Secs:7.469 Gb/s:34.5 Mb/s:34503.6 GiB/s:4.0 MiB/s:4113.2
Run:7 Secs:7.812 Gb/s:33.0 Mb/s:32987.7 GiB/s:3.8 MiB/s:3932.4
Run:8 Secs:7.305 Gb/s:35.3 Mb/s:35276.1 GiB/s:4.1 MiB/s:4205.2
Run:9 Secs:7.655 Gb/s:33.7 Mb/s:33665.8 GiB/s:3.9 MiB/s:4013.3
Run:10 Secs:7.561 Gb/s:34.1 Mb/s:34082.8 GiB/s:4.0 MiB/s:4063.0
Overall stats; Throughput Mean:32556.0 Mb/s Throughput Variance:312213.7 Mb/s Duration Mean:7.916 s Duration Variance:0.825 s Peak RSS:7021.082 Mb

The average throughput was improved from 24 Gbps to 32 Gbps.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[codecov-commenter]

Codecov Report

Merging #360 (1690c52) into main (f354c53) will increase coverage by 0.18%. The diff coverage is 98.85%.

Additional details and impacted files

[](https://app.codecov.io/gh/awslabs/aws-c-s3/pull/360?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs) ```diff @@ Coverage Diff @@ ## main #360 +/- ## ========================================== + Coverage 88.26% 88.44% +0.18% ========================================== Files 19 19 Lines 5036 5116 +80 ========================================== + Hits 4445 4525 +80 Misses 591 591 ``` | [Files](https://app.codecov.io/gh/awslabs/aws-c-s3/pull/360?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs) | Coverage Δ | | |---|---|---| | [source/s3\_auto\_ranged\_put.c](https://app.codecov.io/gh/awslabs/aws-c-s3/pull/360?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs#diff-c291cmNlL3MzX2F1dG9fcmFuZ2VkX3B1dC5j) | `92.60% <100.00%> (+0.05%)` | :arrow_up: | | [source/s3\_meta\_request.c](https://app.codecov.io/gh/awslabs/aws-c-s3/pull/360?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs#diff-c291cmNlL3MzX21ldGFfcmVxdWVzdC5j) | `92.92% <100.00%> (+0.08%)` | :arrow_up: | | [source/s3\_request.c](https://app.codecov.io/gh/awslabs/aws-c-s3/pull/360?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs#diff-c291cmNlL3MzX3JlcXVlc3QuYw==) | `95.21% <100.00%> (-0.03%)` | :arrow_down: | | [source/s3\_client.c](https://app.codecov.io/gh/awslabs/aws-c-s3/pull/360?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs#diff-c291cmNlL3MzX2NsaWVudC5j) | `88.99% <98.48%> (+0.73%)` | :arrow_up: | ... and [1 file with indirect coverage changes](https://app.codecov.io/gh/awslabs/aws-c-s3/pull/360/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs)