Partial fix: Too many "stalled" uploads can deadlock the S3 Client

[graebm]

Issue: Mountpoint (which wraps aws-c-s3 with a filesystem-like API) had a user that opened 100+ files at once. The user wrote data to some of the later files they opened, and waited for those writes to complete. But aws-c-s3 was waiting on data from the first few files. Both sides were waiting on each other. It was a deadlock.

Description of changes:

• Add regression test that demonstrates the issue.
• Add aws_s3_meta_request_options.maximize_async_stream_reads_internal_use_only setting that increases the number of "stalled" uploads the S3 client can tolerate.
  • Instead being limited by maximum connections (25, given default target throughput), it will be limited by memory_limit_in_bytes (240, given 64bit platform with default values for target throughput, memory limit, and part size).
  • This option will be removed when a better solution is developed. Likely it will be some kind of aws_s3_meta_request_write_async() call, which lets the user tell the S3 client when data is available. Vs the current situation with async-stream, where the S3 client is telling the user when it's a good time to provide the data.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 89.35%. Comparing base (ce78b1e) to head (6be7bf6).

:exclamation: Current head 6be7bf6 differs from pull request most recent head b1cd10f. Consider uploading reports for the commit b1cd10f to get more accurate results

Additional details and impacted files

[](https://app.codecov.io/gh/awslabs/aws-c-s3/pull/417?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs) ```diff @@ Coverage Diff @@ ## main #417 +/- ## ========================================== - Coverage 89.37% 89.35% -0.02% ========================================== Files 20 20 Lines 5854 5854 ========================================== - Hits 5232 5231 -1 - Misses 622 623 +1 ``` | [Files](https://app.codecov.io/gh/awslabs/aws-c-s3/pull/417?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs) | Coverage Δ | | |---|---|---| | [source/s3\_client.c](https://app.codecov.io/gh/awslabs/aws-c-s3/pull/417?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs#diff-c291cmNlL3MzX2NsaWVudC5j) | `88.16% <ø> (ø)` | | | [source/s3\_meta\_request.c](https://app.codecov.io/gh/awslabs/aws-c-s3/pull/417?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs#diff-c291cmNlL3MzX21ldGFfcmVxdWVzdC5j) | `92.23% <ø> (ø)` | | ... and [1 file with indirect coverage changes](https://app.codecov.io/gh/awslabs/aws-c-s3/pull/417/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs)

[graebm]

I updated the PR description with some more description of where those numbers came from