Add async write() function - fixes "stalled" uploads deadlocking S3 Client

[graebm]

Issue: The S3 client can deadlock if too many uploads are "stalled" (not providing data when the S3 client asks for it).

Mountpoint (which wraps aws-c-s3 with a filesystem-like API) had a user that opened 100+ files at once. The user wrote data to some of the later files they opened, and waited for those writes to complete. But aws-c-s3 was waiting on data from the first few files. Both sides were waiting on each other. It was a deadlock.

Description of changes:

• Add async aws_s3_meta_request_write() function.
  • Mountpoint should use this, instead of the async-streams. Async-streams were still "pull based", where the S3 client decided when data should be provided. The new write() function is push-based. Mountpoint can send data whenever. "Stalled" uploads will not deadlock the S3 client.
• Revert https://github.com/awslabs/aws-c-s3/pull/417
  • This workaround didn't help all that much. The S3 client still deadlocked at the same value of "N stalled uploads" if your workloads also mixed in other types of non-upload meta requests.

Design: The memory-ownership for this API evolved over time. Here were the stages: 1) At first, the user was required to keep memory valid until the async-write completed. (as of commit 5614546)

• PRO: No locks held while memcpy'ing data into a part. Good memory usage, only copy the data when it's ready to send.
• CON: no way for user to drop memory without canceling the meta-request and blocking their thread until the write completed, which risks deadlock if blocking on a thread the meta-request needs to use. 2) Next, we had write() copy all memory immediately (as of commit 5e4dd89)
• PRO: user does not need to keep memory alive after write() call
• CON: a huge write immediately forces a huge allocation. Writing 10GiB means at least 20GiB total memory usage during the write() call. Users would end up needing to chunk up large work themselves, rather than let aws-c-s3 handle it. 3) Finally, go back to approach 1, but guarantee that canceling a meta-request will immediately complete any pending writes.
• PRO: Back to good memory usage
• CON: Need to hold a lock when memcpy'ing data into a part.

Some quick benchmarking showed similar performance from all 3 approaches (tested 30GiB upload, and simultaneous upload of 100 5GiB objects, calling write in 256KiB chunks), so 🤷‍♀️. Going with approach #3 since it doesn't risk deadlock, and doesn't risk enormous memory usage.

TODO:

• Improve buffering of small writes (use buffer pool?)
• Get async-write working with Content-Length (currently only works when Content-Length omitted)
• Get async-write working with DEFAULT meta requests (currently only works with PUT)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 88.76404% with 10 lines in your changes are missing coverage. Please review.

Project coverage is 89.52%. Comparing base (0ce756e) to head (04457df).

Additional details and impacted files

[](https://app.codecov.io/gh/awslabs/aws-c-s3/pull/418?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs) ```diff @@ Coverage Diff @@ ## main #418 +/- ## ========================================== + Coverage 89.40% 89.52% +0.12% ========================================== Files 20 20 Lines 5872 5949 +77 ========================================== + Hits 5250 5326 +76 - Misses 622 623 +1 ``` | [Files](https://app.codecov.io/gh/awslabs/aws-c-s3/pull/418?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs) | Coverage Δ | | |---|---|---| | [source/s3.c](https://app.codecov.io/gh/awslabs/aws-c-s3/pull/418?src=pr&el=tree&filepath=source%2Fs3.c&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs#diff-c291cmNlL3MzLmM=) | `96.00% <ø> (ø)` | | | [source/s3\_auto\_ranged\_put.c](https://app.codecov.io/gh/awslabs/aws-c-s3/pull/418?src=pr&el=tree&filepath=source%2Fs3_auto_ranged_put.c&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs#diff-c291cmNlL3MzX2F1dG9fcmFuZ2VkX3B1dC5j) | `92.75% <100.00%> (+0.16%)` | :arrow_up: | | [source/s3\_meta\_request.c](https://app.codecov.io/gh/awslabs/aws-c-s3/pull/418?src=pr&el=tree&filepath=source%2Fs3_meta_request.c&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs#diff-c291cmNlL3MzX21ldGFfcmVxdWVzdC5j) | `93.14% <94.87%> (+0.89%)` | :arrow_up: | | [source/s3\_client.c](https://app.codecov.io/gh/awslabs/aws-c-s3/pull/418?src=pr&el=tree&filepath=source%2Fs3_client.c&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs#diff-c291cmNlL3MzX2NsaWVudC5j) | `87.69% <33.33%> (-0.48%)` | :arrow_down: |