Improve async write (fewer copies, polling API)

[graebm]

Issue: Mountpoint's upload throughput took a 20% hit when they started using the new async write API (PR https://github.com/awslabs/aws-c-s3/pull/418).

We knew the new API could, in the worst case, do an additional copy (see TODOs in PR description and code.). Some quick experimentation showed this was the cause.

Description of changes:

• Remove additional copy.
  • Async write gets buffers from the buffer-pool (made possible by PR https://github.com/awslabs/aws-c-s3/pull/429)
• Add new poll_write() function, which is simpler to use from Mountpoint.
  • Mountpoint typically does 1MiB or 256KiB writes. So we pretty much always need to copy the data immediately. So let's optimize for that.
  • Rust needed tricky code to cope with the original write() API's demand that data stay alive until the write-future completes. And aws-c-s3 needed tricky code to guarantee cancel() would synchronously fire any pending write-futures. If we just offer a rust-polling-style API that always copies synchronously, we can simplify a lot of code. So let's do that.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 89.61%. Comparing base (3647b4b) to head (fed0631).

Additional details and impacted files

[](https://app.codecov.io/gh/awslabs/aws-c-s3/pull/430?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs) ```diff @@ Coverage Diff @@ ## main #430 +/- ## ========================================== + Coverage 89.54% 89.61% +0.06% ========================================== Files 20 20 Lines 6036 6045 +9 ========================================== + Hits 5405 5417 +12 + Misses 631 628 -3 ``` | [Files](https://app.codecov.io/gh/awslabs/aws-c-s3/pull/430?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs) | Coverage Δ | | |---|---|---| | [source/s3\_auto\_ranged\_put.c](https://app.codecov.io/gh/awslabs/aws-c-s3/pull/430?src=pr&el=tree&filepath=source%2Fs3_auto_ranged_put.c&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs#diff-c291cmNlL3MzX2F1dG9fcmFuZ2VkX3B1dC5j) | `92.80% <100.00%> (+0.03%)` | :arrow_up: | | [source/s3\_meta\_request.c](https://app.codecov.io/gh/awslabs/aws-c-s3/pull/430?src=pr&el=tree&filepath=source%2Fs3_meta_request.c&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=awslabs#diff-c291cmNlL3MzX21ldGFfcmVxdWVzdC5j) | `93.47% <100.00%> (+0.33%)` | :arrow_up: |