Description of changes:
This is my first Rust PR, so feel free to provide lots of feedback.
Similar to https://github.com/awslabs/aws-s3-transfer-manager-rs/pull/47, this PR refactors the upload_part implementation from a fixed pool of workers to tower. This will allow us to easily implement and test higher-level abstractions like hedging for slow parts. I saw no performance difference with this.
This also refactors the upload pipeline to distinguish between the read_body and upload_part phases. read_body still uses a fixed pool of workers because we need to support unknown content length, and I couldn't figure out how to implement an unknown amount of work in tower. The upload_part phase uses tower's concurrency_limit layer to manage the concurrency.
I used two different JoinSets because they both have different return types, and upload_tasks requires a lock. If I combine them into one JoinSet, we run into a deadlock where we are still trying to read_body and join all the tasks. Both the join function and read_body need a lock to join or read the body and spawn upload_tasks. Please feel free to suggest if there is a better way to accomplish this.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Description of changes: This is my first Rust PR, so feel free to provide lots of feedback.
upload_partimplementation from a fixed pool of workers totower. This will allow us to easily implement and test higher-level abstractions like hedging for slow parts. I saw no performance difference with this.read_bodyandupload_partphases.read_bodystill uses a fixed pool of workers because we need to support unknown content length, and I couldn't figure out how to implement an unknown amount of work intower. Theupload_partphase uses tower'sconcurrency_limitlayer to manage the concurrency.JoinSetsbecause they both have different return types, andupload_tasksrequires a lock. If I combine them into oneJoinSet, we run into a deadlock where we are still trying toread_bodyand join all the tasks. Both thejoinfunction andread_bodyneed a lock to join or read the body and spawnupload_tasks. Please feel free to suggest if there is a better way to accomplish this.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.