"Globalize" the concurrency management of the transfer manager such that all uploads and downloads respect the configured concurrency.
Introduce a Scheduler that is part of the client handle that is threaded throughout each operation already.
Replace the tower::limit::ConcurrencyLimit middleware with an equivalent based on our new scheduler. The tower middleware was based on a simple semaphore. For now we've just moved the semaphore into the scheduler. In time I expect we'll drop the semaphore for something that allows priority based scheduling, perhaps coupled with acquire(...) taking a description/metadata of the work to acquire a permit for.
fix: make object discovery coordinate work with the scheduler
I also upped the default concurrency. I think 8 is going to be way too low. I chose 128 based on roughly reaching 10 Gbps throughput. Assuming 90 MB/request throughput for S3 that gives us around 11.5 Gbps. We could make this more exact of course but a power of 2 felt right.
Question: Uploads and downloads are both going through this scheduler. Do we (eventually) want to split the concurrency such that uploads and downloads are independently scheduled/limited?
I did a quick check on c5n.18xlarge download test to verify there are no regressions. If there are it's in the noise.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Issue #, if available:
https://github.com/awslabs/aws-s3-transfer-manager-rs/issues/31
Description of changes:
"Globalize" the concurrency management of the transfer manager such that all uploads and downloads respect the configured concurrency.
Schedulerthat is part of the client handle that is threaded throughout each operation already.tower::limit::ConcurrencyLimitmiddleware with an equivalent based on our new scheduler. The tower middleware was based on a simple semaphore. For now we've just moved the semaphore into the scheduler. In time I expect we'll drop the semaphore for something that allows priority based scheduling, perhaps coupled withacquire(...)taking a description/metadata of the work to acquire a permit for.I did a quick check on
c5n.18xlargedownload test to verify there are no regressions. If there are it's in the noise.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.