The primary change in this PR is to refactor download internals from a fixed worker pool size to use tower instead. The motivation for using tower is to be able to build higher level abstractions that would have been difficult with the simple channel/worker approach. Additionally I cleaned some things up and added some new tests. I benchmarked this on a c5n.18xlarge and saw no regression in download times for a single 30 GB object downloaded to RAM. The peak throughput was actually consistently a bit higher (closer to 77-78 Gbps vs the previous ~72 Gbps).
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Issue #, if available: n/a
Description of changes:
The primary change in this PR is to refactor download internals from a fixed worker pool size to use tower instead. The motivation for using tower is to be able to build higher level abstractions that would have been difficult with the simple channel/worker approach. Additionally I cleaned some things up and added some new tests. I benchmarked this on a c5n.18xlarge and saw no regression in download times for a single 30 GB object downloaded to RAM. The peak throughput was actually consistently a bit higher (closer to 77-78 Gbps vs the previous ~72 Gbps).
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.