terricain / aioboto3

Wrapper to use boto3 resources with the aiobotocore async backend
Apache License 2.0
742 stars 76 forks source link

S3 Copy (managed transfer) should limit concurrency #349

Open mjpieters opened 1 month ago

mjpieters commented 1 month ago

When copying a large S3 object with s3.object.copy() / s3.client.copy(), aioboto3 will use multiple uploader() tasks to manage the transfer. However, it doesn't put any limit on the number of tasks created, ignoring the TransferConfig.max_request_concurrency value.

This should be easy to fix; just create a asyncio.Semaphore() and pass in the max_request_concurrency value for this transfer, then have the uploader() task start with async with <shared semaphore>: to limit the number of active concurrent tasks.

terricain commented 1 month ago

Ah yep, spot on, will fix at some point soon