search

terricain / aioboto3

Wrapper to use boto3 resources with the aiobotocore async backend
Apache License 2.0
769 stars 75 forks source link

S3 Copy (managed transfer) should limit concurrency #349

Open mjpieters opened 3 months ago

mjpieters commented 3 months ago

When copying a large S3 object with s3.object.copy() / s3.client.copy(), aioboto3 will use multiple uploader() tasks to manage the transfer. However, it doesn't put any limit on the number of tasks created, ignoring the TransferConfig.max_request_concurrency value.

This should be easy to fix; just create a asyncio.Semaphore() and pass in the max_request_concurrency value for this transfer, then have the uploader() task start with async with <shared semaphore>: to limit the number of active concurrent tasks.

terricain commented 3 months ago

Ah yep, spot on, will fix at some point soon

terricain commented 1 month ago

This is live in 13.2.0