search

terricain / aioboto3

Wrapper to use boto3 resources with the aiobotocore async backend
Apache License 2.0
748 stars 76 forks source link

Can't use s3.object.copy() because it uses copy_object #348

Closed mjpieters closed 2 months ago

mjpieters commented 2 months ago

Description

When accessing S3 over a VPC endpoint, cross-region copies with CopyObject and UploadPartCopy are not available:

AccessDenied: VPC endpoints do not support cross-region requests

See https://repost.aws/knowledge-center/s3-troubleshoot-copy-between-buckets

Unfortunately, aioboto3 recently rewrote how s3.Object.copy() / s3.Client.copy() works, where it'll use CopyObject or UploadPartCopy depending on the object size. boto3 uses managed transfers to stream the data through the client, effectively creating a copy of an object by downloading and then uploading.

As a work-around we can fall back to re-implementing the old GetObject / upload_fileobj path, but that's not ideal. Can we have an option to disable the new version or, even better, have aioboto3 handle the AccessDenied error that's thrown and fall back to streaming the copy?

mjpieters commented 2 months ago

Turns out I misunderstood what boto3's s3.Object.copy() / s3.Client.copy() actually does in this case. The documentation calls it a 'managed transfer' but it uses CopyObject / UploadPartCopy for this as well.