Closed mjpieters closed 2 months ago
Turns out I misunderstood what boto3's s3.Object.copy()
/ s3.Client.copy()
actually does in this case. The documentation calls it a 'managed transfer' but it uses CopyObject
/ UploadPartCopy
for this as well.
Description
When accessing S3 over a VPC endpoint, cross-region copies with
CopyObject
andUploadPartCopy
are not available:See https://repost.aws/knowledge-center/s3-troubleshoot-copy-between-buckets
Unfortunately, aioboto3 recently rewrote how
s3.Object.copy()
/s3.Client.copy()
works, where it'll useCopyObject
orUploadPartCopy
depending on the object size.boto3
uses managed transfers to stream the data through the client, effectively creating a copy of an object by downloading and then uploading.As a work-around we can fall back to re-implementing the old
GetObject
/upload_fileobj
path, but that's not ideal. Can we have an option to disable the new version or, even better, have aioboto3 handle the AccessDenied error that's thrown and fall back to streaming the copy?