S3 copy file to itself should work

[mdavis-xyz]

I want to copy a file, where the source and destination are the same.

I think S3 used to disallow this (unless you changed the metadata), but now they allow it. But boto hasn't caught up.

MWE

import boto3
import urllib.parse
import os

from moto import mock_s3

bucket_name = 'mybucket'
key = 'test/selfcopy'

mock_s3().start()

s3 = boto3.resource('s3')
bucket = s3.Bucket(bucket_name)
try:
    bucket.create(
        CreateBucketConfiguration={
            'LocationConstraint': 'ap-southeast-2'
        }
    )
except boto3.client('s3').exceptions.BucketAlreadyOwnedByYou:
    pass

data = b"123"

obj = bucket.Object(key)

obj.put(Body=data, Metadata={
        'foo': 'bar'
    },
)

obj.copy_from(
    CopySource={
        'Bucket': obj.bucket_name, 
        'Key': obj.key, 
    },
)

Run this script.

Now comment out mock_s3().start() and run again to test against real S3.

Expected behavior

If mock_s3().start() is commented out and the script is run against real S3, the script throws no exceptions. If the bucket is versioned, you can see two versions of the file.

Actual behavior

Traceback (most recent call last):
  File "main.py", line 32, in <module>
    obj.copy_from(
  File "/home/ec2-user/.pyenv/versions/3.8.11/lib/python3.8/site-packages/boto3/resources/factory.py", line 580, in do_action
    response = action(self, *args, **kwargs)
  File "/home/ec2-user/.pyenv/versions/3.8.11/lib/python3.8/site-packages/boto3/resources/action.py", line 88, in __call__
    response = getattr(parent.meta.client, operation_name)(*args, **params)
  File "/home/ec2-user/.pyenv/versions/3.8.11/lib/python3.8/site-packages/botocore/client.py", line 514, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/ec2-user/.pyenv/versions/3.8.11/lib/python3.8/site-packages/botocore/client.py", line 938, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidRequest) when calling the CopyObject operation: This copy request is illegal because it is trying to copy an object to itself without changing the object's metadata, storage class, website redirect location or encryption attributes.

Details

We should change this test:

https://github.com/spulec/moto/blob/1a8ddc0f2b89672c4bea81a257fa2ec86e0bc789/tests/test_s3/test_s3_copyobject.py#L153-L161

So that it checks there is no exception, instead of checking there is an exception.

I think we can delete this class:

https://github.com/spulec/moto/blob/ba4104c38e47f408e9f6bd356f491d9c9bc4b25d/moto/s3/exceptions.py#L574-L581

Then delete this if statement (and the import at the top of that file):

https://github.com/spulec/moto/blob/0588db704a08f19a00f8cb908d3cc579a65b7dea/moto/s3/models.py#L2177-L2187

[bblommers]

Hi @mdavis-xyz, I can't reproduce this I'm afraid. Both versioned and non-versioned buckets still give the exception for me.

How did you create the original bucket? Looking at the example you gave, it looks like this was created earlier, and I wonder if there's some specific configuration that makes this behaviour possible.

[mdavis-xyz]

Yes my bucket is a few years old. It has versioning enabled.

I have just tested on a new bucket, both versioned and unversioned. I get the error.

So this error happens for some buckets, not others. I have raised a support ticket with AWS for more info.

By the way, I'm using:

boto3 1.24.82 botocore 1.27.82 moto 3.1.16

[mdavis-xyz]

I received an answer from AWS Support.

The difference between my new and old buckets is that my new ones did not have default encryption enabled.

If a bucket has default encryption enabled, e.g. with:

client.put_bucket_encryption(
            Bucket=bucket_name,
            ServerSideEncryptionConfiguration={
                'Rules': [
                    {
                        'ApplyServerSideEncryptionByDefault': {
                            'SSEAlgorithm': 'AES256'
                        },
                        'BucketKeyEnabled': False
                    },
                ]
            },
        )

Then boto will automatically include some encryption-related headers in the API call.

Even though we're not changing the encryption settings of the object, for this particular check, S3 only looks for the presence of encryption headers, and assumes that we're changing encryption, so let's the copy through.

I've checked, and moto does accurately reflect this behavior. i.e. moto allows a self-copy for buckets with default-encryption enabled.

[bblommers]

The more you know! Thanks for letting us know @mdavis-xyz