Add .copy() and .rename() methods

[emesik]

Hi, I've added these two potentially useful methods. Done this way, the copy operation should be done across S3 servers without doenloading and uploading again.

The tests, however, I haven't ran them, as I don't have a testing environment present ATM.

[etianen]

Thanks for the MR, these methods do indeed look useful!

However, the tests don't pass.

$ coverage run tests/manage.py test tests
System check identified no issues (0 silenced).
E...................E............
======================================================================
ERROR: testCopy (django_s3_storage_test.tests.TestS3Storage)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dave/Workspace/django-s3-storage/django_s3_storage/storage.py", line 36, in _do_wrap_errors
    return func(self, name, *args, **kwargs)
  File "/home/dave/Workspace/django-s3-storage/django_s3_storage/storage.py", line 365, in copy
    self.s3_connection.copy_object(
  File "/home/dave/Workspace/django-s3-storage/venv/lib/python3.9/site-packages/botocore/client.py", line 391, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/dave/Workspace/django-s3-storage/venv/lib/python3.9/site-packages/botocore/client.py", line 719, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the CopyObject operation: The specified key does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/dave/Workspace/django-s3-storage/tests/django_s3_storage_test/tests.py", line 174, in testCopy
    default_storage.copy("foo.txt", "bar.txt")
  File "/home/dave/Workspace/django-s3-storage/django_s3_storage/storage.py", line 42, in _do_wrap_errors
    raise err_cls("S3Storage error at {!r}: {}".format(name, force_str(ex)))
FileNotFoundError: S3Storage error at 'foo.txt': An error occurred (NoSuchKey) when calling the CopyObject operation: The specified key does not exist.

======================================================================
ERROR: testRename (django_s3_storage_test.tests.TestS3Storage)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dave/Workspace/django-s3-storage/django_s3_storage/storage.py", line 36, in _do_wrap_errors
    return func(self, name, *args, **kwargs)
  File "/home/dave/Workspace/django-s3-storage/django_s3_storage/storage.py", line 365, in copy
    self.s3_connection.copy_object(
  File "/home/dave/Workspace/django-s3-storage/venv/lib/python3.9/site-packages/botocore/client.py", line 391, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/dave/Workspace/django-s3-storage/venv/lib/python3.9/site-packages/botocore/client.py", line 719, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the CopyObject operation: The specified key does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/dave/Workspace/django-s3-storage/tests/django_s3_storage_test/tests.py", line 181, in testRename
    default_storage.rename("foo.txt", "bar.txt")
  File "/home/dave/Workspace/django-s3-storage/django_s3_storage/storage.py", line 36, in _do_wrap_errors
    return func(self, name, *args, **kwargs)
  File "/home/dave/Workspace/django-s3-storage/django_s3_storage/storage.py", line 372, in rename
    self.copy(src_name, dst_name, src_bucket)
  File "/home/dave/Workspace/django-s3-storage/django_s3_storage/storage.py", line 42, in _do_wrap_errors
    raise err_cls("S3Storage error at {!r}: {}".format(name, force_str(ex)))
FileNotFoundError: S3Storage error at 'foo.txt': An error occurred (NoSuchKey) when calling the CopyObject operation: The specified key does not exist.

----------------------------------------------------------------------
Ran 33 tests in 77.703s

FAILED (errors=2)

I think you need to set up a testing environment. Setting up an AWS account is pretty quick, and costs pennies to run these tests.

[emesik]

Could you please share some example settings file?

[emesik]

OK, I've found how to configure S3 access and I'm doing it like:

export AWS_ACCESS_KEY_ID="XXXXXXXXXXXXXXXXX"
export AWS_SECRET_ACCESS_KEY="yyyyyyyyyyyyyyyyyyyyyyyyyyyyy"
export AWS_S3_BUCKET_NAME="test"
export AWS_REGION="eu-west-1"
python3.9 -m coverage run tests/manage.py test tests

Then I receive the following error:

Traceback (most recent call last):
  File "/home/emes/devel/django-s3-storage/tests/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/emes/.local/lib/python3.9/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/home/emes/.local/lib/python3.9/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/emes/.local/lib/python3.9/site-packages/django/core/management/commands/test.py", line 23, in run_from_argv
    super().run_from_argv(argv)
  File "/home/emes/.local/lib/python3.9/site-packages/django/core/management/base.py", line 315, in run_from_argv
    parser = self.create_parser(argv[0], argv[1])
  File "/home/emes/.local/lib/python3.9/site-packages/django/core/management/base.py", line 289, in create_parser
    self.add_arguments(parser)
  File "/home/emes/.local/lib/python3.9/site-packages/django/core/management/commands/test.py", line 44, in add_arguments
    test_runner_class = get_runner(settings, self.test_runner)
  File "/home/emes/.local/lib/python3.9/site-packages/django/test/utils.py", line 303, in get_runner
    test_runner_class = test_runner_class or settings.TEST_RUNNER
  File "/home/emes/.local/lib/python3.9/site-packages/django/conf/__init__.py", line 79, in __getattr__
    self._setup(name)
  File "/home/emes/.local/lib/python3.9/site-packages/django/conf/__init__.py", line 66, in _setup
    self._wrapped = Settings(settings_module)
  File "/home/emes/.local/lib/python3.9/site-packages/django/conf/__init__.py", line 157, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'django_s3_storage_test'
/home/emes/.local/lib/python3.9/site-packages/coverage/inorout.py:520: CoverageWarning: Module django_s3_storage_test was never imported. (module-not-imported)
  self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")
/home/emes/.local/lib/python3.9/site-packages/coverage/control.py:768: CoverageWarning: No data was collected. (no-data-collected)
  self._warn("No data was collected.", slug="no-data-collected")

[etianen]

Possibly you also want to set PYTHONPATH=$(pwd):$PYTHONPATH

[emesik]

I think I managed to make it work. All tests are passing on my machine.

[etianen]

Thanks for taking the time to get the tests working! I'll get this released ASAP.