No such key error on copy object

[georgemillard]

I have a file system set up to connect to S3 using boto3.

I have written unit tests which create folders and files and test that they have been correctly uploaded to my bucket.

One test is passing 99 times out of 100 (on average). Occasionally it will fail with "An error occurred (NoSuchKey) when calling the CopyObject operation: The specified key does not exist."

A printout of the bucket contents shows that the key does in fact exists just before the call is made.

I can't understand why this would work most of the time but not always. Any suggestions would be really appreciated.

[georgemillard]

I believe the issue is a race condition due to repeatedly running put/copy/delete requests to boto3.

import boto3

from botocore.exceptions import ClientError

s3 = boto3.client(
    's3',
    aws_access_key_id = '****************',
    aws_secret_access_key = '****************')
bucket = '****************'

import logging

logging.basicConfig(
    filename='log.txt',
    level=logging.INFO,
    format=' %(asctime)s - %(levelname)s - %(message)s'
    )

def create_s3_folder(s3_key):
    """
    Creates an S3 'folder' (an empty S3 key)
    """
    response = ''
    try:
        response = s3.put_object(Bucket=bucket, Key=s3_key)
        logging.info('Created {0}, HTTPStatusCode: {1}'.format(s3_key, response['ResponseMetadata']['HTTPStatusCode']))
    except ClientError as e:
        logging.error("Received error: {0}".format(e), exc_info=True)
    return response

def copy_s3_object(old_s3_key, new_s3_key):
    """
    Copies an S3 Object
    """
    response = ''
    try:
        response = s3.copy_object(Bucket=bucket, CopySource=bucket+'/'+old_s3_key, Key=new_s3_key)
        logging.info('Copied {0} to {1}'.format(old_s3_key, new_s3_key))
    except ClientError as e:
        logging.error("Received error: {0}".format(e), exc_info=True)
    return response

def delete_s3_object(s3_key):
    """
    Deletes an S3 Object
    """
    response = ''
    try:
        response = s3.delete_object(Bucket=bucket, Key=s3_key)
        logging.info('Deleted {0}, HTTPStatusCode: {1}'.format(s3_key, response['ResponseMetadata']['HTTPStatusCode']))
    except ClientError as e:
        logging.error("Received error: {0}".format(e), exc_info=True)
    return response

for x in range(10000):
    logging.info('Running test {0} of 10000...'.format(x))
    r = create_s3_folder('media/f2/')
    r = create_s3_folder('media/f0/')
    r = copy_s3_object('media/f2/', 'media/f0/f2/')
    r = delete_s3_object('media/f2/')

This code produces the following error, usually after a few thousand tries! I have included the output for the previous passing test (1498) for comparison.

2017-07-04 15:53:08,822 - INFO - Running test 1498 of 10000...
 2017-07-04 15:53:09,051 - INFO - Created media/f2/, HTTPStatusCode: 200
 2017-07-04 15:53:09,226 - INFO - Created media/f0/, HTTPStatusCode: 200
 2017-07-04 15:53:09,340 - INFO - Copied media/f2/ to media/f0/f2/
 2017-07-04 15:53:09,448 - INFO - Deleted media/f2/, HTTPStatusCode: 204
 2017-07-04 15:53:09,448 - INFO - Running test 1499 of 10000...
 2017-07-04 15:53:09,607 - INFO - Created media/f2/, HTTPStatusCode: 200
 2017-07-04 15:53:09,718 - INFO - Created media/f0/, HTTPStatusCode: 200
 2017-07-04 15:53:09,830 - ERROR - Received error: An error occurred (NoSuchKey) when calling the CopyObject operation: The specified key does not exist.
Traceback (most recent call last):
  File "test.py", line 63, in copy_s3_object
    response = s3.copy_object(Bucket=bucket, CopySource=bucket+'/'+old_s3_key, Key=new_s3_key)
  File "/Users/georgemillard/programming/projects/virtualenvs/asset_manager_venv/lib/python3.6/site-packages/botocore/client.py", line 253, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/georgemillard/programming/projects/virtualenvs/asset_manager_venv/lib/python3.6/site-packages/botocore/client.py", line 557, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the CopyObject operation: The specified key does not exist.
 2017-07-04 15:53:09,940 - INFO - Deleted media/f2/, HTTPStatusCode: 204

[dstufft]

I believe that this is just the fact that S3 is eventually consistent and the key hasn't been replicated onto whatever node served your CopyObject.

[georgemillard]

Thank you for your explanation. Following eventual consistency, could you suggest how I might alter my code to handle this error when it happens?

[thanuj11]

@georgemillard I have the same issue, did you find any alternate ways of solving this problem?. If you can suggest what changes I can do to fix this issue will help

[georgemillard]

Hey @thanuj11 I'm afraid I didn't find a solution to this one, however, it only ever cropped up when I was stress testing the system with hundreds of create, copy, delete calls. I've never once encountered a problem in production.

[thanuj11]

Hey @georgemillard, I did a workaround to solve this eventually consistent issue by introducing a time delay to make the object eventually consistent and I will share what I did it may help you in your work also.

In your case: def delete_s3_object(s3_key): """ Deletes an S3 Object """ response = '' try: response = s3.delete_object(Bucket=bucket, Key=s3_key) logging.info('Deleted {0}, HTTPStatusCode: {1}'.format(s3_key, response['ResponseMetadata']['HTTPStatusCode'])) except ClientError as e: if e.response['Error']['Code'] == "NoSuchKey": print("No Such Key") time.sleep(30) response = s3.delete_object(Bucket=bucket, Key=s3_key) logging.info('Deleted {0}, HTTPStatusCode: {1}'.format(s3_key, response['ResponseMetadata']['HTTPStatusCode'])) else: logging.error("Received error: {0}".format(e), exc_info=True) return response

Note: I used 30 seconds delay as my file size is 100 MB, but it all depends on how big is your file.

You can also add the same lines to your copy and create functions