s3 copy_from fails if "key" has spaces in it.

[jeffmacdonald]

I'm writing a lambda handler that takes event notifications from a remote S3 bucket and copies a newly uploaded object into "local" bucket.

Here is the important code:

    bucket = event['Records'][0]['s3']['bucket']['name']
    key = event['Records'][0]['s3']['object']['key']

    try:
        s3.Object(bucket + '-clone', key).copy_from(CopySource=bucket + '/' + key)

When I upload a file to execute the code I get the following error in Cloudwatch:

An error occurred (AccessDenied) when calling the CopyObject operation: 
Access Denied: ClientError Traceback (most recent call last): 
File "/var/task/lambda_function.py", line 15, in lambda_handler raise e ClientError: 
An error occurred (AccessDenied) when calling the CopyObject operation: Access Denied

It seems that a space gets translated to a + somewhere along the way. I added the following key = key.replace("+", " ") to my code, and now it copies properly without error.

[JordonPhillips]

It seems like this has something to do with url encoding since all those params go into the url or into headers. Spaces should get translated as %20 though, so that's a bit strange. Could you post a debug log so I can get a better idea of whats going on under the hood (as I am unable to reproduce)? You can enable debug logging with boto3.set_stream_logger('botocore').

[jamesls]

Closing due to inactivity.

[seanpaulkelley]

I am also having the same problem. A key with a space in it fails to get retrieved.

for record in event['Records']: bucket = record['s3']['bucket']['name'] key = record['s3']['object']['key'] file_name_pattern = re.match( r'(.*?([^/]+)/?)$', key) file_name = file_name_pattern.group(2) download_path = '/tmp/{}{}'.format(uuid.uuid4(), file_name) s3_client.download_file(bucket, key, download_path)

My work around is to set key like key = urllib.unquote_plus(event['Records'][0]['s3']['object']['key'].encode("utf8"))

I added the stream loggger and here is my output

2016-12-21 17:48:48,423 botocore.vendored.requests.packages.urllib3.connectionpool [INFO] Starting new HTTPS connection (1): my-photos.s3.amazonaws.com [INFO] 2016-12-21T17:48:48.423Z b97124d2-c7a5-11e6-b626-712e694e52b4 Starting new HTTPS connection (1): my-photos.s3.amazonaws.com 2016-12-21 17:48:48,630 botocore.vendored.requests.packages.urllib3.connectionpool [DEBUG] "HEAD /9d781651-f5b9-445c-b238-721e0e581297Hannah%2BTrot.JPG HTTP/1.1" 404 0 [DEBUG] 2016-12-21T17:48:48.630Z b97124d2-c7a5-11e6-b626-712e694e52b4 "HEAD /9d781651-f5b9-445c-b238-721e0e581297Hannah%2BTrot.JPG HTTP/1.1" 404 0 2016-12-21 17:48:48,630 botocore.parsers [DEBUG] Response headers: {'x-amz-id-2': 'rp4ws0BtuN07AbdxRcN4JOM3BNg/kHFOixWhgPQL0PmUnNN7em3AP1I2n+PUHvl9j3ncpEeyPg4=', 'server': 'AmazonS3', 'transfer-encoding': 'chunked', 'x-amz-request-id': '24BB44067C63C2EE', 'date': 'Wed, 21 Dec 2016 17:48:47 GMT', 'content-type': 'application/xml'} [DEBUG] 2016-12-21T17:48:48.630Z b97124d2-c7a5-11e6-b626-712e694e52b4 Response headers: {'x-amz-id-2': 'rp4ws0BtuN07AbdxRcN4JOM3BNg/kHFOixWhgPQL0PmUnNN7em3AP1I2n+PUHvl9j3ncpEeyPg4=', 'server': 'AmazonS3', 'transfer-encoding': 'chunked', 'x-amz-request-id': '24BB44067C63C2EE', 'date': 'Wed, 21 Dec 2016 17:48:47 GMT', 'content-type': 'application/xml'} [DEBUG] 2016-12-21T17:48:48.630Z b97124d2-c7a5-11e6-b626-712e694e52b4 Response body:

[DEBUG] 2016-12-21T17:48:48.631Z b97124d2-c7a5-11e6-b626-712e694e52b4 Event needs-retry.s3.HeadObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7f09428fa690> [DEBUG] 2016-12-21T17:48:48.631Z b97124d2-c7a5-11e6-b626-712e694e52b4 No retry needed. [DEBUG] 2016-12-21T17:48:48.631Z b97124d2-c7a5-11e6-b626-712e694e52b4 Event needs-retry.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7f09426b4ad0>> 2016-12-21 17:48:48,630 botocore.parsers [DEBUG] Response body:

2016-12-21 17:48:48,631 botocore.hooks [DEBUG] Event needs-retry.s3.HeadObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7f09428fa690> 2016-12-21 17:48:48,631 botocore.retryhandler [DEBUG] No retry needed. 2016-12-21 17:48:48,631 botocore.hooks [DEBUG] Event needs-retry.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7f09426b4ad0>> An error occurred (404) when calling the HeadObject operation: Not Found: ClientError Traceback (most recent call last): File "/var/task/createThumbnail.py", line 47, in handler s3_client.download_file(bucket, key, download_path) File "/var/runtime/boto3/s3/inject.py", line 125, in download_file extra_args=ExtraArgs, callback=Callback) File "/var/runtime/boto3/s3/transfer.py", line 269, in download_file future.result() File "/var/runtime/s3transfer/futures.py", line 71, in result return self._coordinator.result() File "/var/runtime/s3transfer/futures.py", line 231, in result raise self._exception ClientError: An error occurred (404) when calling the HeadObject operation: Not Found

END RequestId: b97124d2-c7a5-11e6-b626-712e694e52b4

[moijes12]

Facing this same issue.

[charanjit-singh]

Me too while Accessing the File

[charanjit-singh]

From AWS S3 Web Interface:

URL: /Real Estate/ is encoded as: /Real%2520Estate/

---

From Jupyter Notebook:

When I simply use /Real Estate/:

Raises No Such Key Error

---

From Jupyter Notebook using Web Interface's encoding:

When I use /Real%20Estate/ as input:

Same No Such Key Error!!!

How to process names with spaces in BOTO ???

[mikeatlas]

I also encountered this issue; key = key.replace("+", " ") before making the boto call fixed it for me.

[chilcom]

You probably meant key = key.replace("%20", " ") That worked for me

[trulearn]

Neither '+' or '%20' works for white space(" ") of Key in s3.head_object (boto3 1.16.10) Need help to resolve

[GitHK]

Neither '+' or '%20' works for white space(" ") of Key in s3.head_object (boto3 1.16.10) Need help to resolve

I can confirm this as well.

[joshuamoreno1]

Hi, I have the same issue

[KAR2022]

I have the same issue. When i try to copy file from one folder to another in the same bucket using aws s3 cp.

[ibardarov-fms]

I have to do chomp and it worked!

[vadirajks]

this worked :

from urllib.parse import unquote_plus
file_to_transfer = S3.get_object(Bucket=bucket, Key=unquote_plus(key))