search

boto / boto3

AWS SDK for Python
https://aws.amazon.com/sdk-for-python/
Apache License 2.0
9.04k stars 1.87k forks source link

Couldn't parse the specified URI when with s3 bucket copy and unicode #1451

Closed boltronics closed 4 years ago

boltronics commented 6 years ago

The problem I'm having looks very similar to the issue here.

Using the same example as the one over at https://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Bucket.copy

$ pip freeze
boto3==1.5.26
botocore==1.8.40
docutils==0.14
jmespath==0.9.3
pkg-resources==0.0.0
python-dateutil==2.6.1
s3transfer==0.1.12
six==1.11.0
$ python
Python 3.5.3 (default, Jan 19 2017, 14:11:04) 
[GCC 6.3.0 20170118] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>> s3 = boto3.resource('s3')
>>> bucket = s3.Bucket('my-dst-bucket')
>>> test_key = 'test_key_with_🔊_in_name.jpg'
>>> type(test_key)
<class 'str'>
>>> copy_source = {'Bucket': 'my-src-bucket', 'Key': test_key}
>>> bucket.copy(copy_source, test_key)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/boto3-venv/lib/python3.5/site-packages/boto3/s3/inject.py", line 323, in bucket_copy
    Callback=Callback, SourceClient=SourceClient, Config=Config)
  File "/home/user/boto3-venv/lib/python3.5/site-packages/boto3/s3/inject.py", line 271, in copy
    return future.result()
  File "/home/user/boto3-venv/lib/python3.5/site-packages/s3transfer/futures.py", line 73, in result
    return self._coordinator.result()
  File "/home/user/boto3-venv/lib/python3.5/site-packages/s3transfer/futures.py", line 233, in result
    raise self._exception
  File "/home/user/boto3-venv/lib/python3.5/site-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/home/user/boto3-venv/lib/python3.5/site-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/home/user/boto3-venv/lib/python3.5/site-packages/s3transfer/copies.py", line 279, in _main
    CopySource=copy_source, Bucket=bucket, Key=key, **extra_args)
  File "/home/user/boto3-venv/lib/python3.5/site-packages/botocore/client.py", line 317, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/user/boto3-venv/lib/python3.5/site-packages/botocore/client.py", line 615, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidRequest) when calling the CopyObject operation: Couldn't parse the specified URI.
>>>

I've tried various methods to address this, such as replacing the 🔊 character with %F0%9F%94%8A (which is what S3 seems to use when generating a https link) but that just results in a botocore.exceptions.ClientError: An error occurred (404) when calling the HeadObject operation: Not Found exception.

I'm open to any suggestions or work-arounds.

jamesls commented 6 years ago

I'm not able to repro. Here's what I get when I try your example:

$ python
Python 3.6.3 (default, Oct  4 2017, 06:09:15)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.37)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>> boto3.__version__
'1.5.24'
>>> import botocore
>>> botocore.__version__
'1.8.40'
>>> s3 = boto3.resource('s3')
>>> bucket = s3.Bucket('mybucket')
>>> bucket
s3.Bucket(name='mybucket')
>>> key = 'test_key_with_🔊_in_name.jpg'
>>> bucket.copy({'Bucket': 'mybucket', 'Key': 'test_key_with_🔊_in_name.jpg'}, 'test_key_with_🔊_in_name2.jpg')
>>> list(bucket.objects.all())
[s3.ObjectSummary(bucket_name='mybucket', key='test_key_with_🔊_in_name.jpg'), s3.ObjectSummary(bucket_name='mybucket', key='test_key_with_🔊_in_name2.jpg')]

I'd suggest looking at your debug logs to see exactly what's being sent over the wire. You should see something like this:

"PUT /mybucket/test_key_with_%F0%9F%94%8A_in_name2.jpg HTTP/1.1" 200 234

If you share your debug logs I can take a look and see if anything stands out.

boltronics commented 6 years ago

Thanks! I appreciate your help. Please let me know if there's anything else I can provide that would be useful.

Note that I have performed a search/replace to change the bucket names and local file paths. I decided to leave the full key name untouched here as it's not particularly sensitive.

(boto3-venv) user@desktop:~/boto3-venv$ cat test.py 
#!/usr/bin/env python3

import logging

import boto3
import botocore

print(boto3.__version__)
print(botocore.__version__)

logging.basicConfig(format='%(levelname)s:%(message)s', level=logging.DEBUG)

logging.getLogger('boto3').setLevel(logging.DEBUG)
logging.getLogger('botocore').setLevel(logging.DEBUG)

s3 = boto3.resource('s3')
key = 'wp-content/uploads/2016/06/1466122480Eagle_Fruit_Store_and_Capital_Hotel___Lincoln__Nebraska__LOC____Flickr_-_Photo_Sharing__🔊-150x150.jpg'

bucket = s3.Bucket('my-dst-bucket')
print(bucket)

s3.Bucket(name='my-dst-bucket')
bucket.copy({'Bucket': 'my-src-bucket', 'Key': key}, key)
(boto3-venv) user@desktop:~/boto3-venv$ ./test.py 
1.5.26
1.8.40
DEBUG:Loading variable profile from defaults.
DEBUG:Loading variable config_file from defaults.
DEBUG:Loading variable credentials_file from defaults.
DEBUG:Loading variable data_path from defaults.
DEBUG:Loading JSON file: /home/user/boto3-venv/boto3/lib/python3.5/site-packages/boto3/data/s3/2006-03-01/resources-1.json
DEBUG:Loading variable region from environment with value 'us-west-2'.
DEBUG:Loading variable profile from defaults.
DEBUG:Loading variable ca_bundle from defaults.
DEBUG:Loading variable profile from defaults.
DEBUG:Loading variable credentials_file from defaults.
DEBUG:Loading variable config_file from defaults.
DEBUG:Loading variable profile from defaults.
DEBUG:Loading variable metadata_service_timeout from defaults.
DEBUG:Loading variable profile from defaults.
DEBUG:Loading variable metadata_service_num_attempts from defaults.
DEBUG:Loading variable profile from defaults.
DEBUG:Looking for credentials via: env
INFO:Found credentials in environment variables.
DEBUG:Loading JSON file: /home/user/boto3-venv/boto3/lib/python3.5/site-packages/botocore/data/endpoints.json
DEBUG:Loading variable profile from defaults.
DEBUG:Event choose-service-name: calling handler <function handle_service_name_alias at 0x7fa1422c1e18>
DEBUG:Loading JSON file: /home/user/boto3-venv/boto3/lib/python3.5/site-packages/botocore/data/s3/2006-03-01/service-2.json
DEBUG:Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7fa14277f8c8>
DEBUG:Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x7fa144b19488>
DEBUG:Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7fa14277f6a8>
DEBUG:The s3 config key is not a dictionary type, ignoring its value of: None
DEBUG:Setting s3 timeout as (60, 60)
DEBUG:Loading JSON file: /home/user/boto3-venv/boto3/lib/python3.5/site-packages/botocore/data/_retry.json
DEBUG:Registering retry handlers for service: s3
DEBUG:Defaulting to S3 virtual host style addressing with path style addressing fallback.
DEBUG:Loading s3:s3
DEBUG:Loading s3:Bucket
DEBUG:Renaming Bucket attribute name
DEBUG:Event creating-resource-class.s3.Bucket: calling handler <function lazy_call.<locals>._handler at 0x7fa14222b620>
s3.Bucket(name='my-dst-bucket')
DEBUG:Loading s3:Bucket
DEBUG:Renaming Bucket attribute name
DEBUG:Event creating-resource-class.s3.Bucket: calling handler <function lazy_call.<locals>._handler at 0x7fa14222b620>
DEBUG:Acquiring 0
DEBUG:CopySubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7fa141d69f28>}) about to wait for the following futures []
DEBUG:CopySubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7fa141d69f28>}) done waiting for dependent futures
DEBUG:Executing task CopySubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x7fa141d69f28>}) with kwargs {'request_executor': <s3transfer.futures.BoundedExecutor object at 0x7fa141d697f0>, 'transfer_future': <s3transfer.futures.TransferFuture object at 0x7fa141d69f28>, 'osutil': <s3transfer.utils.OSUtils object at 0x7fa141d695c0>, 'config': <boto3.s3.transfer.TransferConfig object at 0x7fa141ddbda0>, 'client': <botocore.client.S3 object at 0x7fa142090dd8>}
DEBUG:Event before-parameter-build.s3.HeadObject: calling handler <function sse_md5 at 0x7fa1422d9620>
DEBUG:Event before-parameter-build.s3.HeadObject: calling handler <function validate_bucket_name at 0x7fa1422d9598>
DEBUG:Event before-parameter-build.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7fa141dc3908>>
DEBUG:Event before-parameter-build.s3.HeadObject: calling handler <function generate_idempotent_uuid at 0x7fa1422d91e0>
DEBUG:Event before-call.s3.HeadObject: calling handler <function add_expect_header at 0x7fa1422d9a60>
DEBUG:Event before-call.s3.HeadObject: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7fa141dc3908>>
DEBUG:Making request for OperationModel(name=HeadObject) (verify_ssl=True) with params: {'url': 'https://s3.us-west-2.amazonaws.com/my-src-bucket/wp-content/uploads/2016/06/1466122480Eagle_Fruit_Store_and_Capital_Hotel___Lincoln__Nebraska__LOC____Flickr_-_Photo_Sharing__%F0%9F%94%8A-150x150.jpg', 'headers': {'User-Agent': 'Boto3/1.5.26 Python/3.5.3 Linux/4.14.0-0.bpo.3-amd64 Botocore/1.8.40 Resource'}, 'query_string': {}, 'method': 'HEAD', 'url_path': '/my-src-bucket/wp-content/uploads/2016/06/1466122480Eagle_Fruit_Store_and_Capital_Hotel___Lincoln__Nebraska__LOC____Flickr_-_Photo_Sharing__%F0%9F%94%8A-150x150.jpg', 'body': b'', 'context': {'has_streaming_input': False, 'client_config': <botocore.config.Config object at 0x7fa14209b2e8>, 'client_region': 'us-west-2', 'signing': {'bucket': 'my-src-bucket'}, 'auth_type': None}}
DEBUG:Event request-created.s3.HeadObject: calling handler <function signal_not_transferring at 0x7fa141e5c620>
DEBUG:Event request-created.s3.HeadObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fa142090e48>>
DEBUG:Event choose-signer.s3.HeadObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7fa1421ee668>>
DEBUG:Event choose-signer.s3.HeadObject: calling handler <function set_operation_specific_signer at 0x7fa1422d90d0>
DEBUG:Event before-sign.s3.HeadObject: calling handler <function fix_s3_host at 0x7fa14286dea0>
DEBUG:Calculating signature using v4 auth.
DEBUG:CanonicalRequest:
HEAD
/my-src-bucket/wp-content/uploads/2016/06/1466122480Eagle_Fruit_Store_and_Capital_Hotel___Lincoln__Nebraska__LOC____Flickr_-_Photo_Sharing__%F0%9F%94%8A-150x150.jpg

host:s3.us-west-2.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20180212T225420Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
DEBUG:StringToSign:
AWS4-HMAC-SHA256
20180212T225420Z
20180212/us-west-2/s3/aws4_request
ab18d7a1f95a268f1eec749ea9e39223cfcc1b5f340df3b4ec04430c2dd88f56
DEBUG:Signature:
57b29d54e96d4005add2c3ad89e3240a341a7efd639f58e0c25037ac911b0764
DEBUG:Event request-created.s3.HeadObject: calling handler <function signal_transferring at 0x7fa141e5c6a8>
DEBUG:Sending http request: <PreparedRequest [HEAD]>
INFO:Starting new HTTPS connection (1): s3.us-west-2.amazonaws.com
DEBUG:"HEAD /my-src-bucket/wp-content/uploads/2016/06/1466122480Eagle_Fruit_Store_and_Capital_Hotel___Lincoln__Nebraska__LOC____Flickr_-_Photo_Sharing__%F0%9F%94%8A-150x150.jpg HTTP/1.1" 301 0
DEBUG:Response headers: {'server': 'AmazonS3', 'date': 'Mon, 12 Feb 2018 22:54:20 GMT', 'x-amz-request-id': '3BF3723D40E38D0A', 'x-amz-id-2': 'Pr4ysAIxNDrg+J+EFIzKSMXL+QXnSwYYVqR4iEG4wsBOZuhkT+rYsCJOQG+urbVXP3wNc2pzcJk=', 'transfer-encoding': 'chunked', 'content-type': 'application/xml', 'x-amz-bucket-region': 'us-east-1'}
DEBUG:Response body:
b''
DEBUG:Event needs-retry.s3.HeadObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7fa141dc38d0>
DEBUG:No retry needed.
DEBUG:Event needs-retry.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7fa141dc3908>>
DEBUG:S3 client configured for region us-west-2 but the bucket my-src-bucket is in region us-east-1; Please configure the proper region to avoid multiple unnecessary redirects and signing attempts.
DEBUG:Updating URI from https://s3.us-west-2.amazonaws.com/my-src-bucket/wp-content/uploads/2016/06/1466122480Eagle_Fruit_Store_and_Capital_Hotel___Lincoln__Nebraska__LOC____Flickr_-_Photo_Sharing__%F0%9F%94%8A-150x150.jpg to https://s3.amazonaws.com/my-src-bucket/wp-content/uploads/2016/06/1466122480Eagle_Fruit_Store_and_Capital_Hotel___Lincoln__Nebraska__LOC____Flickr_-_Photo_Sharing__%F0%9F%94%8A-150x150.jpg
DEBUG:Response received to retry, sleeping for 0 seconds
DEBUG:Event request-created.s3.HeadObject: calling handler <function signal_not_transferring at 0x7fa141e5c620>
DEBUG:Event request-created.s3.HeadObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fa142090e48>>
DEBUG:Event choose-signer.s3.HeadObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7fa1421ee668>>
DEBUG:Event choose-signer.s3.HeadObject: calling handler <function set_operation_specific_signer at 0x7fa1422d90d0>
DEBUG:Event before-sign.s3.HeadObject: calling handler <function fix_s3_host at 0x7fa14286dea0>
DEBUG:Calculating signature using v4 auth.
DEBUG:CanonicalRequest:
HEAD
/my-src-bucket/wp-content/uploads/2016/06/1466122480Eagle_Fruit_Store_and_Capital_Hotel___Lincoln__Nebraska__LOC____Flickr_-_Photo_Sharing__%F0%9F%94%8A-150x150.jpg

host:s3.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20180212T225421Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
DEBUG:StringToSign:
AWS4-HMAC-SHA256
20180212T225421Z
20180212/us-east-1/s3/aws4_request
1088b0756aec816af69110aed59e9585071fc4c70df81ba08aaff31b2a46d010
DEBUG:Signature:
8d467c3e23a810a67570ec68e9e4b01d5cfb8f483a004d42430b770b935035a3
DEBUG:Event request-created.s3.HeadObject: calling handler <function signal_transferring at 0x7fa141e5c6a8>
DEBUG:Sending http request: <PreparedRequest [HEAD]>
INFO:Starting new HTTPS connection (1): s3.amazonaws.com
DEBUG:"HEAD /my-src-bucket/wp-content/uploads/2016/06/1466122480Eagle_Fruit_Store_and_Capital_Hotel___Lincoln__Nebraska__LOC____Flickr_-_Photo_Sharing__%F0%9F%94%8A-150x150.jpg HTTP/1.1" 200 0
DEBUG:Response headers: {'server': 'AmazonS3', 'x-amz-request-id': 'D18C616C3EBD23AB', 'date': 'Mon, 12 Feb 2018 22:54:23 GMT', 'last-modified': 'Fri, 17 Jun 2016 00:14:42 GMT', 'x-amz-id-2': 'GKGPCodf1wXnobWuoXxQf6B/ANL+WdjxHX7Cn4kp5Dk3/WGi/Gcfo+zvwYYW9wWjcZkPHWG4FTw=', 'etag': '"8b5865de71d2731526fbf692b9159352"', 'expires': 'Sat, 17 Jun 2017 00:14:41 GMT', 'content-type': 'image/jpeg', 'content-length': '9727', 'accept-ranges': 'bytes'}
DEBUG:Response body:
b''
DEBUG:Event needs-retry.s3.HeadObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7fa141dc38d0>
DEBUG:No retry needed.
DEBUG:Event needs-retry.s3.HeadObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7fa141dc3908>>
DEBUG:Submitting task CopyObjectTask(transfer_id=0, {'extra_args': {}, 'bucket': 'my-dst-bucket', 'key': 'wp-content/uploads/2016/06/1466122480Eagle_Fruit_Store_and_Capital_Hotel___Lincoln__Nebraska__LOC____Flickr_-_Photo_Sharing__🔊-150x150.jpg'}) to executor <s3transfer.futures.BoundedExecutor object at 0x7fa141d697f0> for transfer request: 0.
DEBUG:Acquiring 0
DEBUG:CopyObjectTask(transfer_id=0, {'extra_args': {}, 'bucket': 'my-dst-bucket', 'key': 'wp-content/uploads/2016/06/1466122480Eagle_Fruit_Store_and_Capital_Hotel___Lincoln__Nebraska__LOC____Flickr_-_Photo_Sharing__🔊-150x150.jpg'}) about to wait for the following futures []
DEBUG:CopyObjectTask(transfer_id=0, {'extra_args': {}, 'bucket': 'my-dst-bucket', 'key': 'wp-content/uploads/2016/06/1466122480Eagle_Fruit_Store_and_Capital_Hotel___Lincoln__Nebraska__LOC____Flickr_-_Photo_Sharing__🔊-150x150.jpg'}) done waiting for dependent futures
DEBUG:Executing task CopyObjectTask(transfer_id=0, {'extra_args': {}, 'bucket': 'my-dst-bucket', 'key': 'wp-content/uploads/2016/06/1466122480Eagle_Fruit_Store_and_Capital_Hotel___Lincoln__Nebraska__LOC____Flickr_-_Photo_Sharing__🔊-150x150.jpg'}) with kwargs {'size': 9727, 'extra_args': {}, 'bucket': 'my-dst-bucket', 'client': <botocore.client.S3 object at 0x7fa142090dd8>, 'callbacks': [], 'copy_source': {'Key': 'wp-content/uploads/2016/06/1466122480Eagle_Fruit_Store_and_Capital_Hotel___Lincoln__Nebraska__LOC____Flickr_-_Photo_Sharing__🔊-150x150.jpg', 'Bucket': 'my-src-bucket'}, 'key': 'wp-content/uploads/2016/06/1466122480Eagle_Fruit_Store_and_Capital_Hotel___Lincoln__Nebraska__LOC____Flickr_-_Photo_Sharing__🔊-150x150.jpg'}
DEBUG:Event before-parameter-build.s3.CopyObject: calling handler <function handle_copy_source_param at 0x7fa1422d9b70>
DEBUG:Event before-parameter-build.s3.CopyObject: calling handler <function validate_ascii_metadata at 0x7fa1422da268>
DEBUG:Releasing acquire 0/None
DEBUG:Event before-parameter-build.s3.CopyObject: calling handler <function sse_md5 at 0x7fa1422d9620>
DEBUG:Event before-parameter-build.s3.CopyObject: calling handler <function copy_source_sse_md5 at 0x7fa1422d96a8>
DEBUG:Event before-parameter-build.s3.CopyObject: calling handler <function validate_bucket_name at 0x7fa1422d9598>
DEBUG:Event before-parameter-build.s3.CopyObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7fa141dc3908>>
DEBUG:Event before-parameter-build.s3.CopyObject: calling handler <function generate_idempotent_uuid at 0x7fa1422d91e0>
DEBUG:Event before-call.s3.CopyObject: calling handler <function add_expect_header at 0x7fa1422d9a60>
DEBUG:Event before-call.s3.CopyObject: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7fa141dc3908>>
DEBUG:Making request for OperationModel(name=CopyObject) (verify_ssl=True) with params: {'url': 'https://s3.us-west-2.amazonaws.com/my-dst-bucket/wp-content/uploads/2016/06/1466122480Eagle_Fruit_Store_and_Capital_Hotel___Lincoln__Nebraska__LOC____Flickr_-_Photo_Sharing__%F0%9F%94%8A-150x150.jpg', 'headers': {'User-Agent': 'Boto3/1.5.26 Python/3.5.3 Linux/4.14.0-0.bpo.3-amd64 Botocore/1.8.40 Resource', 'x-amz-copy-source': 'my-src-bucket/wp-content/uploads/2016/06/1466122480Eagle_Fruit_Store_and_Capital_Hotel___Lincoln__Nebraska__LOC____Flickr_-_Photo_Sharing__%F0%9F%94%8A-150x150.jpg'}, 'query_string': {}, 'method': 'PUT', 'url_path': '/my-dst-bucket/wp-content/uploads/2016/06/1466122480Eagle_Fruit_Store_and_Capital_Hotel___Lincoln__Nebraska__LOC____Flickr_-_Photo_Sharing__%F0%9F%94%8A-150x150.jpg', 'body': b'', 'context': {'has_streaming_input': False, 'client_config': <botocore.config.Config object at 0x7fa14209b2e8>, 'client_region': 'us-west-2', 'signing': {'bucket': 'my-dst-bucket'}, 'auth_type': None}}
DEBUG:Event request-created.s3.CopyObject: calling handler <function signal_not_transferring at 0x7fa141e5c620>
DEBUG:Event request-created.s3.CopyObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fa142090e48>>
DEBUG:Event choose-signer.s3.CopyObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7fa1421ee668>>
DEBUG:Event choose-signer.s3.CopyObject: calling handler <function set_operation_specific_signer at 0x7fa1422d90d0>
DEBUG:Event before-sign.s3.CopyObject: calling handler <function fix_s3_host at 0x7fa14286dea0>
DEBUG:Calculating signature using v4 auth.
DEBUG:CanonicalRequest:
PUT
/my-dst-bucket/wp-content/uploads/2016/06/1466122480Eagle_Fruit_Store_and_Capital_Hotel___Lincoln__Nebraska__LOC____Flickr_-_Photo_Sharing__%F0%9F%94%8A-150x150.jpg

host:s3.us-west-2.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-copy-source:my-src-bucket/wp-content/uploads/2016/06/1466122480Eagle_Fruit_Store_and_Capital_Hotel___Lincoln__Nebraska__LOC____Flickr_-_Photo_Sharing__%F0%9F%94%8A-150x150.jpg
x-amz-date:20180212T225422Z

host;x-amz-content-sha256;x-amz-copy-source;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
DEBUG:StringToSign:
AWS4-HMAC-SHA256
20180212T225422Z
20180212/us-west-2/s3/aws4_request
82e237be4b1e0c7ee46435155d75c41066751af81b14ebd62c888349a44bd2ee
DEBUG:Signature:
851660f1b58e30e57d6abcd466c77841d7e9d39d5a70d2fbff4c64d890ae69e2
DEBUG:Event request-created.s3.CopyObject: calling handler <function signal_transferring at 0x7fa141e5c6a8>
DEBUG:Sending http request: <PreparedRequest [PUT]>
INFO:Starting new HTTPS connection (2): s3.us-west-2.amazonaws.com
DEBUG:"PUT /my-dst-bucket/wp-content/uploads/2016/06/1466122480Eagle_Fruit_Store_and_Capital_Hotel___Lincoln__Nebraska__LOC____Flickr_-_Photo_Sharing__%F0%9F%94%8A-150x150.jpg HTTP/1.1" 400 None
DEBUG:Response headers: {'server': 'AmazonS3', 'date': 'Mon, 12 Feb 2018 22:54:22 GMT', 'x-amz-request-id': 'A69F9EF4300CB7DD', 'x-amz-id-2': 'MBsYpZ6+SSMObxOeFK325+QNkItb8W2p81g26zRD+PpKJ6dxQ8zC6ouhp4k79x78utqwddEYvjc=', 'transfer-encoding': 'chunked', 'content-type': 'application/xml', 'connection': 'close'}
DEBUG:Response body:
b"<Error><Code>InvalidRequest</Code><Message>Couldn't parse the specified URI.</Message><RequestId>A69F9EF4300CB7DD</RequestId><HostId>MBsYpZ6+SSMObxOeFK325+QNkItb8W2p81g26zRD+PpKJ6dxQ8zC6ouhp4k79x78utqwddEYvjc=</HostId></Error>"
DEBUG:Event needs-retry.s3.CopyObject: calling handler <function check_for_200_error at 0x7fa1422d6f28>
DEBUG:Event needs-retry.s3.CopyObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7fa141dc38d0>
DEBUG:No retry needed.
DEBUG:Event needs-retry.s3.CopyObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7fa141dc3908>>
DEBUG:Exception raised.
Traceback (most recent call last):
  File "/home/user/boto3-venv/boto3/lib/python3.5/site-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/home/user/boto3-venv/boto3/lib/python3.5/site-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/home/user/boto3-venv/boto3/lib/python3.5/site-packages/s3transfer/copies.py", line 279, in _main
    CopySource=copy_source, Bucket=bucket, Key=key, **extra_args)
  File "/home/user/boto3-venv/boto3/lib/python3.5/site-packages/botocore/client.py", line 317, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/user/boto3-venv/boto3/lib/python3.5/site-packages/botocore/client.py", line 615, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidRequest) when calling the CopyObject operation: Couldn't parse the specified URI.
DEBUG:Releasing acquire 0/None
Traceback (most recent call last):
  File "./test.py", line 24, in <module>
    bucket.copy({'Bucket': 'my-src-bucket', 'Key': key}, key)
  File "/home/user/boto3-venv/boto3/lib/python3.5/site-packages/boto3/s3/inject.py", line 323, in bucket_copy
    Callback=Callback, SourceClient=SourceClient, Config=Config)
  File "/home/user/boto3-venv/boto3/lib/python3.5/site-packages/boto3/s3/inject.py", line 271, in copy
    return future.result()
  File "/home/user/boto3-venv/boto3/lib/python3.5/site-packages/s3transfer/futures.py", line 73, in result
    return self._coordinator.result()
  File "/home/user/boto3-venv/boto3/lib/python3.5/site-packages/s3transfer/futures.py", line 233, in result
    raise self._exception
  File "/home/user/boto3-venv/boto3/lib/python3.5/site-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/home/user/boto3-venv/boto3/lib/python3.5/site-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/home/user/boto3-venv/boto3/lib/python3.5/site-packages/s3transfer/copies.py", line 279, in _main
    CopySource=copy_source, Bucket=bucket, Key=key, **extra_args)
  File "/home/user/boto3-venv/boto3/lib/python3.5/site-packages/botocore/client.py", line 317, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/user/boto3-venv/boto3/lib/python3.5/site-packages/botocore/client.py", line 615, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidRequest) when calling the CopyObject operation: Couldn't parse the specified URI.
(boto3-venv) user@desktop:~/boto3-venv$

I see there are debug lines like S3 client configured for region us-west-2 but the bucket my-src-bucket is in region us-east-1, but switching to us-east-1 still results in a ClientError exception. The source bucket is in us-east-1 but the destination bucket is in us-west-1 (in a different AWS account).

FWIW, I've been running a script that iterates over thousands of S3 keys doing the same operations. I only hit issues when I encountered this key, which happens to contain the unicode emoji character in the name.

no-response[bot] commented 6 years ago

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

boltronics commented 6 years ago

Wait, what? I was the last person to respond, and I gave all the requested information. I think the bot here is broken.

Also confirming this problem still exists in boto3 1.7.25 with Python 3.5.3 on Debian Stretch in a clean venv.

pip freeze
boto3==1.7.25
botocore==1.10.25
docutils==0.14
jmespath==0.9.3
pkg-resources==0.0.0
python-dateutil==2.7.3
s3transfer==0.1.13
six==1.11.0
joguSD commented 6 years ago

@boltronics I was able to reproduce this. The key here is the buckets being in different regions. I tried various permutations of keys with unicode/not, src/dest in different regions, and src/dest in different accounts. The only case I found it wouldn't work is if the src/dest buckets are in different regions AND have you use that key (even other unicode keys worked fine). I tried to do the same copy in the console and it failed as well. I'm 99% sure this is actually an issue with the S3 service.

boltronics commented 6 years ago

Thanks @joguSD for your efforts in narrowing it down! I'm glad someone else was able to reproduce this, even if there's nothing we can do but report it to AWS.

swetashre commented 4 years ago

@boltronics I was able to reproduce this. The key here is the buckets being in different regions. I tried various permutations of keys with unicode/not, src/dest in different regions, and src/dest in different accounts. The only case I found it wouldn't work is if the src/dest buckets are in different regions AND have you use that key (even other unicode keys worked fine). I tried to do the same copy in the console and it failed as well. I'm 99% sure this is actually an issue with the S3 service.

According to the reply this issue is related to s3. So we can't do much on the SDK side to fix it. I am closing the issue.

boltronics commented 4 years ago

Wouldn't it be possible to catch the exception and print a more helpful error message?