search

boto / boto3

AWS SDK for Python
https://aws.amazon.com/sdk-for-python/
Apache License 2.0
8.96k stars 1.86k forks source link

generate_presigned_url not working with complete_multipart_upload #2192

Open Caian opened 4 years ago

Caian commented 4 years ago

Hello, I am trying to generate a presigned URL for complete_multipart_upload, but I'm getting this error when running my code:

Traceback (most recent call last):
  File "/dev-env/test/chalicelib/rmodel.py", line 280, in __call__
    return funct(*args, **kwargs)
  File "/dev-env/test/chalicelib/upload.py", line 201, in create_complete_upload
    'MultipartUpload': upload})
  File "/dev-env/venv/lib/python3.7/site-packages/botocore/signers.py", line 595, in generate_presigned_url
    operation_name=operation_name)
  File "/dev-env/venv/lib/python3.7/site-packages/botocore/signers.py", line 268, in generate_presigned_url
    'presign-url', expires_in, signing_name)
  File "/dev-env/venv/lib/python3.7/site-packages/botocore/signers.py", line 157, in sign
    auth.add_auth(request)
  File "/dev-env/venv/lib/python3.7/site-packages/botocore/auth.py", line 362, in add_auth
    self._modify_request_before_signing(request)
  File "/dev-env/venv/lib/python3.7/site-packages/botocore/auth.py", line 525, in _modify_request_before_signing
    query_dict.update(self._get_body_as_dict(request))
  File "/dev-env/venv/lib/python3.7/site-packages/botocore/auth.py", line 550, in _get_body_as_dict
    data = json.loads(data.decode('utf-8'))
  File "/dev-env/Packages/miniconda3/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/dev-env/Packages/miniconda3/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/dev-env/Packages/miniconda3/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

This is the code:

s3_client = boto3.client('s3')
signed_url = s3_client.generate_presigned_url('complete_multipart_upload',
    ExpiresIn=3600,
    HttpMethod='POST',
    Params={
        'Bucket': _BUCKET,
        'Key': key,
        'UploadId': upload_id,
        'MultipartUpload': upload})

If I call s3_client.complete_multipart_upload with the same arguments specified in Params, then the operation completes without issues. From the error it looks like AWS is returning an unexpected response message to boto.

The boto3 version is 1.10.2

Thank you,

swetashre commented 4 years ago

@Caian - Thank you for your post. generate_presigned_url is a local operation. It does not make any api call to aws. I believe you are running some other code in addition to the code you have given me. It would be helpful if you could give me full debug log which might help us in knowing the exact issue. You can enable log by adding boto3.set_stream_logger('') to your code.

This is the code i ran for generating presigned url for complete_multipart_upload and i am not getting error with this piece of code. Can you try running this code only and see if you are still getting that error ?

import boto3
s3 = boto3.client('s3')

upload = {'Parts': [{'ETag': 'string','PartNumber': 123}]} 
signed_url = s3.generate_presigned_url('complete_multipart_upload',
    ExpiresIn=3600,
    HttpMethod='POST',
    Params={
        'Bucket': 'bucket_name',
        'Key': 'key',
        'UploadId': 'upload_id',
        'MultipartUpload': upload})
print(signed_url)
Caian commented 4 years ago

@swetashre - Thank you for the response. Running your snippet under the virtual environment I'm using produces the same json decoder error.

I would like to correct my previous statement regarding the version of boto3 that is actually 1.10.7.

Also Python iterpreter is Anaconda Python 3.7.1.

Here is the list of installed packages in case there are any compatibility issues:

Package             Version  
------------------- ---------
arrow               0.15.2   
astroid             2.3.2    
attrs               19.3.0   
aws-lambda-builders 0.5.0    
aws-sam-cli         0.30.0   
aws-sam-translator  1.15.1   
awscli              1.16.271 
binaryornot         0.4.4    
boto3               1.10.7   
botocore            1.13.7   
certifi             2019.9.11
chalice             1.12.0   
chardet             3.0.4    
chevron             0.13.1   
Click               7.0      
colorama            0.4.1    
cookiecutter        1.6.0    
dateparser          0.7.2    
docker              4.1.0    
docutils            0.15.2   
enum-compat         0.0.3    
Flask               1.0.4    
future              0.18.2   
httpie              1.0.3    
idna                2.8      
importlib-metadata  0.23     
isort               4.3.21   
itsdangerous        1.1.0    
Jinja2              2.10.3   
jinja2-time         0.2.0    
jmespath            0.9.4    
jsonschema          3.1.1    
lazy-object-proxy   1.4.3    
MarkupSafe          1.1.1    
mccabe              0.6.1    
more-itertools      7.2.0    
pip                 10.0.1   
poyo                0.5.0    
pyasn1              0.4.7    
Pygments            2.4.2    
pylint              2.4.3    
pyrsistent          0.15.5   
python-dateutil     2.8.0    
pytz                2019.3   
PyYAML              5.1.2    
regex               2019.11.1
requests            2.22.0   
rsa                 3.4.2    
s3transfer          0.2.1    
serverlessrepo      0.1.9    
setuptools          39.0.1   
six                 1.12.0   
tqdm                4.37.0   
typed-ast           1.4.0    
tzlocal             2.0.0    
urllib3             1.25.6   
websocket-client    0.56.0   
Werkzeug            0.16.0   
wheel               0.33.6   
whichcraft          0.6.1    
wrapt               1.11.2   
zipp                0.6.0
Caian commented 4 years ago

Digging a bit further into the error it looks like the json decoder is having problem decoding the following string:

<CompleteMultipartUpload xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Part><ETag>string</ETag><PartNumber>123</PartNumber></Part></CompleteMultipartUpload>

I wonder why an XML string is being passed to a JSON decoder.

Caian commented 4 years ago

Ok found it. The problem was the s3v4 signature I had in my configuration so I could also work with S3-compatible-ish providers:

s3 =
    signature_version = s3v4

Removing these lines, although not ideal, solved the error. Curiously I didn't have any problems signing my part upload PUT requests. Is this a specific limitation of POST/complete_multipart_upload or a bug?

swetashre commented 4 years ago

I am glad you got it working. But the error you were getting is nothing to do with signature_version = s3v4 . Can you please give me full stack trace which can help us in knowing the cause of the issue? You can enable log by adding boto3.set_stream_logger('') to your code.

You can specify s3v4 signature in the config during client creation. something like this:

import boto3
from botocore.client import Config

s3 = boto3.client('s3' , config=Config(signature_version='s3v4'))
Caian commented 4 years ago

Here is the log output without s3v4 being passed on either the configuration or the client call (does not fail):

2019-11-17 08:05:38,200 botocore.hooks [DEBUG] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2019-11-17 08:05:38,220 botocore.hooks [DEBUG] Changing event name from before-call.apigateway to before-call.api-gateway
2019-11-17 08:05:38,222 botocore.hooks [DEBUG] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2019-11-17 08:05:38,232 botocore.hooks [DEBUG] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2019-11-17 08:05:38,233 botocore.hooks [DEBUG] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2019-11-17 08:05:38,236 botocore.hooks [DEBUG] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2019-11-17 08:05:38,241 botocore.hooks [DEBUG] Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2019-11-17 08:05:38,255 botocore.hooks [DEBUG] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2019-11-17 08:05:38,257 botocore.hooks [DEBUG] Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2019-11-17 08:05:38,259 botocore.hooks [DEBUG] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2019-11-17 08:05:38,260 botocore.hooks [DEBUG] Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2019-11-17 08:05:38,297 botocore.credentials [DEBUG] Looking for credentials via: env
2019-11-17 08:05:38,299 botocore.credentials [DEBUG] Looking for credentials via: assume-role
2019-11-17 08:05:38,300 botocore.credentials [DEBUG] Looking for credentials via: assume-role-with-web-identity
2019-11-17 08:05:38,301 botocore.credentials [DEBUG] Looking for credentials via: shared-credentials-file
2019-11-17 08:05:38,302 botocore.credentials [INFO] Found credentials in shared credentials file: ~/.aws/credentials
2019-11-17 08:05:38,307 botocore.loaders [DEBUG] Loading JSON file: /home/caian/Projects/Cloud/venv/lib/python3.6/site-packages/botocore/data/endpoints.json
2019-11-17 08:05:38,318 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f80ebf4ae18>
2019-11-17 08:05:38,332 botocore.loaders [DEBUG] Loading JSON file: /home/caian/Projects/Cloud/venv/lib/python3.6/site-packages/botocore/data/s3/2006-03-01/service-2.json
2019-11-17 08:05:38,349 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7f80ebfb4158>
2019-11-17 08:05:38,350 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x7f810406d6a8>
2019-11-17 08:05:38,381 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7f80ebfaaea0>
2019-11-17 08:05:38,383 botocore.args [DEBUG] The s3 config key is not a dictionary type, ignoring its value of: None
2019-11-17 08:05:38,396 botocore.endpoint [DEBUG] Setting s3 timeout as (60, 60)
2019-11-17 08:05:38,401 botocore.loaders [DEBUG] Loading JSON file: /home/caian/Projects/Cloud/venv/lib/python3.6/site-packages/botocore/data/_retry.json
2019-11-17 08:05:38,406 botocore.client [DEBUG] Registering retry handlers for service: s3
2019-11-17 08:05:38,409 botocore.client [DEBUG] Defaulting to S3 virtual host style addressing with path style addressing fallback.
2019-11-17 08:05:38,412 botocore.hooks [DEBUG] Event before-parameter-build.s3.CompleteMultipartUpload: calling handler <function validate_bucket_name at 0x7f80ebf6fb70>
2019-11-17 08:05:38,413 botocore.hooks [DEBUG] Event before-parameter-build.s3.CompleteMultipartUpload: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7f80eb87acf8>>
2019-11-17 08:05:38,413 botocore.hooks [DEBUG] Event before-parameter-build.s3.CompleteMultipartUpload: calling handler <function generate_idempotent_uuid at 0x7f80ebf6f7b8>
2019-11-17 08:05:38,417 botocore.hooks [DEBUG] Event choose-signer.s3.CompleteMultipartUpload: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7f81071520f0>>
2019-11-17 08:05:38,418 botocore.hooks [DEBUG] Event before-sign.s3.CompleteMultipartUpload: calling handler <function fix_s3_host at 0x7f810020d7b8>
2019-11-17 08:05:38,419 botocore.utils [DEBUG] Checking for DNS compatible bucket for: https://s3.amazonaws.com/bucket_name/key?uploadId=upload_id
2019-11-17 08:05:38,420 botocore.utils [DEBUG] Not changing URI, bucket is not DNS compatible: bucket_name
2019-11-17 08:05:38,420 botocore.auth [DEBUG] Calculating signature using hmacv1 auth.
2019-11-17 08:05:38,421 botocore.auth [DEBUG] HTTP request method: POST
2019-11-17 08:05:38,424 botocore.auth [DEBUG] StringToSign:

And here is with s3v4 in the client as per your request (does fail):

2019-11-17 08:00:49,169 botocore.hooks [DEBUG] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2019-11-17 08:00:49,182 botocore.hooks [DEBUG] Changing event name from before-call.apigateway to before-call.api-gateway
2019-11-17 08:00:49,184 botocore.hooks [DEBUG] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2019-11-17 08:00:49,191 botocore.hooks [DEBUG] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2019-11-17 08:00:49,192 botocore.hooks [DEBUG] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2019-11-17 08:00:49,194 botocore.hooks [DEBUG] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2019-11-17 08:00:49,197 botocore.hooks [DEBUG] Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2019-11-17 08:00:49,205 botocore.hooks [DEBUG] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2019-11-17 08:00:49,207 botocore.hooks [DEBUG] Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2019-11-17 08:00:49,208 botocore.hooks [DEBUG] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2019-11-17 08:00:49,209 botocore.hooks [DEBUG] Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2019-11-17 08:00:49,249 botocore.credentials [DEBUG] Looking for credentials via: env
2019-11-17 08:00:49,253 botocore.credentials [DEBUG] Looking for credentials via: assume-role
2019-11-17 08:00:49,254 botocore.credentials [DEBUG] Looking for credentials via: assume-role-with-web-identity
2019-11-17 08:00:49,255 botocore.credentials [DEBUG] Looking for credentials via: shared-credentials-file
2019-11-17 08:00:49,258 botocore.credentials [INFO] Found credentials in shared credentials file: ~/.aws/credentials
2019-11-17 08:00:49,260 botocore.loaders [DEBUG] Loading JSON file: /home/caian/Projects/Cloud/venv/lib/python3.6/site-packages/botocore/data/endpoints.json
2019-11-17 08:01:02,480 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f3de8730268>
2019-11-17 08:01:02,499 botocore.loaders [DEBUG] Loading JSON file: /home/caian/Projects/Cloud/venv/lib/python3.6/site-packages/botocore/data/s3/2006-03-01/service-2.json
2019-11-17 08:01:02,517 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7f3ddd7241e0>
2019-11-17 08:01:02,518 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x7f3de57f19d8>
2019-11-17 08:01:02,551 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7f3ddd718f28>
2019-11-17 08:01:02,553 botocore.args [DEBUG] The s3 config key is not a dictionary type, ignoring its value of: None
2019-11-17 08:01:02,575 botocore.endpoint [DEBUG] Setting s3 timeout as (60, 60)
2019-11-17 08:01:02,582 botocore.loaders [DEBUG] Loading JSON file: /home/caian/Projects/Cloud/venv/lib/python3.6/site-packages/botocore/data/_retry.json
2019-11-17 08:01:02,584 botocore.client [DEBUG] Registering retry handlers for service: s3
2019-11-17 08:01:02,587 botocore.client [DEBUG] Defaulting to S3 virtual host style addressing with path style addressing fallback.
2019-11-17 08:01:02,590 botocore.hooks [DEBUG] Event before-parameter-build.s3.CompleteMultipartUpload: calling handler <function validate_bucket_name at 0x7f3ddd6cfbf8>
2019-11-17 08:01:02,591 botocore.hooks [DEBUG] Event before-parameter-build.s3.CompleteMultipartUpload: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7f3ddcff3dd8>>
2019-11-17 08:01:02,591 botocore.hooks [DEBUG] Event before-parameter-build.s3.CompleteMultipartUpload: calling handler <function generate_idempotent_uuid at 0x7f3ddd6cf840>
2019-11-17 08:01:02,602 botocore.hooks [DEBUG] Event choose-signer.s3.CompleteMultipartUpload: calling handler <function set_operation_specific_signer at 0x7f3ddd6cf730>
2019-11-17 08:01:02,603 botocore.hooks [DEBUG] Event before-sign.s3.CompleteMultipartUpload: calling handler <function fix_s3_host at 0x7f3ddd967840>
2019-11-17 08:01:02,604 botocore.utils [DEBUG] Checking for DNS compatible bucket for: https://s3.amazonaws.com/bucket_name/key?uploadId=upload_id
2019-11-17 08:01:02,605 botocore.utils [DEBUG] Not changing URI, bucket is not DNS compatible: bucket_name

The full stack trace is the one from the issue itself, is there anything I missed when pasting it?

I wish I could go further into the JSON library but the method being called is not Python code anymore so I can't step into it.

Caian commented 4 years ago

So instead of comparing the execution with and without s3v4, I decided to compare it with s3v4 signing an upload_part. The following code works.

import boto3

boto3.set_stream_logger('')
s3 = boto3.client('s3', config=boto3.session.Config(signature_version='s3v4'))

upload = {'Parts': [{'ETag': 'string','PartNumber': 123}]}
signed_url = s3.generate_presigned_url('upload_part',
    ExpiresIn=3600,
    HttpMethod='PUT',
    Params={
        'Bucket': 'bucket_name',
        'Key': 'key',
        'UploadId': 'upload_id',
        'PartNumber': 1})
print(signed_url)

Both cases use botocore.auth.S3SigV4QueryAuth to handle add_auth.

From what I can debug, complete_multipart_upload populates the body member of request (upload_part doesn't), this causes _get_body_as_dict to be called on line 525 of auth.py, which calls json.loads with the aforementioned string in request.body that is nothing like a JSON.

Now this request.body is being generated in signers.py:581 when calling serializer.serialize_to_request. Unfortunately I can't evaluate the correctness of the process itself, but what I can tell is that the shape of the request (StructureShape(CompleteMultipartUploadRequest) wants to serialize the MultipartUpload attribute to the body using the RestXMLSerializer.

Best regards,

github-actions[bot] commented 3 years ago

Greetings! It looks like this issue hasn’t been active in longer than one year. We encourage you to check if this is still an issue in the latest release. Because it has been longer than one year since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment to prevent automatic closure, or if the issue is already closed, please feel free to reopen it.

Caian commented 3 years ago

The problem persists in boto3 1.17.19, botocore 1.20.19, Python 3.8.5.

Swapping from:

s3 = boto3.client('s3', config=boto3.session.Config())

To:

s3 = boto3.client('s3', config=boto3.session.Config(signature_version='s3v4'))

Still manifests the issue.

kdaily commented 3 years ago

Hi @Caian, I can reproduce this issue. Thanks for your debugging thus far. Marking as a bug and will investigate further.

alonbl commented 2 years ago

Reproduction does not need any credentials or actual objects, but simple invocation.

import boto3
from botocore.client import Config

s3 = boto3.client(
    's3',
    region_name="us-east-1",
    config=Config(signature_version="s3v4"),
)

signed_url = s3.generate_presigned_url(
    ClientMethod ='complete_multipart_upload',
    Params = {
       'Bucket': "bucket1",
       'UploadId': "id1",
       'Key': "key1",
       'UploadId': "id1",
       'MultipartUpload': {'Parts': []},
    }
)
Traceback (most recent call last):
  File "/home/alonbl/exodigo/alonbl-misc/aws-example/a.py", line 10, in <module>
    signed_url = s3.generate_presigned_url(
  File "/home/alonbl/exodigo/alonbl-misc/aws-example/x1/lib/python3.10/site-packages/botocore/signers.py", line 661, in generate_presigned_url
    return request_signer.generate_presigned_url(
  File "/home/alonbl/exodigo/alonbl-misc/aws-example/x1/lib/python3.10/site-packages/botocore/signers.py", line 312, in generate_presigned_url
    self.sign(
  File "/home/alonbl/exodigo/alonbl-misc/aws-example/x1/lib/python3.10/site-packages/botocore/signers.py", line 187, in sign
    auth.add_auth(request)
  File "/home/alonbl/exodigo/alonbl-misc/aws-example/x1/lib/python3.10/site-packages/botocore/auth.py", line 412, in add_auth
    self._modify_request_before_signing(request)
  File "/home/alonbl/exodigo/alonbl-misc/aws-example/x1/lib/python3.10/site-packages/botocore/auth.py", line 576, in _modify_request_before_signing
    query_dict.update(_get_body_as_dict(request))
  File "/home/alonbl/exodigo/alonbl-misc/aws-example/x1/lib/python3.10/site-packages/botocore/auth.py", line 96, in _get_body_as_dict
    data = json.loads(data.decode('utf-8'))
  File "/usr/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Sheethal00 commented 1 year ago

@swetashre I am facing the same problem with client method "put_object_tagging". Could u please look into this issue

Nilaik commented 5 months ago

This stackoverflow article gave me the curcial hint: Link

At least for complete_multipart_upload the Parts are send in the body as xml data and therefore should not be part of the parameters to be signed.