search

boto / botocore

The low-level, core functionality of boto3 and the AWS CLI.
Apache License 2.0
1.5k stars 1.09k forks source link

StubResponseError on add_client_error with Stubber #2561

Open nicolalandro opened 3 years ago

nicolalandro commented 3 years ago

Describe the bug Using the stubber, addind a client error it generate a "StubResponseError".

Versions:

Steps to reproduce With this code you can reproduce:

import boto3
from botocore.exceptions import ClientError
from botocore.stub import Stubber`

client = boto3.client('s3')
stubber = Stubber(client)
stubber.add_client_error('download_file')

with stubber:
    try:
        r = client.download_file('xxx', 'xxx', 'xxx')
    except ClientError as c:
        print(c)

Expected behavior I expect that the code go into the exception handle

Debug logs Full stack trace:

Traceback (most recent call last):
  File "/media/mint/Barracuda/Digitiamo/aiknowyou-ai-clusterizer/src/boto_error.py", line 11, in <module>
    r = client.download_file('xxx', 'xxx', 'xxx')
  File "/media/mint/Barracuda/Digitiamo/aiknowyou-ai-clusterizer/venv/lib/python3.8/site-packages/boto3/s3/inject.py", line 171, in download_file
    return transfer.download_file(
  File "/media/mint/Barracuda/Digitiamo/aiknowyou-ai-clusterizer/venv/lib/python3.8/site-packages/boto3/s3/transfer.py", line 307, in download_file
    future.result()
  File "/media/mint/Barracuda/Digitiamo/aiknowyou-ai-clusterizer/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 106, in result
    return self._coordinator.result()
  File "/media/mint/Barracuda/Digitiamo/aiknowyou-ai-clusterizer/venv/lib/python3.8/site-packages/s3transfer/futures.py", line 265, in result
    raise self._exception
  File "/media/mint/Barracuda/Digitiamo/aiknowyou-ai-clusterizer/venv/lib/python3.8/site-packages/s3transfer/tasks.py", line 255, in _main
    self._submit(transfer_future=transfer_future, **kwargs)
  File "/media/mint/Barracuda/Digitiamo/aiknowyou-ai-clusterizer/venv/lib/python3.8/site-packages/s3transfer/download.py", line 340, in _submit
    response = client.head_object(
  File "/media/mint/Barracuda/Digitiamo/aiknowyou-ai-clusterizer/venv/lib/python3.8/site-packages/botocore/client.py", line 388, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/media/mint/Barracuda/Digitiamo/aiknowyou-ai-clusterizer/venv/lib/python3.8/site-packages/botocore/client.py", line 680, in _make_api_call
    request_dict = self._convert_to_request_dict(
  File "/media/mint/Barracuda/Digitiamo/aiknowyou-ai-clusterizer/venv/lib/python3.8/site-packages/botocore/client.py", line 726, in _convert_to_request_dict
    api_params = self._emit_api_params(
  File "/media/mint/Barracuda/Digitiamo/aiknowyou-ai-clusterizer/venv/lib/python3.8/site-packages/botocore/client.py", line 755, in _emit_api_params
    self.meta.events.emit(
  File "/media/mint/Barracuda/Digitiamo/aiknowyou-ai-clusterizer/venv/lib/python3.8/site-packages/botocore/hooks.py", line 357, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/media/mint/Barracuda/Digitiamo/aiknowyou-ai-clusterizer/venv/lib/python3.8/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/media/mint/Barracuda/Digitiamo/aiknowyou-ai-clusterizer/venv/lib/python3.8/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/media/mint/Barracuda/Digitiamo/aiknowyou-ai-clusterizer/venv/lib/python3.8/site-packages/botocore/stub.py", line 350, in _assert_expected_params
    self._assert_expected_call_order(model, params)
  File "/media/mint/Barracuda/Digitiamo/aiknowyou-ai-clusterizer/venv/lib/python3.8/site-packages/botocore/stub.py", line 338, in _assert_expected_call_order
    raise StubResponseError(
botocore.exceptions.StubResponseError: Error getting response stub for operation HeadObject: Operation mismatch: found response for None.

Process finished with exit code 1
stobrien89 commented 3 years ago

Hi @nicolalandro,

Sorry to hear you're having issues. Because download_file is not a native s3 operation (it's part of s3Transfer), I'm not sure if there's any way around the operation mismatch because it uses several native S3 operations to calculate the object size, download the file, etc. I'll check with the team to see if there's a way to accomplish this.

nicolalandro commented 3 years ago

Thank you, I will wait the fix, for now I fix with manual mock that raise exception, to have official version will simplify the code and make it more real.

# download do not return error
s3mock_cli.download_file = lambda x, y, z: True

error = mock.Mock()
error.get = lambda x, y: 200 if x == 'Code' else 'text'

response = mock.Mock()
response.get = lambda x, y: error
response.__contains__ = lambda x, y: False

# upload return error
s3mock_cli.upload_file = lambda x, y, z: (_ for _ in ()).throw(ClientError(response, 'name'))

I'm waiting news.

stobrien89 commented 2 years ago

Hi @nicolalandro,

I was able to review this with the team earlier today and it's currently not possible to work around this using the botocore stubber due to the reasons I mentioned before— We will have to write a customization for the S3Transfer methods (upload_file, download_file, etc.), as they are customizations over s3 APIs themselves. This could be documented better as well. Thanks for pointing this out!