Unable to download file from s3 in integration test

Summary:

This is a new issue we are experiencing after upgrading pytest-localstack to 0.2.0 and boto3 to 1.9.115. A test that was previously working with pytest-localstack==0.1.5 & boto3==1.9.86 is now failing on a boto3.client('s3').download_file() with the error botocore.exceptions.ClientError: An error occurred (416) when calling the GetObject operation: Requested Range Not Satisfiable.

Versions:

Pipenv:

boto3==1.9.115
  - botocore [required: >=1.12.115,<1.13.0, installed: 1.12.115]
  - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.4]
  - s3transfer [required: >=0.2.0,<0.3.0, installed: 0.2.0]
pytest-cov==2.6.1
  - pytest [required: >=3.6, installed: 4.3.1]
pytest-localstack==0.2.0
  - botocore [required: >=1.4.31,!=1.4.45, installed: 1.12.115]
  - docker [required: Any, installed: 3.7.0]
  - pytest [required: >=3.3.0, installed: 4.3.1]
  - six [required: Any, installed: 1.12.0]

Docker: `Version 2.0.0.3 (31259)`

Python: `Python 3.7.2`

Problem:

In one of our integration tests. We are trying to test downloading the correct file from S3. Here is the setup:

import boto3
import pytest_localstack

s3_client = boto3.client('s3')

@pytest.fixture(scope='module', autouse=True)
def setup_test_bucket():
    s3_client.create_bucket(Bucket='test_deployment_bucket')

    s3_client.upload_file(
        Bucket='test_deployment_bucket',
        Key='test_file_key',
        Filename='/tmp/test_file',
    )

The code we are trying to test is:

import os

import boto3

s3_client = boto3.client('s3')

def download_file(download_path):
    if not os.path.isfile(download_path):
        s3_client.download_file(
            Bucket='test_deployment_bucket',
            Key='test_file_key',
            Filename=download_path,
        )

But botocore is raising an exception: botocore.exceptions.ClientError: An error occurred (416) when calling the GetObject operation: Requested Range Not Satisfiable

The file is 98.6 MB.

Stacktrace:

main.py:12: in download_file
    Filename=download_path,
.venv/lib/python3.7/site-packages/boto3/s3/inject.py:172: in download_file
    extra_args=ExtraArgs, callback=Callback)
.venv/lib/python3.7/site-packages/boto3/s3/transfer.py:307: in download_file
    future.result()
.venv/lib/python3.7/site-packages/s3transfer/futures.py:106: in result
    return self._coordinator.result()
.venv/lib/python3.7/site-packages/s3transfer/futures.py:265: in result
    raise self._exception
.venv/lib/python3.7/site-packages/s3transfer/tasks.py:126: in __call__
    return self._execute_main(kwargs)
.venv/lib/python3.7/site-packages/s3transfer/tasks.py:150: in _execute_main
    return_value = self._main(**kwargs)
.venv/lib/python3.7/site-packages/s3transfer/download.py:513: in _main
    Bucket=bucket, Key=key, **extra_args)
.venv/lib/python3.7/site-packages/botocore/client.py:357: in _api_call
    return self._make_api_call(operation_name, kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <botocore.client.S3 object at 0x7f83944a9cc0>
operation_name = 'GetObject'
api_params = {'Bucket': 'test_deployment_bucket', 'Key': 'test_file_key, 'Range': 'bytes=41943040-50331647'}

    def _make_api_call(self, operation_name, api_params):
        operation_model = self._service_model.operation_model(operation_name)
        service_name = self._service_model.service_name
        history_recorder.record('API_CALL', {
            'service': service_name,
            'operation': operation_name,
            'params': api_params,
        })
        if operation_model.deprecated:
            logger.debug('Warning: %s.%s() is deprecated',
                         service_name, operation_name)
        request_context = {
            'client_region': self.meta.region_name,
            'client_config': self.meta.config,
            'has_streaming_input': operation_model.has_streaming_input,
            'auth_type': operation_model.auth_type,
        }
        request_dict = self._convert_to_request_dict(
            api_params, operation_model, context=request_context)

        service_id = self._service_model.service_id.hyphenize()
        handler, event_response = self.meta.events.emit_until_response(
            'before-call.{service_id}.{operation_name}'.format(
                service_id=service_id,
                operation_name=operation_name),
            model=operation_model, params=request_dict,
            request_signer=self._request_signer, context=request_context)

        if event_response is not None:
            http, parsed_response = event_response
        else:
            http, parsed_response = self._make_request(
                operation_model, request_dict, request_context)

        self.meta.events.emit(
            'after-call.{service_id}.{operation_name}'.format(
                service_id=service_id,
                operation_name=operation_name),
            http_response=http, parsed=parsed_response,
            model=operation_model, context=request_context
        )

        if http.status_code >= 300:
            error_code = parsed_response.get("Error", {}).get("Code")
            error_class = self.exceptions.from_code(error_code)
>           raise error_class(parsed_response, operation_name)
E           botocore.exceptions.ClientError: An error occurred (416) when calling the GetObject operation: Requested Range Not Satisfiable

.venv/lib/python3.7/site-packages/botocore/client.py:661: ClientError

mintel / pytest-localstack