boto library mismatch

[kennovak-zy]

Context

A slim_handler zappa fails at instantiation due to a missing name from boto. This seems to be new as of February 2019, and is related to this boto bug of 8 Feb 2019: https://github.com/boto/boto3/issues/1867

Expected Behavior

The lambda package does not contain boto3 or botocore -- they are excluded in the zappa settings. Normally all dependencies on these resolve when the lambda is instantiated and the slim_handler zip file is expanded

Actual Behavior

With debug on during lambda instantiation, there are about 20 lines of output including [DEBUG] .. Loading JSON file: /var/runtime/botocore/data/s3/2006-03-01/service-2.json [DEBUG] .. Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7f155a2842f0> [DEBUG] .. Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x7f155a004a60> Then there's a Traceback ending with File "/var/task/s3transfer/utils.py", line 27, in <module> from botocore.exceptions import ReadTimeoutError ImportError: cannot import name 'ReadTimeoutError'

Possible Fix

Removing boto3 and botocore from the zappa settings exclude list eliminated the problem.

Your Environment

• Zappa version used: 0.47.1
• Operating System and Python version: 3.6

[trollefson]

I see the same behavior as @kennovak-zy when I run pipenv run zappa manage dev "collectstatic --noinput"

Environment

Amazon Linux AMI release 2018.03 Zappa 0.47.1 s3transfer 0.2.0 boto3 1.9.93 (specified in Pipfile.lock, but excluded) botocore 1.12.93 (specified in Pipfile.lock, but excluded)

Stack Trace

cannot import name 'ReadTimeoutError': ImportError Traceback (most recent call last): File "/var/task/handler.py", line 580, in lambda_handler return LambdaHandler.lambda_handler(event, context) File "/var/task/handler.py", line 248, in lambda_handler return handler.handler(event, context) File "/var/task/handler.py", line 413, in handler management.call_command(*event['manage'].split(' ')) File "/var/task/django/core/management/__init__.py", line 148, in call_command return command.execute(*args, **defaults) File "/var/task/django/core/management/base.py", line 353, in execute output = self.handle(*args, **options) File "/var/task/django/contrib/staticfiles/management/commands/collectstatic.py", line 188, in handle collected = self.collect() File "/var/task/django/contrib/staticfiles/management/commands/collectstatic.py", line 114, in collect handler(path, prefixed_path, storage) File "/var/task/django/contrib/staticfiles/management/commands/collectstatic.py", line 343, in copy_file if not self.delete_file(path, prefixed_path, source_storage): File "/var/task/django/contrib/staticfiles/management/commands/collectstatic.py", line 249, in delete_file if self.storage.exists(prefixed_path): File "/var/task/storages/backends/s3boto3.py", line 532, in exists self.connection.meta.client.head_object(Bucket=self.bucket_name, Key=name) File "/var/task/storages/backends/s3boto3.py", line 315, in connection verify=self.verify, File "/var/runtime/boto3/session.py", line 389, in resource aws_session_token=aws_session_token, config=config) File "/var/runtime/boto3/session.py", line 263, in client aws_session_token=aws_session_token, config=config) File "/var/runtime/botocore/session.py", line 885, in create_client client_config=config, api_version=api_version) File "/var/runtime/botocore/client.py", line 70, in create_client cls = self._create_client_class(service_name, service_model) File "/var/runtime/botocore/client.py", line 95, in _create_client_class base_classes=bases) File "/var/runtime/botocore/hooks.py", line 227, in emit return self._emit(event_name, kwargs) File "/var/runtime/botocore/hooks.py", line 210, in _emit response = handler(**kwargs) File "/var/runtime/boto3/utils.py", line 61, in _handler module = import_module(module) File "/var/runtime/boto3/utils.py", line 52, in import_module __import__(name) File "/var/runtime/boto3/s3/inject.py", line 15, in <module> from boto3.s3.transfer import create_transfer_manager File "/var/runtime/boto3/s3/transfer.py", line 129, in <module> from s3transfer.manager import TransferConfig as S3TransferConfig File "/var/task/s3transfer/manager.py", line 21, in <module> from s3transfer.utils import get_callbacks File "/var/task/s3transfer/utils.py", line 27, in <module> from botocore.exceptions import ReadTimeoutError ImportError: cannot import name 'ReadTimeoutError'

Workaround

Pinning the following packages worked for me:

boto3==1.9.80 botocore==1.12.80 s3transfer==0.1.13

[joguSD]

Boto3 dev here, noticed you referenced one of our issues. boto/boto3#1867 isn't a bug in boto3 from what we've seen, it just comes down to invalid package configurations.

Note in the above logs botocore/boto3 are coming from /var/runtime (the lambda runtime provided) because they were excluded but s3transfer is coming from /var/task. This leads to the latest s3transfer which is incompatible with the older versions of botocore/boto3 in the lambda runtime.

[MrBretticus]

Workaround

Pinning the following packages worked for me:

boto3==1.9.80 botocore==1.12.80 s3transfer==0.1.13

This also worked for me