importing boto3 fails (python3.6)

[pokryfka]

To reproduce just

import boto3

somewhere in the code, for example in service.py and deploy for python3.6

Removing concurrent directory from the zip package solves the problem.

Exception details:

s3_client = boto3.client('s3')
File "/var/task/boto3/__init__.py", line 83, in client
return _get_default_session().client(*args, **kwargs)
File "/var/task/boto3/session.py", line 263, in client
aws_session_token=aws_session_token, config=config)
File "/var/task/botocore/session.py", line 836, in create_client
client_config=config, api_version=api_version)
File "/var/task/botocore/client.py", line 65, in create_client
cls = self._create_client_class(service_name, service_model)
File "/var/task/botocore/client.py", line 89, in _create_client_class
base_classes=bases)
File "/var/task/botocore/hooks.py", line 227, in emit
return self._emit(event_name, kwargs)
File "/var/task/botocore/hooks.py", line 210, in _emit
response = handler(**kwargs)
File "/var/task/boto3/utils.py", line 61, in _handler
module = import_module(module)
File "/var/task/boto3/utils.py", line 52, in import_module
__import__(name)
File "/var/task/boto3/s3/inject.py", line 15, in <module>
from boto3.s3.transfer import create_transfer_manager
File "/var/task/boto3/s3/transfer.py", line 127, in <module>
from s3transfer.exceptions import RetriesExceededError as \
File "/var/task/s3transfer/__init__.py", line 134, in <module>
import concurrent.futures
File "/var/task/concurrent/futures/__init__.py", line 8, in <module>
from concurrent.futures._base import (FIRST_COMPLETED,
File "/var/task/concurrent/futures/_base.py", line 357
raise type(self._exception), self._exception, self._traceback
^
SyntaxError: invalid syntax

[dvnrvn]

Yep, I am experiencing the same issue. I added concurrent== to the blacklist of packages here:

https://github.com/nficano/python-lambda/blob/master/aws_lambda/aws_lambda.py#L274

However, I am not sure if this should be the best solution.

[cwharland]

Any thought on the proper way to fix this problem? Why is concurrent installed in the first place?

[galindro]

@nficano it seems that this fix wasn't included in version 3.0.0. The error continues ocurring using version 3.0.0.

[galindro]

Compiling from sources it works...

[loren]

I'm running into this, too, but I'm not sure what the workaround is exactly. Can somebody please jot down the steps they took?

[jonahjon]

The fix for this is to change either

1. Use pip3 install for lambda runtimes of 3.X
2. Delete the folder in your packaged concurrent/futures/

[YajJackson]

@jonahjones777 could you explain how you're going about deleting concurrent/futures/? We've tried switching to pip3 and that didn't seem to resolve the issue. still failing due to syntax error on s3_resource = boto3.resource("s3") We're running on python 3.7

[ERROR] SyntaxError: invalid syntax (_base.py, line 414)
Traceback (most recent call last):
  File "/var/task/function.py", line 156, in lambda_handler
    s3_resource = boto3.resource("s3")
  File "/var/task/boto3/__init__.py", line 100, in resource
    return _get_default_session().resource(*args, **kwargs)
  File "/var/task/boto3/session.py", line 389, in resource
    aws_session_token=aws_session_token, config=config)
  File "/var/task/boto3/session.py", line 263, in client
    aws_session_token=aws_session_token, config=config)
  File "/var/task/botocore/session.py", line 839, in create_client
    client_config=config, api_version=api_version)
  File "/var/task/botocore/client.py", line 80, in create_client
    cls = self._create_client_class(service_name, service_model)
  File "/var/task/botocore/client.py", line 110, in _create_client_class
    base_classes=bases)
  File "/var/task/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/var/task/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/var/task/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/var/task/boto3/utils.py", line 61, in _handler
    module = import_module(module)
  File "/var/task/boto3/utils.py", line 52, in import_module
    __import__(name)
  File "/var/task/boto3/s3/inject.py", line 15, in <module>
    from boto3.s3.transfer import create_transfer_manager
  File "/var/task/boto3/s3/transfer.py", line 127, in <module>
    from s3transfer.exceptions import RetriesExceededError as \
  File "/var/task/s3transfer/__init__.py", line 134, in <module>
    import concurrent.futures
  File "/var/task/concurrent/futures/__init__.py", line 8, in <module>
    from concurrent.futures._base import (FIRST_COMPLETED,

[jonahjon]

@YajJackson

pip3 uninstall futures

or

Go into your folder where python packages are installed and delete the folder concurrent/futures/

This is a backport of futures installed due to a dependency in another library, for me I had included boto and boto3 in my setup.py, and it was a dependency from boto

[YajJackson]

@jonahjones777 thanks for the reply!

We've already reverted to python 2.7 to get around this issue. I think it's definitely worth noting that any of these workarounds are insufficient, and Lambda shouldn't consider itself supporting 3.X until this boto3 issue is resolved. 🤷‍♂

[thedouglenz]

@jonahjones777 If you were to use the boto and boto3 packages that are already in the py 3.7 lambda environment on AWS without packaging it and shipping it along with your function, how would you then avoid this issue? It seems that perhaps by fetching those dependencies, deleting concurrent and zipping that and sending it maybe you're able to avoid it but in the 3.7 environment, there already exists a /var/task/concurrent/ lib that has this problem.

[jonahjon]

@thedouglenz I did the exact steps mentioned where I ran pip3 install, and ran rm -rf concurrent/futures and my lambda worked even with boto and boto3 both installed.

@YajJackson I think there is a few passable workarounds for this issue, Maybe try pipdeptree and see if any other packages are installing concurrent as a dependency. My guess is they are installing a python 2.7 version of concurrent as dependency for a package. Since it's part of the standard library for 2.7 and 3.7 the installed one can be deleted since your default runtime should have the required version needed, and running concurrent 2.7 in python 3.7 is likely leading the issue.