Include boto3 or other modules in the Zip

ferrangrau commented 7 years ago

Context

I'm doing a lambda that need boto3 1.4.8 but actually aws lambda works with the 1.4.7 version, I can solve that including boto3 in the zip file but zappa doesn't allow me to do it.

Possible Fix

At the beginning I try to add boto3 in the "include" parameter of the zappa config file but nothing happens, I modified the function "create_lambda_zip" in core.py to use the "include" param that now is unused and all works well. The problem is that reading carefully the documentation I see that the objective of the parameter "include" of the config file is not this.

Do you think its a good option include a parameter like "include" to can modify the default excluded files or modules?

Your Environment

Zappa version used: 0.45.1
Operating System and Python version: ubuntu 17.10 and python 3.6
The output of pip freeze:
Link to your project (optional):

Your zappa_settings.py:

production:
lambda_handler: run.handler
apigateway_enabled: false
keep_warm: false
aws_region: null
profile_name: sch-gov
project_name: reserved-instan
runtime: python3.6
s3_bucket: reserved-instances-manager
delete_local_zip: False
include:
- boto3
events:
- function: run.handler
  expression: rate(12 hours)

ceefour commented 6 years ago

I agree this could be an issue even in the future.

For now I think you can use this workaround or patch Zappa

Update: I just take a look at my own package and surprisingly it includes boto3 (1.4.8), so I think you can check whether boto3 is really in your virtualenv? And second you can check output file of zappa package (stage)

ceefour commented 6 years ago

Self note: if exclude is set but does not mention boto3* and botocore* it seems Zappa (0.46.1) will package boto3 which will cause errors:

  File "/var/task/app/services/firebase_service.py", line 25, in __init__
  s3 = boto3.resource('s3')
  File "/var/task/boto3/__init__.py", line 100, in resource
  return _get_default_session().resource(*args, **kwargs)
  File "/var/task/boto3/session.py", line 347, in resource
  has_low_level_client)
boto3.exceptions.ResourceNotExistsError: The 's3' resource does not exist.
The available resources are:
 -

It seems that it is mandatory to use AWS Lambda's preinstalled boto3 (i.e. exclude our own boto3), otherwise it won't work.

sanhardik commented 6 years ago

As mentioned by @ceefour , The work around I found was to add exclude field with just some random value

for example, in the zappa settings json file, "exclude": ["*.a"]

That adds the boto3 and botocore library folders.

Thanks @ceefour

gordonje commented 5 years ago

This is still happening in zappa 0.48.2 (right around here, I believe).

I use django on zappa along with django-storages to keep static and media files on s3. For my most recent project, I also felt the need to use exclude to keep a lot of stuff I need for local development out of production. When I tried to invoke collectstatic from the lambda function, I got the ResourceNotExistsError for s3.

Instead of excluding botocore and boto3 (and s3transfer, dateutil, concurrent, I suppose) by default, should these be excluded from the zip package automatically?

Miserlou / Zappa