[Migrated] Don't exclude *.egg-info from lambda package

jneves commented 3 years ago

Originally from: https://github.com/Miserlou/Zappa/issues/1602 by paulina-mudano

Context

Excluding *.egg-info from the package deployed to lambda, when slim_handler: true causes random "DistributionNotFound" errors for packages that do not have dist-info but have egg-info.

Related issues:

I had exactly the problem described in https://github.com/Miserlou/Zappa/issues/1555 and after investigation it turned out that pkg_resources ignores packages that do not have dist-info or egg-info directories. After removing the *.egg-info' from ZIP_EXCLUDES in Zappa core.py the code executes without issues!

Expected Behavior

All required files from venv should be included in the lambda package

Actual Behavior

.egg-info directories are missing which is causing DistributionNotFound errors.

Possible Fix

Remove '*.egg-info' from ZIP_EXCLUDES.

Steps to Reproduce

docker run -it amazonlinux:1 bash
export LANG=en_US.utf8
yum install -y python36-pip python36-devel vim
pip-3.6 install zappa awscli

ln -fs /usr/bin/python3 /usr/bin/python
ln -fs /usr/bin/pip-3.6 /usr/bin/pip

mkdir /tmp/myproject && cd /tmp/myproject
python -m venv .env
. .env/bin/activate
pip install sagemaker

export AWS_ACCESS_KEY_ID=xxx
export AWS_SECRET_ACCESS_KEY=xxx
export AWS_DEFAULT_REGION=eu-west-1

Create lambda_function.py with:
```
import sagemaker
```

def lambda_handler(): print("Hello")

12. Create `zappa_settings.json` (see below)
13. `zappa deploy dev`
14. Execute Lambda in AWS and get this error:

The 'sagemaker' distribution was not found and is required by the application: DistributionNotFound Traceback (most recent call last): File "/var/task/handler.py", line 567, in lambda_handler return LambdaHandler.lambda_handler(event, context) File "/var/task/handler.py", line 237, in lambda_handler handler = cls() File "/var/task/handler.py", line 129, in init self.app_module = importlib.import_module(self.settings.APP_MODULE) File "/var/lang/lib/python3.6/importlib/init.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 978, in _gcd_import File "", line 961, in _find_and_load File "", line 950, in _find_and_load_unlocked File "", line 655, in _load_unlocked File "", line 678, in exec_module File "", line 205, in _call_with_frames_removed File "/tmp/test-zappa-egginfo/lambda_function.py", line 1, in import sagemaker File "/tmp/test-zappa-egginfo/sagemaker/init.py", line 15, in from sagemaker import estimator # noqa: F401 File "/tmp/test-zappa-egginfo/sagemaker/estimator.py", line 23, in from sagemaker.analytics import TrainingJobAnalytics File "/tmp/test-zappa-egginfo/sagemaker/analytics.py", line 22, in from sagemaker.session import Session File "/tmp/test-zappa-egginfo/sagemaker/session.py", line 28, in from sagemaker.user_agent import prepend_user_agent File "/tmp/test-zappa-egginfo/sagemaker/user_agent.py", line 22, in SDK_VERSION = pkg_resources.require('sagemaker')[0].version File "/tmp/test-zappa-egginfo/pkg_resources/init.py", line 892, in require needed = self.resolve(parse_requirements(requirements)) File "/tmp/test-zappa-egginfo/pkg_resources/init.py", line 778, in resolve raise DistributionNotFound(req, requirers) pkg_resources.DistributionNotFound: The 'sagemaker' distribution was not found and is required by the application

15. Modify `/usr/local/lib/python3.6/site-packages/zappa/core.py`: remove `'*.egg-info'` from `ZIP_EXCLUDES` in line 201
16. `zappa update dev`
17. Execute Lambda again, get successful execution.

## Your Environment
<!--- Include as many relevant details about the environment you experienced the bug in -->
* Zappa version used: 0.46.2
* Operating System and Python version: Amazon Linux, Python 3.6.5. 
Similar issue happens on MacOS 10.12.6 with Python 3.6.5 for other packages.
* The output of `pip freeze`:

boto3==1.8.6 botocore==1.11.6 docutils==0.14 jmespath==0.9.3 numpy==1.15.1 protobuf==3.6.1 protobuf3-to-dict==0.1.5 python-dateutil==2.7.3 PyYAML==3.13 s3transfer==0.1.13 sagemaker==1.9.2 scipy==1.1.0 six==1.11.0 urllib3==1.23

* Link to your project (optional): N/A
* Your `zappa_settings.py`:

{ "dev": { "app_function": "lambda_function.lambda_handler", "aws_region": "eu-west-1", "project_name": "test-zappa-egginfo", "runtime": "python3.6", "s3_bucket": "some-bucket-here", "manage_roles": false, "role_arn": "role-arn-here", "keep_warm": false, "slim_handler": true, "log_level": "DEBUG", "debug": true, "touch": false, "num_retained_versions": 1 } }

github-actions[bot] commented 8 months ago

Hi there! Unfortunately, this Issue has not seen any activity for at least 90 days. If the Issue is still relevant to the latest version of Zappa, please comment within the next 10 days if you wish to keep it open. Otherwise, it will be automatically closed.

github-actions[bot] commented 7 months ago

Hi there! Unfortunately, this Issue was automatically closed as it had not seen any activity in at least 100 days. If the Issue is still relevant to the latest version of Zappa, please open a new Issue.

zappa / Zappa