search

zappa / Zappa

Serverless Python
https://zappa.ws/zappa
MIT License
3.3k stars 363 forks source link

[Migrated] Zappa not installing all dependencies in virtual environment? #644

Closed jneves closed 6 months ago

jneves commented 3 years ago

Originally from: https://github.com/Miserlou/Zappa/issues/1635 by complangorous

I'm trying to spin up a lambda that connects to a database in Snowflake, but am getting a ModuleNotFoundError every time I invoke, despite having installed the packages.

Environment: Ubuntu 160.04, Py3.6, virtualenv Missing Package: snowflake-sqlalchemy

Config

{
   {
    "production": {
        "aws_kms_key_arn": <AWS_KMS_KEY_ARN>,
        "apigateway_enabled": false,
        "slim_handler": true,
        "manage_roles": false,
        "app_function": "snowflake_test.main",
        "aws_region": "us-east-1",
        "profile_name": "default",
        "project_name": "snowflake-test",
        "runtime": "python3.6",
        "s3_bucket": <S3_BUCKET>,
        "role_name": snowflake-test-production-ZappaLambdaExecutionRole,
        "role_arn": <ROLE_ARN>
    }
}

pip freeze

argcomplete==1.9.3
asn1crypto==0.24.0
awscli==1.16.26
azure-common==1.1.16
azure-nspkg==2.0.0
azure-storage==0.36.0
base58==1.0.0
boto3==1.8.9
botocore==1.11.9
certifi==2018.8.24
cffi==1.11.5
cfn-flip==1.0.3
chardet==3.0.4
Click==7.0
colorama==0.3.9
cryptography==2.3.1
docutils==0.14
durationpy==0.5
future==0.16.0
hjson==3.0.1
idna==2.7
ijson==2.3
jmespath==0.9.3
kappa==0.6.0
lambda-packages==0.20.0
numpy==1.15.2
pandas==0.23.4
placebo==0.8.2
pyasn1==0.4.4
pyasn1-modules==0.2.2
pycparser==2.19
pycryptodomex==3.6.6
PyJWT==1.6.4
pyOpenSSL==18.0.0
python-dateutil==2.6.1
python-slugify==1.2.4
pytz==2018.5
PyYAML==3.12
requests==2.19.1
rsa==3.4.2
s3transfer==0.1.13
six==1.11.0
snowflake==0.0.3
snowflake-connector-python==1.6.10
snowflake-sqlalchemy==1.1.2
SQLAlchemy==1.2.12
toml==0.10.0
tqdm==4.19.1
troposphere==2.3.3
Unidecode==1.0.22
urllib3==1.23
Werkzeug==0.14.1
wsgi-request-logger==0.4.6
zappa==0.46.2

Script

import pandas as pd
from sqlalchemy import create_engine

def main():

    engine = create_engine('snowflake://<user_login_name>:<password>@<account_name> 
                        /<database_name>/<schema_name>?warehouse=
                        <warehouse_name>?role=<role_name>')

    df = pd.read_sql("""SELECT * FROM Data LIMIT 100""", engine)
    print(df)

if __name__ == "__main__":
    main()

Expected Behavior

DataFrame prints to screen.

Actual Behavior

No module named 'snowflake.sqlalchemy'; 'snowflake' is not a package: ModuleNotFoundError
Traceback (most recent call last):
  File "/var/task/handler.py", line 567, in lambda_handler
    return LambdaHandler.lambda_handler(event, context)
  File "/var/task/handler.py", line 240, in lambda_handler
    return handler.handler(event, context)
  File "/var/task/handler.py", line 372, in handler
    result = self.run_function(app_function, event, context)
  File "/var/task/handler.py", line 275, in run_function
    result = app_function(event, context) if varargs else app_function()
  File "/tmp/snowflake-test/snowflake_test.py", line 8, in main 
     create_engine('snowflake://<user_login_name>:<password>@<account_name> /<database_name>    /<schema_name>?warehouse=<warehouse_name>?role=<role_name>')
  File "/tmp/snowflake-test/sqlalchemy/engine/__init__.py", line 425, in create_engine
    return strategy.create(*args, **kwargs)
  File "/tmp/snowflake-test/sqlalchemy/engine/strategies.py", line 57, in create
    entrypoint = u._get_entrypoint()
  File "/tmp/snowflake-test/sqlalchemy/engine/url.py", line 156, in _get_entrypoint
    cls = registry.load(name)
  File "/tmp/snowflake-test/sqlalchemy/util/langhelpers.py", line 217, in load
    return impl.load()
  File "/tmp/snowflake-test/pkg_resources/__init__.py", line 2332, in load
    return self.resolve()
  File "/tmp/snowflake-test/pkg_resources/__init__.py", line 2338, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
ModuleNotFoundError: No module named 'snowflake.sqlalchemy'; 'snowflake' is not a package

Has anyone else had this problem, with snowflake or any other package?

mcgodes commented 3 years ago

I'm having this problem as well, but with itsdangerous (and possibly Flask?):

Traceback (most recent call last):
  File "/var/task/handler.py", line 609, in lambda_handler
    return LambdaHandler.lambda_handler(event, context)
  File "/var/task/handler.py", line 240, in lambda_handler
    handler = cls()
  File "/var/task/handler.py", line 134, in __init__
    self.app_module = importlib.import_module(self.settings.APP_MODULE)
  File "/var/lang/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/tmp/mercury-api-dev/server.py", line 4, in <module>
    from flask import Flask, request
  File "/tmp/mercury-api-dev/flask/__init__.py", line 19, in <module>
    from . import json
  File "/tmp/mercury-api-dev/flask/json/__init__.py", line 15, in <module>
    from itsdangerous import json as _json

I've tried packaging and then inspecting the contents of the package locally. Neither Flask nor itsdangerous seem to be included in that package. I've also tried deleting, recreating, and fresh installing virtualenv (named venv). I have confirmed that the application works locally, that the libraries in question are present in venv/lib/python3.7/site-packages/, and match my requirements.txt file. I'm using Python3.7. It's been working up until today and I'm not sure what else to try. That requirements.txt file is as follows:

aiodns==2.0.0 aiosmtpd==1.2.2 appnope==0.1.0 APScheduler==3.0.0 argcomplete==1.12.2 asn1crypto==0.24.0 atomicwrites==1.3.0 atpublic==2.1.2 attrs==19.1.0 backcall==0.1.0 beautifulsoup4==4.9.3 bleach==2.1.3 boto3==1.17.8 botocore==1.20.59 certifi==2019.3.9 cffi==1.12.3 cfn-flip==1.2.3 chardet==3.0.4 click==7.1.2 colorama==0.3.9 coverage==5.5 cryptography==3.4.5 cssselect==1.1.0 cycler==0.10.0 decorator==4.3.0 docopt==0.6.2 durationpy==0.5 entrypoints==0.2.3 enum34==1.1.6 feedfinder2==0.0.4 feedparser==6.0.2 filelock==3.0.12 Flask==1.1.2 future==0.18.2 geographiclib==1.50 googlemaps==4.4.5 greenlet==1.0.0 hjson==3.0.2 html5lib==1.0.1 idna==2.8 importlib-metadata==3.4.0 ipykernel==4.8.2 ipython==6.5.0 ipython-genutils==0.2.0 ipywidgets==7.3.1 isodate==0.6.0 itsdangerous==1.1.0 jedi==0.12.1 jellyfish==0.8.2 jieba3k==0.35.1 Jinja2==2.11.3 jmespath==0.10.0 joblib==0.13.2 jsonschema==2.6.0 jupyter==1.0.0 jupyter-client==5.2.3 jupyter-console==5.2.0 jupyter-core==4.4.0 kappa==0.6.0 kiwisolver==1.0.1 lxml==4.6.2 MarkupSafe==1.1.1 matplotlib==2.2.2 mistune==0.8.3 more-itertools==7.2.0 nbconvert==5.3.1 nbformat==4.4.0 newspaper3k==0.2.8 nltk==3.5 notebook==5.6.0 numpy==1.15.0 packaging==19.1 pandas==0.23.3 pandocfilters==1.4.2 parso==0.3.1 pexpect==4.7.0 pickleshare==0.7.4 Pillow==8.1.2 pip-tools==5.5.0 placebo==0.9.0 pluggy==0.13.0 pprintpp==0.4.0 prometheus-client==0.3.1 prompt-toolkit==1.0.15 psycopg2-binary==2.8.3 ptyprocess==0.6.0 py==1.8.0 pycares==3.1.1 pycountry==20.7.3 pycountry-convert==0.7.2 pycparser==2.19 Pygments==2.2.0 pylodstorage==0.0.26 pyOpenSSL==19.0.0 pyparsing==2.2.0 pytest==5.1.2 pytest-cov==2.11.1 pytest-mock==3.5.1 python-dateutil==2.7.3 python-slugify==4.0.1 pytz==2018.5 pywinpty==0.5.4 PyYAML==5.4.1 pyzmq==17.1.0 qtconsole==4.3.1 rdflib==5.0.0 regex==2021.3.17 repoze.lru==0.7 requests==2.25.1 requests-file==1.5.1 s3transfer==0.3.4 scikit-learn==0.21.1 scipy==1.1.0 seaborn==0.9.0 Send2Trash==1.5.0 sgmllib3k==1.0.0 simple-salesforce==0.74.3 simplegeneric==0.8.1 six==1.11.0 sklearn==0.0 soupsieve==2.2 SPARQLWrapper==1.8.5 SQLAlchemy==1.4.1 terminado==0.8.1 testpath==0.3.1 text-unidecode==1.3 tinysegmenter==0.3 tldextract==3.1.0 toml==0.10.2 tornado==5.1 tqdm==4.56.2 traitlets==4.3.2 troposphere==2.6.3 typing-extensions==3.7.4.3 tzlocal==2.1 urllib3==1.26.4 verify-email==2.4.1 wcwidth==0.1.7 webencodings==0.5.1 Werkzeug==0.16.1 widgetsnbextension==3.3.1 wsgi-request-logger==0.4.6 zappa==0.52.0 zipp==0.6.0

My zappa_settings.json is:

    "dev": {
      "project_name": "<project name>",
      "keep_warm": false,
      "debug": true,
      "log_level": "DEBUG",
      "api_key_required": true,
      "api_key": "<key>",
      "aws_region": "us-east-1",
      "s3_bucket": "<bucket>",
      "app_function": "server.server",
      "http_methods": ["GET"],
      "parameter_depth": 1,
      "memory_size": 512,
      "use_precompiled_packages": true,
      "cors": true,
      "environment_variables": {
          ...
      },
      "slim_handler": true
    }
}
mcgodes commented 3 years ago

Welp...tried one 'last' time and apparently it works now. I made one change to add an exclude to remove some of the larger & unneeded modules from the package. Idk if that fixed everything?

debdutgoswami commented 2 years ago

This issue is very random. I'm not able to set-up a django service through Zappa because it says ModuleNotFound: django.

On the contrary, two days prior I deployed a different application through Zappa and it worked.

github-actions[bot] commented 6 months ago

Hi there! Unfortunately, this Issue has not seen any activity for at least 90 days. If the Issue is still relevant to the latest version of Zappa, please comment within the next 10 days if you wish to keep it open. Otherwise, it will be automatically closed.

github-actions[bot] commented 6 months ago

Hi there! Unfortunately, this Issue was automatically closed as it had not seen any activity in at least 100 days. If the Issue is still relevant to the latest version of Zappa, please open a new Issue.