Unexpected error on zappa update - ImproperlyConfigured: Error loading psycopg2 or psycopg module on AWS

spirovskib commented 1 year ago

Context

I was pushing a minor code update to a zappa managed Django site. The site was operating normally until the update. On the update I got the following error

Waiting for lambda function [goya-core-production] to be updated...
Error: Warning! Status check on the deployed lambda failed. A GET request to '/' yielded a 502 response code.

When running zappa tail production i found the following error:

[1690488128872] [ERROR] ImproperlyConfigured: Error loading psycopg2 or psycopg module
File "/var/lang/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/var/task/django/db/backends/postgresql/base.py", line 29, in <module>
    raise ImproperlyConfigured("Error loading psycopg2 or psycopg module")

During the preparation of the deployment i don't see that psycopg2-binary is loaded from manylinux wheel. The only change i've made was updating to MacOS Ventura 13.5 yesterday. But I'm using homebrew python.

Expected Behavior

zappa update production should properly deploy the application and run it, since no library changes were made

Actual Behavior

Deployment results in an error ImproperlyConfigured: Error loading psycopg2 or psycopg module

Possible Fix

No idea

Steps to Reproduce

Can't be sure that this reproduces, but this is my setup: Have a lambda django application configured to connect to postgress database backend On MacOS Ventura 13.5 have a Homebrew Python 3.8 install, running in virtual environment for the code try a zappa update of the running code.

Your Environment

Zappa version used: zappa -v 0.57.0
Operating System and Python version:
```
MacOS Ventura 13.5, Macbook M1 CPU
```

Python 3.8.17 (default, Jun 21 2023, 05:28:43) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin

Homebrew Python install, running in virtual environment for this code

* The output of `pip freeze`:

alt-fake-useragent==0.2.1 argcomplete==2.0.0 asgiref==3.6.0 backports.zoneinfo==0.2.1 beautifulsoup4==4.12.2 bleach==5.0.0 blurhash==1.1.4 boto3==1.26.114 botocore==1.29.114 build==0.8.0 certifi==2022.12.7 cfn-flip==1.3.0 charset-normalizer==2.1.0 click==8.1.3 configparser==5.3.0 courlan==0.7.1 crispy-bootstrap5==0.7 dateparser==1.1.8 decorator==5.1.1 Django==4.2.3 django-bleach==3.0.1 django-bootstrap-v5==1.0.11 django-ckeditor==6.5.1 django-cleanup==7.0.0 django-countries==7.5.1 django-crispy-forms==2.0 django-js-asset==2.0.0 django-meta==2.2.0 django-ranged-response==0.2.0 django-simple-captcha==0.5.17 django-storages==1.13.2 django-taggit==3.1.0 djangorestframework==3.14.0 dnspython==2.3.0 docutils==0.19 durationpy==0.5 email-validator==2.0.0.post2 feedparser==6.0.10 flake8==6.0.0 fpdf==1.7.2 future==0.18.3 fuzzywuzzy==0.18.0 git-filter-repo==2.38.0 hjson==3.1.0 htmldate==1.3.0 idna==3.4 jarowinkler==1.2.3 jmespath==1.0.1 joblib==1.2.0 jusText==3.0.0 kappa==0.6.0 langcodes==3.3.0 Levenshtein==0.20.1 lxml==4.9.1 markdownify==0.11.6 MarkupSafe==2.1.2 Mastodon.py==1.8.1 mccabe==0.7.0 mixpanel==4.10.0 nltk==3.7 oauthlib==3.2.2 packaging==21.3 pep517==0.13.0 Pillow==9.5.0 pip-tools==6.13.0 placebo==0.9.0 psycopg2-binary==2.9.4 pycodestyle==2.10.0 pyflakes==3.0.1 pyparsing==3.0.9 python-dateutil==2.8.2 python-dotenv==1.0.0 python-magic==0.4.27 python-slugify==8.0.1 pytz==2023.3 pytz-deprecation-shim==0.1.0.post0 PyYAML==6.0.1 rapidfuzz==2.15.1 regex==2023.3.23 requests==2.28.2 requests-oauthlib==1.3.1 s3transfer==0.6.0 sgmllib3k==1.0.0 six==1.16.0 slack-sdk==3.21.1 smart-open==6.3.0 soupsieve==2.3.2.post1 sqlparse==0.4.4 text-unidecode==1.3 tinycss2==1.2.1 tld==0.13 toml==0.10.2 tomli==2.0.1 tqdm==4.64.0 troposphere==4.0.2 tweepy==4.14.0 typing_extensions==4.5.0 tzdata==2023.3 tzlocal==4.3 urllib3==1.26.15 webencodings==0.5.1 Werkzeug==2.2.3 wsgi-request-logger==0.4.6 zappa==0.57.0

* Link to your project (optional): closed source, but the site should be visible here `https://www.beyondmachines.net/`
* Your `zappa_settings.json`:

{ "production": { "django_settings": "goya_core.settings", "aws_region": "eu-central-1", "profile_name": "_profile_nameredacted", "project_name": "goya-core", "runtime": "python3.8", "s3_bucket": "goya-zappa-files", "exclude": ["local_data_store",".git",".md","CODEOWNERS","github_key.txt", "secret_key.txt", ".sqlite3", "posgtres_env.json","static_assets","media_assets","static_data",".code-workspace","Dockerfile","docker-compose.yml","local_data_store","zappashell.sh"], "timeout_seconds": 600, "manage_roles": false, "role_name": "ZappaRuntimeRole", "vpc_config": { "SubnetIds": [ "subnet-0d390740cf414cb13", "subnet-0f6e2a7f4a66cfeeb" ], "SecurityGroupIds": [ "sg-03656bdc887694ed8" ] }, "xray_tracing": false, "certificate_arn": "arn:aws:acm:us-east-1:_accountredacted:certificate/_cert_redacted", "domain": ".beyondmachines.net", "debug": false, "log_level": "DEBUG", "events": [{ "function": "communicator.views.social_media_poster", "expression": "cron(1 8-20 ? MON-SUN )" }, { "function": "content.views.generate_weekly_report_content", "expression": "cron(0 16 ? MON )" }, { "function": "content.views.get_daily_events", "expression": "cron(0 10 ? MON-FRI )" } ] } }

spirovskib commented 1 year ago

Update, how I made it work after 4 hours:

deactivate virtualenv
remove the entire virtualenv (the bin and lib folders and pyenv.cfg)
recreate the virtalenv with python 3.9 in the same folder virtualenv -p python3.9 .
navigate to the folder with requirements.txt and install them without cache pip install -r requirements.txt --no-cache-dir
test whether Django will run locally python manage.py runserver

update the deployment zappa update production On this iteration the loading of libraries includes psycopg2-binary==2.9.4: Using locally cached manylinux wheel (quoting the second run of the update so there is cache):

Downloading and installing dependencies..
- levenshtein==0.20.1: Downloading
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.43M/1.43M [00:00<00:00, 3.69MB/s]
- markupsafe==2.1.2: Downloading
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 25.5k/25.5k [00:00<00:00, 3.82MB/s]
- pillow==9.5.0: Downloading
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3.31M/3.31M [00:01<00:00, 3.29MB/s]
- pyyaml==6.0.1: Downloading
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 739k/739k [00:00<00:00, 3.41MB/s]
- jarowinkler==1.2.3: Using locally cached manylinux wheel
- lxml==4.9.1: Using locally cached manylinux wheel
- psycopg2-binary==2.9.4: Using locally cached manylinux wheel
- rapidfuzz==2.15.1: Using locally cached manylinux wheel
- regex==2023.3.23: Using locally cached manylinux wheel

spirovskib commented 1 year ago

I was reviewing my actions last evening, and although I've managed to recover, it wasn't because I discovered the root cause. I'm reopening the issue for wider consideration, maybe some extra debugging is possible for everyone to benefit if this event happens again.

monkut commented 1 year ago

I assume that this problem occurred due to the latest version of psycopg2-binary not having a manylinux wheel available for the desired python runtime. If that's the case, I recommend pinning the version of psycopg2-binary to the latest version where a manylinux wheel is available.

spirovskib commented 1 year ago

This is a very good insight. Is it possible to add a check and fail the update command if the manylinux wheel is not available and architecture of the machine is not x64?

monkut commented 1 year ago

That's a good point.
It would be better to abort the update when the associated library is not able to be installed.

At the moment the lambda package is built on the system where the command is run. I suspect there could be a check during the package build process to determine if a package has a wheel or not.
Although it might be hard to determine which libraries would need a 'wheel' (typically those that require to be compiled/built) and those that don't.

In addition, we do allow deployment to docker, so that would need to be taken into consideration. (Since a docker deployment could potentially support the case where the expected 'wheel' is not available.

RadySonabu commented 7 months ago

When I pin this packages, it works on my end. psycopg==3.1.10 psycopg-binary==3.1.10 psycopg2-binary==2.9.7 zappa==0.57.0

github-actions[bot] commented 4 months ago

Hi there! Unfortunately, this Issue has not seen any activity for at least 90 days. If the Issue is still relevant to the latest version of Zappa, please comment within the next 10 days if you wish to keep it open. Otherwise, it will be automatically closed.

github-actions[bot] commented 4 months ago

Hi there! Unfortunately, this Issue was automatically closed as it had not seen any activity in at least 100 days. If the Issue is still relevant to the latest version of Zappa, please open a new Issue.

zappa / Zappa