BUG -psycopg error - Githubissues

Description of the issue

We get import error of psycopg on AWS lambda. I initially thought it was due to imap-data-access version0.10.0. Initial debugging pointed to that because I tried creating new lambda layer with imap-data-access version 0.7.0 through 0.10.0 and deploy the CDK each time with new layer. All versions worked besides version 0.10.0. Therefore, I thought that was the issue.

Then to test why version 0.10.0 was causing the issue, I created a local working branch. I did not make any change to imap-data-access beside updating version to 0.10.1rc2 on it. Then I tried creating package locally using this command python -m build --sdist and then copying the zip file to sds-data-manager/lambda_layer/python and update lambda_layer_construct.py to pip install that independently like this

        code_bundle = lambda_.Code.from_asset(
            layer_dependencies_dir,
            bundling=cdk.BundlingOptions(
                image=runtime.bundling_image,
                command=[
                    "bash",
                    "-c",
                    (
                        "pip install -r requirements.txt -t /asset-output/python && "
                        "pip install imap_data_access-0.10.1rc2.tar.gz -t /asset-output/python && "
                        "cp -au . /asset-output/python"
                    ),
                ],
            ),
        )

        super().__init__(
            id=f"{id}-Layer", code=code_bundle, compatible_runtimes=[runtime], **kwargs
        )

When I deployed with above changes through CDK, the issue went away on AWS lambda.

But when everything is deployed through requirements.txt and through GitHub deploy action, it raises this issue on AWS lambda. I am not sure now what the issue is.

Steps to reproduce the issue

Deploy CDK to AWS through GitHub Actions or locally from this snapshot of sds-data-managercommit

Expected behavior (What should happen)

No error

Actual behavior (What does happen)

{ "errorMessage": "No module named 'psycopg2._psycopg'", "errorType": "ModuleNotFoundError", "requestId": "150f8d02-8a1c-41c2-bd29-11a9f79778ec", "stackTrace": [ " File \"/var/task/SDSCode/create_schema.py\", line 29, in lambda_handler\n Base.metadata.create_all(db.get_engine())\n", " File \"/var/task/SDSCode/database/database.py\", line 27, in get_engine\n return create_engine(db_uri)\n", " File \"\", line 2, in create_engine\n", " File \"/opt/python/sqlalchemy/util/deprecations.py\", line 281, in warned\n return fn(*args, kwargs) # type: ignore[no-any-return]\n", " File \"/opt/python/sqlalchemy/engine/create.py\", line 599, in create_engine\n dbapi = dbapi_meth(dbapi_args)\n", " File \"/opt/python/sqlalchemy/dialects/postgresql/psycopg2.py\", line 690, in import_dbapi\n import psycopg2\n", " File \"/opt/python/psycopg2/init.py\", line 51, in \n from psycopg2._psycopg import ( # noqa\n" ] }

Code Snippet:

Code

Additional notes, affected areas, and suggested fixes

Files and resources that caused this issue could be these:

imap-data-access - version 0.10.0
mismatch of psycopg = "^3.1" and psycopg2-binary = "^2.9.9" in pyproject.toml file
GitHub workflow yaml not installing dependency for psycopg

IMAP-Science-Operations-Center / sds-data-manager

BUG -psycopg error #364