1.2.2 build seems to fail with Python 3.8 and AWS SAM local lambda

[ewenmcneill]

Thank you for maintaining aws-psycopg2; it's been very useful not to have to build the psycopg2 binary for AWS myself when getting started.

It looks like 1.2.2 got released to PyPI in about the last 24 hours, after a series of updates to this repo this week.

Unfortunately at least on Python 3.8 on AWS SAM's "local lambda" support it seems not to be working as expected immediately (I was using Python 3.8 as it's the last version suported by aws-psycopg2 before; see #3 that someone else created).

After working fine for months, my tests suddenly fail with:

[ERROR] Runtime.ImportModuleError: Unable to import module 'app': libpq-a407f694.so.5.12: cannot open shared object file: No such file or directory

although that specific .so does seem to be in the binary .whl package:

ewen@basadi:/tmp$ unzip -v aws_psycopg2-1.2.2-py3-none-any.whl | grep libpq-a407f694.so.5.12
  345280  Stored   345280   0% 06-22-2022 19:34 57701d4a  psycopg2/libpq-a407f694.so.5.12
ewen@basadi:/tmp$ 

so I'm currently unsure why it's not being found, although I notice that in the previous .whl package (1.2.1) the libraries were in a different location:

ewen@basadi:/tmp$ unzip -v aws_psycopg2-1.2.1-py3-none-any.whl| grep libpq                                                                                           
  268888  Defl:N   115095  57% 01-01-1980 00:00 60c2591c  psycopg2/.dylibs/libpq.5.11.dylib
  329164  Defl:N   121725  63% 10-19-2019 20:56 14c92234  psycopg2/.libs/libpq-2be1e2df.so.5.11
  341408  Defl:N   121961  64% 10-20-2019 00:57 b17482a2  psycopg2/.libs/libpq-3d107b13.so.5.11
ewen@basadi:/tmp$ 

so maybe the .so location in the built binary .whl isn't as intended, or maybe there's some change to a search path required?

(For now I'm going to pin on 1.2.1; I was leaving it open previously during development in the hope a Python 3.9 compatible version would emerge, and I'm guessing we're now close.)

Ewen

[ewenmcneill]

FTR, it looks like the tests on GitHub are also failing with a very similar error message:

https://github.com/AbhimanyuHK/aws-psycopg2/runs/7011699582?check_suite_focus=true

and it seems like it might affect Python 3.9:

https://github.com/AbhimanyuHK/aws-psycopg2/runs/7011699710?check_suite_focus=true

and Python 3.10 too:

https://github.com/AbhimanyuHK/aws-psycopg2/runs/7011699924?check_suite_focus=true

(At this point Python 3.8 is probably the only one anyone is relying on; it's the newest Python suported by 1.2.1, and the older Python's are pretty much unsupported now.)

Ewen

[ncgl-syngenta]

Noticing the same behavior in our stacks

[mbustamanteo]

I'm having the exact same issue as described

[AbhimanyuHK]

Hi Guys.

Here is the latest build https://pypi.org/project/aws-psycopg2/1.3.8/

Many Thanks for waiting

[ewenmcneill]

Here is the latest build https://pypi.org/project/aws-psycopg2/1.3.8/

Thank you very much.

I've changed my pin from:

aws-psycopg2==1.2.1

to:

aws-psycopg2==1.3.8

and rebuilt my AWS SAM project. It seems to have picked up 1.3.8 okay, and the local test suite (AWS SAM local-lambda) is running successfully, so I think it's finding the Python 3.8 libraries properly this time.

I'll do some more testing locally, and if it works out, might try bumping the Python version to 3.9 and pushing it up into our staging AWS account for some more testing.

Thanks again for all your work on getting the aws-ready pre-built binary libraries for psycopg2.

Ewen