Lambda Import Error: No module named 'psycopg2._psycopg'

[brentonmallen1]

I tried to both use the compiled binary as well as using the following the manual build instruction but I still get a No module named 'psycopg2._psycopg' error on the lambda function. It seem like there is a _psycopg module that is attempting to be imported here:

https://github.com/jkehler/awslambda-psycopg2/commit/cd0f954a0a3fbe604a546346c9bcbef357313299#diff-1166ada1a9105364bc73bd85f308d8e9R50

but that file doesn't seem to exist in the compiled directory. Am I missing something?

I've also gone ahead and dockerized the build process, I was going to submit it in a PR with some instructions in the readme if that would be useful:

FROM lambci/lambda:build-python3.7

ENV \
 POSTGRES_VER=11.4 \
 PSYCOPG_VER=2.8 \
 PSYCOPG_VER_PATH=PSYCOPG-2-8 \
 PYTHON_VER=3.7

RUN yum install -y wget tar makecache fast automake16

#setup output dir for library extraction
RUN mkdir /var/output

# PSYCOPG2
WORKDIR /var/psycopg

RUN wget -nv https://ftp.postgresql.org/pub/source/v${POSTGRES_VER}/postgresql-${POSTGRES_VER}.tar.gz
RUN tar -zxf postgresql-${POSTGRES_VER}.tar.gz
RUN wget -nv http://initd.org/psycopg/tarballs/${PSYCOPG_VER_PATH}/psycopg2-${PSYCOPG_VER}.tar.gz
RUN tar -zxf psycopg2-${PSYCOPG_VER}.tar.gz

# build postgres
RUN cd postgresql-${POSTGRES_VER} && \
./configure --prefix /var/psycopg/postgresql-${POSTGRES_VER} --without-readline --without-zlib --with-openssl && \
make && \
make install

# build psycopg2
# need to replace some values in the config file so they point to postgres install and enable ssl
RUN cd psycopg2-${PSYCOPG_VER} && \
sed -ie "s/pg_config =/pg_config = \/var\/psycopg\/postgresql-$POSTGRES_VER\/bin\/pg_config/g" setup.cfg && \
sed -i 's/static_libpq = 0/static_libpq = 1/g' setup.cfg && \
sed -i 's/libraries =/libraries = ssl crypto/g' setup.cfg
RUN cat psycopg2-${PSYCOPG_VER}/setup.cfg
RUN ls /var/psycopg/postgresql-${POSTGRES_VER}
RUN cd psycopg2-${PSYCOPG_VER} && python setup.py build

# copy compiled library to output to deliever to host
RUN cp -r /var/psycopg/psycopg2-2.8/build/lib.linux-x86_64-${PYTHON_VER}/psycopg2 /var/output

Also, I'm aware of this issue: https://github.com/jkehler/awslambda-psycopg2/issues/23 but seeing as I'm having the issue on lambda, and the change noted above is recent, I figured I'd make an issue.

[kumprj]

Also experiencing this issue. Tried compiling in python 2.7 and 3.6 (and also python 3.7) using @jkehler 's psycopg2 folders (with the respective rename). Maybe I'm missing something with the pre-requisities but every step works in this tutorial except for Step 5.5 where you invoke the lambda function. aws apigateway get-rest-apis generates an api but it gets an internal server error when trying to curl query it. I do think the issue lies with just this lambda/psycopg2 step.

edit: also tried sudo yum install python-psycopg2 and no luck after zipping that way either. Also tried using an s3 bucket made by my lambda/cloudformation.

[cklyyung]

When you added the psycopg2 folders to the lambda package, were they placed inpython/lib/python3.7/site-packages? I was having also an issue with the No module named error, and I had to change the folder structure to follow what was documented here to fix it.

[brentonmallen1]

I was able to get this to work today. My use case was using this as a lambda layer in a custom directory path. I was initially trying to add the path using sys.path.append() but that didn’t work. Turns out adding the path to the psycopg2 folder using sys.path.insert() worked.

[updated] os.path.append() to sys.path.append()

[kumprj]

@brentonmallen1 can you elaborate a little more on that?

@cklyyung yeah I moved the files there there manually so I don't think that's it :\

[brentonmallen1]

@kumprj certainly.

my use case was to use a lambda layer. To do that, I had to zip up the contents of the psycopg2 directory (after compiling) and submit it to aws.

When a lambda function is invoked, it will unpack that zip file to the /opt directory. The different thing in my case was that I had a parent directory in which the psycopg2 directory was contained. So the unpacked path to the module looks something like:

/opt
  /parent
    /psycopg2

In order for the lambda function to be able to import the module, I had to append the PYTHONPATH environment variable with the path to the module. I originally tried doing that using the sys.path.append('/opt/parent') (I mistyped earlier) method at the beginning of the function script, but that didn't work as it doesn't append the PYTHONPATH per se. In order to get it to work, I had to use the sys.path.insert(0, '/opt/parent') method at the beginning of the function script.

For example:

import sys
sys.path.insert(0, '/opt/parent')
import psycopg2

def lambda_handler():
   print('hello')

I hope that helps, if not let me know and I'll see how I can help.

[kumprj]

serverless-query.zip I'm uploading this zip file. However, I am having trouble understanding the layers element.

[kumprj]

When you added the psycopg2 folders to the lambda package, were they placed inpython/lib/python3.7/site-packages? I was having also an issue with the No module named error, and I had to change the folder structure to follow what was documented here to fix it.

I have no layers at the moment, maybe adding them will help.

[brentonmallen1]

@kumprj if you don't want to use lambda layers, the psycopg2/ directory needs to be in the same directory as your lambda handler script. That way the import is relative to the current directory. So the when the function zip file is unzipped, the structure should look something like:

/
  psycopg2/
  lambda_function.py

If you're going to try to put it in a lambda layer, the zipped library needs to be of the same path structure as what you want it to be on the lambda function. For example, the file structure in the zip folder, when unzipped, should look like this:

python/
  lib/
    python3.7/
      site-packages/
        psycopg2/

Also, ensure that your runtime environment is indeed python3.7 otherwise the directory might not be there.

This path only works because the site-packages path is automatically added to the PYTHONPATH in the lambda function.

[kumprj]

psycopg2 and the lambda function.py are in the same directory. I appreciate your help but I'm clearly off the mark somewhere. I'm open to using the lambda layers but this is my first time using lambda so I'm not sure what I'm doing.

I'm using python 2.7 and I made sure the template was using python2.7 too, so that should all be right.

[brentonmallen1]

It could be as annoying as a typo somewhere. Double check the handler call in the aws console for the lambda function in question and see if it matches the name of your script exactly.

[kumprj]

Now its getting Unable to import module 'serverless-query': libpq.so.5: cannot open shared object file: No such file or directory which I thought this repo was supposed to solve, so I need to check out where my gap is there.

Gave up after 10 hours and ended up switching to this nodejs guide: https://medium.com/@timo.wagner/how-to-build-an-api-endpoint-with-node-js-aws-lambda-and-aws-rds-postgresql-db-part-1-a9766bb0de3

[brentonmallen1]

@kumprj another potential route to take would be taking advantage of Serverless to handle the library packaging: https://twitter.com/tdhopper/status/1159111484925984769?s=20

[oynix]

I did as README.md said but it was not worked and that made me confused. But now, I may be know what problem was.

After you compiled on your own computer, there was a so file in the directory named psycopg2 in psycopg2-2.6.1/build/lib.xxxxxx/.

Python import so library by the file name that called ELF header depend on runtime environment. The name of so file after I compiled on my Mac is psycopg.cpython-37m-darwin.so, it was suitable for MaxOS, but AWS Lambda run on linux.

So, when I deploy my code on AWS Lambda, it always report that No module named 'psycopg2._psycopg'.

Resolution I found was that redo the step 1/2/3/4 in README.md on a Linux machine, after compile I got a so file named _psycopg.cpython-37m-x86_64-linux-gnu.so in directory psycopg2.

When I copied this directory into zip file and redeployed my code to Lambda, it worked.

BTW, psycopg2-2.8.3 is a better choice rather than psycopg2-2.6.1.

[coltonbh]

I can confirm that building using the psycopg2-binary on a Mac system and then uploading that .zip file to lambda results in the missing import. Building on a linux system and uploading the .zip file will solve the missing import problem.

[brentonmallen1]

This is most likely because lambda run on Amazon Linux which is a CentOS variant. My usual course of action is to build the dependencies in an Amazon Linux Docker container and then deploy the package with those dependencies.

[rodrich]

This is most likely because lambda run on Amazon Linux which is a CentOS variant. My usual course of action is to build the dependencies in an Amazon Linux Docker container and then deploy the package with those dependencies.

Yes, you right! Thanks!. I had same issue on my macOS System and I was able to solve it by creating an Amazon Linux EC2 Instance based on Amazon Linux AMI to lambda runtimes. Check below:

https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html https://console.aws.amazon.com/ec2/v2/home#Images:visibility=public-images;search=amzn-ami-hvm-2018.03.0.20181129-x86_64-gp2

After that, I followed great contribution of @jkehler and I downloaded and compiled from scratch: https://github.com/jkehler/awslambda-psycopg2

[kumprj]

This is most likely because lambda run on Amazon Linux which is a CentOS variant. My usual course of action is to build the dependencies in an Amazon Linux Docker container and then deploy the package with those dependencies.

Yes, you right! Thanks!. I had same issue on my macOS System and I was able to solve it by creating an Amazon Linux EC2 Instance based on Amazon Linux AMI to lambda runtimes. Check below:

https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html https://console.aws.amazon.com/ec2/v2/home#Images:visibility=public-images;search=amzn-ami-hvm-2018.03.0.20181129-x86_64-gp2

After that, I followed great contribution of @jkehler and I downloaded and compiled from scratch: https://github.com/jkehler/awslambda-psycopg2

There’s also a pip package for this now: https://pypi.org/project/aws-psycopg2/

[rodrich]

This is most likely because lambda run on Amazon Linux which is a CentOS variant. My usual course of action is to build the dependencies in an Amazon Linux Docker container and then deploy the package with those dependencies.

Yes, you right! Thanks!. I had same issue on my macOS System and I was able to solve it by creating an Amazon Linux EC2 Instance based on Amazon Linux AMI to lambda runtimes. Check below: https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html https://console.aws.amazon.com/ec2/v2/home#Images:visibility=public-images;search=amzn-ami-hvm-2018.03.0.20181129-x86_64-gp2 After that, I followed great contribution of @jkehler and I downloaded and compiled from scratch: https://github.com/jkehler/awslambda-psycopg2

There’s also a pip package for this now: https://pypi.org/project/aws-psycopg2/

In my case, Pip package doesn't work, for that reason I had to use a workaround.

[rodrich]

This is most likely because lambda run on Amazon Linux which is a CentOS variant. My usual course of action is to build the dependencies in an Amazon Linux Docker container and then deploy the package with those dependencies.

Yes, you right! Thanks!. I had same issue on my macOS System and I was able to solve it by creating an Amazon Linux EC2 Instance based on Amazon Linux AMI to lambda runtimes. Check below: https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html https://console.aws.amazon.com/ec2/v2/home#Images:visibility=public-images;search=amzn-ami-hvm-2018.03.0.20181129-x86_64-gp2 After that, I followed great contribution of @jkehler and I downloaded and compiled from scratch: https://github.com/jkehler/awslambda-psycopg2

There’s also a pip package for this now: https://pypi.org/project/aws-psycopg2/

In my case, Pip package doesn't work, for that reason I had to use a workaround. Remember that It just happened at time to create and deploy aws lambda, locally It works fine and no issue.

[viktor-idenfy]

@coltonbh Hello, I face same issue on mac os. I did not understand how can I solve this, here is my question: https://stackoverflow.com/questions/67556906/improperlyconfigured-error-loading-psycopg2-module-no-module-named-psycopg2 Can you assist? :)

[kumprj]

@coltonbh Hello, I face same issue on mac os. I did not understand how can I solve this, here is my question: https://stackoverflow.com/questions/67556906/improperlyconfigured-error-loading-psycopg2-module-no-module-named-psycopg2 Can you assist? :)

Is the pip package an option?

[coltonbh]

@viktor-idenfy I've added an answer to your SO question. Hope it helps!

[pratik-soni916]

I was able to get this to work today. My use case was using this as a lambda layer in a custom directory path. I was initially trying to add the path using sys.path.append() but that didn’t work. Turns out adding the path to the psycopg2 folder using sys.path.insert() worked.

[updated] os.path.append() to sys.path.append()

Where did you updated this?

[brentonmallen1]

I was able to get this to work today. My use case was using this as a lambda layer in a custom directory path. I was initially trying to add the path using sys.path.append() but that didn’t work. Turns out adding the path to the psycopg2 folder using sys.path.insert() worked. [updated] os.path.append() to sys.path.append()

Where did you updated this?

Are you asking where to put the sys.path.append()? If so, I believe it would go at the top of the script you’re attempting to use the library in. It’s been a good while since I originally posted so I could be remembering incorrectly.