Building a static libpq for AWS Lambda

[whatupdave]

Trying to get this working on AWS Lambda, getting this:

/var/task/node_modules/pg-native/node_modules/libpq/node_modules/bindings/bindings.js:83 
 throw e 
 ^ 
Error: libpq.so.5: cannot open shared object file: No such file or directory 
 at Module.load (module.js:356:32) 
 at Function.Module._load (module.js:312:12) 
 at Module.require (module.js:364:17) 
 at require (module.js:380:17) 
 at bindings (/var/task/node_modules/pg-native/node_modules/libpq/node_modules/bindings/bindings.js:76:44) 
 at Object.<anonymous> (/var/task/node_modules/pg-native/node_modules/libpq/index.js:1:108) 
 at Module._compile (module.js:456:26) 
 at Object.Module._extensions..js (module.js:474:10) 
 at Module.load (module.js:356:32) 
 at Function.Module._load (module.js:312:12) 
 at Module.require (module.js:364:17) 
 at require (module.js:380:17) 
 at Object.<anonymous> (/var/task/node_modules/pg-native/index.js:1:75) 
 at Module._compile (module.js:456:26) 
 at Object.Module._extensions..js (module.js:474:10) 
 at Module.load (module.js:356:32)

According to this article: https://aws.amazon.com/blogs/compute/nodejs-packages-in-lambda/ libpq needs to be statically compiled. I'm a little out of my depth here, but is this something that can be done easily?

[boriscosic]

libpq-static.zip

@whatupdave @dcrockwell I've uploaded a libpq created with static binding to libpq for use on aws lambda. All lib files should go inside lib folder and libpq folder should go inside node_modules wherever its required.

To create this I used the postgres 9.4 source and created libpq and pg_config.

./configure --with-openssl --without-readline --with-prefix=~/lib/libpq
cd src/interfaces/libpq && make && make install

cd ./src/bin/pg_config && make && make install
export PATH=$PATH:[path to postgres source]/src/bin/pg_config

Make sure that PATH contains path to pg_config and LD_LIBRARY_PATH points to your new libpq so's. Then you should be able to build node-libpq for aws via: npm install node-libpq.

Hope that helps. B

[srayhunter]

@boriscosic Can you provide some more details on how you compiled libpq statically for includes in node_modules?

[lrowe]

I've been able to mostly replicate this with the following Dockerfile. However if I build --with-openssl (after installing openssl-devel) then it segfaults when running in Lambda. Not sure how to get the right version of openssl to build with or whether it's some other library pulled in when building with openssl.

FROM public.ecr.aws/sam/build-nodejs14.x
RUN yum install which -y \
      && curl -sO https://ftp.postgresql.org/pub/source/v14.1/postgresql-14.1.tar.bz2 \
      && tar -xjf postgresql-14.1.tar.bz2 \
      && cd /var/task/postgresql-14.1 \
      && ./configure --without-openssl --without-readline --prefix=/var/task/lib/libpq \
      && cd src/interfaces/libpq && make && make install && cd - \
      && cd src/bin/pg_config && make && make install && cd -

ENV PATH /var/task/postgresql-14.1/src/bin/pg_config:$PATH
ENV CPATH /var/task/postgresql-14.1/src/include

COPY package.json package-lock.json ./
RUN npm ci

Then to extract the result from the container:

docker build .
docker run $(docker build . -q) zip -r - lib/ node_modules/ > lib.zip

I've just been adding the rest of my files to this zip and uploading, but I should probably make it a layer.

[lrowe]

Updating to use openssl11 enables me to build / run successfully with ssl support:

FROM public.ecr.aws/sam/build-nodejs14.x
RUN yum install -y which openssl11-devel openssl11-static \
      && curl -sO https://ftp.postgresql.org/pub/source/v14.1/postgresql-14.1.tar.bz2 \
      && tar -xjf postgresql-14.1.tar.bz2 \
      && cd /var/task/postgresql-14.1 \
      && ./configure --with-openssl --without-readline --prefix=/var/task \
      && cd src/interfaces/libpq && make && make install && cd - \
      && cd src/bin/pg_config && make && make install && cd - \
      && cp src/include/postgres_ext.h src/include/pg_config_ext.h src/include/pg_config.h /var/task/include/ \
      && cp /lib64/libcrypto.so.1.1 /lib64/libssl.so.1.1 /var/task/lib/

ENV PATH /var/task/bin:$PATH
ENV CPATH /var/task/include
COPY package.json package-lock.json ./
RUN npm ci

Then to extract the result from the container:

docker build .
docker run $(docker build . -q) zip -r9 - lib/ node_modules/ > lib.zip