Running pgloader in AWS Lambdas

[vijmen]

Hi Dimitri

I have a use-case which would be a good fit for AWS Lambdas within a VPC.

Basically, once a message / file is received within AWS SQS / SNS / S3, I want to trigger a lambda function which would call pgloader to load the contents into a db table. As they are small messages, it would be very fast.

As per the Lambda docs, we can call any statically linked executable that's built for the matching version of Amazon Linux (docs at: https://aws.amazon.com/blogs/compute/running-executables-in-aws-lambda/ and http://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html)

Was wondering whether you'd be open to providing a prebuilt version for this.

Thanks vm

[dimitri]

Apparently we need a kind of a wrapper in the form of a supported Lambda function (in either Node, Python or Java) that would then run our static binary from the zip data. Please try it and let me know of any help required? I will add the necessary wrapper code right in the pgloader source tree to make it easier, and we will need to cover that use case with some docs too.

[vijmen]

Thanks for responding. I am comfortable with the lambda framework as have used this for a few other things using the same approach of a aws linux compatible executable. However, I wasn't able to build a statically linked executable that I can call from within this. Was wondering whether you could provide / maintain just the linux executable.

[dimitri]

The pgloader binary is self-contained except from some .so dependencies. My understanding is that those libssl and libsqlite and freetds are problems in the lambda framework, right?

I've been told there's a way to build a SBCL binary that could statically link those libs, will search more into that.

[vijmen]

Thanks, I'm not really an expert at building this kind of binary.

[phsilva]

@dimitri I got into this ame route trying to run pgloader on a AWS Lambda.

I built SBCL with compress-core enabled and managed to build pgloader. When running locally everything goes right, but on Lambda it fails with exit code -9 (let me know if this makes any sense).

I noticed that the final binary, while a dynamicaly linked, does not depend on things I would expect from the build dependencies (openssl and so on). Are those loaded at runtime (dlopen)?

I basically followed your Dockerfile and built pgloader using lambci/docker-lambda image which are recommended to replicate AWS Lambda environment.

Just pushing here to see if you have any clue, things are fuzzy to me on this, will keep testing.

[dimitri]

It is indeed using dlopen IIUC... it might be possible to build a static binary, but I don’t know how. Maybe you could get help from SBCL people?

[phsilva]

@dimitri thanks for the prompt answer.

I managed to run it after your mention to dlopen. libssl.so was missing in my AWS lambda bundle, after that everything runs well. (LD_DEBUG=all was really helpful here).

I will post a recipe about how to do it and we could close this one. Thanks again.

[dimitri]

I'm quite interested into your result, can't wait to read your post!

[phsilva]

@dimitri I ran into an issue. I am using a docker image that tries to recreate the AWS Lambda environment exactly. In this image I was able to run the binary I created, unfortunately when I upload the same program and files to AWS it fails.

I still figuring out why, will add an issue into lambci/docker-lambda repo so they could take a look. They say the images are "almost" the same as real env, this is a case where behavior are different, think they will like to investigate.

Will keep ou posted.

[bizunas]

@phsilva Did you manage to figure a solution for this or did you find some other solution ?