Error deploying to AWS

[peter-borgstedt]

Running an EventTrigger on a S3 bucket, trigged when a PDF is uplloaded.

As soon as I require("hummus") it will break with following. Any suggestion on what I'm doing wrong.

module initialization error: Error at Object.exports.find (/var/task/pdf_watcher/index.js:142672:15) at Object. (/var/task/pdf_watcher/index.js:140663:27) at Object.459../PDFRStreamForFile (/var/task/pdf_watcher/index.js:140689:4) at s (/var/task/pdf_watcher/index.js:1:682) at /var/task/pdf_watcher/index.js:1:733 at Object.518.../lib/fileHelper (/var/task/pdf_watcher/index.js:148752:16) at s (/var/task/pdf_watcher/index.js:1:682) at e (/var/task/pdf_watcher/index.js:1:853) at /var/task/pdf_watcher/index.js:1:871 at a (/var/task/pdf_watcher/index.js:1:150) at Object. (/var/task/pdf_watcher/index.js:1:384) at Module._compile (module.js:570:32) at Object.Module._extensions..js (module.js:579:10) at Module.load (module.js:487:32) at tryModuleLoad (module.js:446:12) at Function.Module._load (module.js:438:3)

[peter-borgstedt]

Looking at the code in AWS. Following is set, and hummus is the only library that is doing this, using "/opt/atlassian/pipelines/agent/build/node_modules/hummus" as path defining this..

/ addon class for simple node wrappers / hummus.PDFStreamForResponse = require('./PDFStreamForResponse'); hummus.PDFWStreamForFile = require('./PDFWStreamForFile'); hummus.PDFRStreamForFile = require('./PDFRStreamForFile'); }).call(this,"/opt/atlassian/pipelines/agent/build/node_modules/hummus") },{"./PDFRStreamForFile":456,"./PDFStreamForResponse":457,"./PDFWStreamForFile":458,"events":undefined,"fs":undefined,"node-pre-gyp":492,"path":undefined}],460:[function(require,module,exports){

[peter-borgstedt]

Is it even possible to run hummusjs on aws as a lambda?

[peter-borgstedt]

I'm using serverless for this.

[peter-borgstedt]

}).call(this,"/opt/atlassian/pipelines/agent/build/node_modules/hummus")

seems wrong.

[galkahana]

it's cause its native. had the same problem with imagemagik.

got some options here: https://aws.amazon.com/blogs/compute/nodejs-packages-in-lambda/

[or look for "lambda nodejs binary"]

[peter-borgstedt]

That url and other url's basically is: host on a EC2.

Well, then its not possible to do it with lambda. Because I've tried attaching files with lambda but this just doesn't work.

We should conclude that this package is not possible to be used on AWS lambdas. I guess I will try to host with a EC2, but that really is not they way i wanted to do these thing.

And no one here seems to have any knowledge of this problem, so I guess there not much more to discuss here.

[galkahana]

look. if you want to give up, it's up to you. you should be able to use it with lambda, as i did with imagemagick. all you have to do is compile it, along with other modules, in an env that's like the lambda's env, and then load up the result to the lambda.

you can do this the hard way with finding such a machine, or an easy way with dockers. this should help you if you want to investigate the docker option, which i used successfully: https://github.com/lambci/docker-lambda

[chunyenHuang]

I have been using Hummus with Lambda for a while. I just created an example using HummusRecipe in Lambda HERE for this ticket.

Lambda has payload size limit to 6MB, so my workaround is to upload pdf files to s3 and trigger Lambda from there. You can use either s3 watching or sns to do the trick.

(Sorry I was busy working on something else recently, could not reply until now.)

[tekamoment]

@galkahana I've tried building on an EC2 instance and using what was generated as a replacement for the node_module hummus that comes with a standard npm install, but it still gives the same error.

[chunyenHuang]

What's the error message? It will be good if you can also post what you have installed, like node, npm, g++, make and their version?

I have tested with a t2.micro linux EC2 instance and installed HummusJS successfully.

curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.32.0/install.sh | bash
. ~/.nvm/nvm.sh
nvm install 8.7.0
yum install git
git clone https://github.com/galkahana/HummusJS.git
cd HummusJS/
npm i node-pre-gyp -g
npm i
npm test

If you are seeing under sudo NPM issue

npm WARN lifecycle hummus@1.0.83~install: cannot run in wd %s %s (wd=%s) hummus@1.0.83 
node-pre-gyp install --fallback-to-build /home/ec2-user/HummusJS

Then you can install with this command directly

node-pre-gyp install --fallback-to-build

[galkahana]

b.t.w ran through my notes from when i used native modules in lambda. was reminded of this: https://github.com/lambci/docker-lambda

very good docker images that can be used for both testing and building your lambda (including native modules) without having to setup an EC2

[peter-borgstedt]

Hey. Sorry for the outburst, and the late answer. The problem has been solved for a while now.

I had issues with following:

• Deploying manually (obviously) you need to compile the native code bundle in the correct architecture. I'm running on a Mac and I needed to compile it first on Linux (so used EC2 for that).

• When deploying with Serverless and Bitbucket I had to downgrade to 1.21.1. The later versions have the profile bug which does work well with my setup on bitbucket. Lower than 1.21.1 has the bug that corrupt binary files. So 1.21.1 is one version that actually seem to work now.

Thanks for the responses though!

[greg-benner-klick-sensei]

Dockerfile

FROM amazonlinux:latest

# set locale
RUN echo LC_ALL=en_GB.UTF-8 >> /etc/environment
ENV LC_ALL=en_GB.UTF-8

# install node and build tools
RUN curl -sL https://rpm.nodesource.com/setup_8.x | bash - && \
yum install -y nodejs gcc-c++ make git

# install serverless
RUN npm install -g serverless

---

docker run --rm -it -v $HOME/.aws:/root/.aws -v $PWD:/src -v $PWD/.tmp/node_modules:/src/node_modules sls-amazonlinux:latest

You can deploy from within docker (serverless deploy) or copy ~/.tmp/node_modules which are now compiled to a lambda equivalent environment

[rsshilli]

I've created the lambda hummus library specifically to solve this problem of trying to deploy this thing to Lambda from Windows. It's been working for us.