search

johntitus / node-horseman

Run PhantomJS from Node
MIT License
1.45k stars 124 forks source link

running node-horseman on AWS Lambda #232

Closed Raidus closed 8 years ago

Raidus commented 8 years ago

Hey there!

I'm struggling with running node-horseman on AWS Lambda. The script runs locally just fine. But when I deploy it via ClaudiaJS to AWS Lambda I'm getting the error for my request:

{
name: "HeadlessError",
message: "Phantom immediately exited with: 126"
}

The path to phantomjs is set:

var horseman = new Horseman({phantomPath: phantomjs.path})

I'm using the phantomjs-prebuilt: phantomjs-1.9.8-linux-x86_64.tar.bz2 (following the advice of this post: "How do I run PhantomJS on AWS Lambda with NodeJS")

Any ideas?

Raidus commented 8 years ago

I'm getting the same error on a EC2 machine:

{ HeadlessError: Phantom immediately exited with: 126
    at ChildProcess.immediateExit (/home/ec2-user/projects/iQseller/keyword-hunter/node_modules/node-phantom-simple/node-phantom-simple.js:153:23)
    at ChildProcess.g (events.js:291:16)
    at emitTwo (events.js:106:13)
    at ChildProcess.emit (events.js:191:7)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:215:12)
  name: 'HeadlessError',
  message: 'Phantom immediately exited with: 126' }
awlayton commented 8 years ago

It seems that code 126 means the PhantomJS binary could not be executed.

It is probably using the version for the wrong platform, or the permissions on the file are wrong.

Could you be more specific about what "using the phantomjs-prebuilt: phantomjs-1.9.8-linux-x86_64.tar.bz2" means? What exactly is the phantomjs variable?

Raidus commented 8 years ago

I was able to run the script on a EC2 AMI machine by installing the phantom-prebuilt using:

PHANTOMJS_PLATFORM=linux PHANTOMJS_ARCH=x64 npm install phantomjs-prebuilt --phantomjs_cdnurl=https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-1.9.8-linux-x86_64.tar.bz2 -S

The prebuilt is load via

var phantomjs = require('phantomjs-prebuilt')
var horseman = new Horseman({phantomPath: phantomjs.path})

As far as I know, Lambda uses the EC2 AMI architecture. My simple approach was to download the working node_modules from the EC2 instance and uploading it locally via ClaudiaJS.

Unfortunately, I'm getting the same error for the Lambda function.

awlayton commented 8 years ago

Are you able to get a simple script to run in AWS using just node-phantom or preferably node-phantom-simple? I would start there.

This is not a horseman issue. PhantomJS is not even running. It might be a node-phantom-simple issue, but it might just be something weird about the environment.

Raidus commented 8 years ago

Yes, you're right. It's not related to node-horseman. That's what I just realized :-)

If I'll find a solution, I'll mention this here.

awlayton commented 8 years ago

Please do @Raidus, it would be good to know how to make this work.

Raidus commented 8 years ago

I've found working solution for phantomjs with AWS Lambda: https://github.com/akrylysov/lambda-phantom-scraper

I'm not sure how this could work with node-horseman?

awlayton commented 8 years ago

It looks like they just use phantomjs-prebuilt normally...

LabN36 commented 7 years ago

Hey, is it possible to run phantom with node-horseman on AWS Lambda ?

vietbui commented 7 years ago

@LabN36 Solution found. I had to rebuild (npm rebuild) on an Amazon Linux machine (EC2 instance). Then use the rebuilt node_modules/phantomjs-prebuilt/lib/phantom/bin/phantomjs instead of the default one comes with phantomjs-prebuilt.

New phantomjs file has size of ~68mb compared to the default one ~45mb.