search

thomasdondorf / puppeteer-cluster

Puppeteer Pool, run a cluster of instances in parallel
MIT License
3.24k stars 310 forks source link

Would it be possible to run puppeteer-cluster on AWS Lambda? #60

Open krunalshah68 opened 5 years ago

krunalshah68 commented 5 years ago

I have tried running puppeteer-cluster with AWS Lambda Optimised chrome binaries from https://github.com/Kikobeats/aws-lambda-chrome. but running into errors. Is it possible to run cluster on AWS Lamda?

thomasdondorf commented 5 years ago

I have not tried it myself, but if you are able to launch two puppeteer instances in parallel, there should be no problems using the library.

What did you try and what errors did you get?

krunalshah68 commented 5 years ago

I've tried to provide chrome executable path from https://github.com/Kikobeats/aws-lambda-chrome and getting this error

 UnhandledPromiseRejectionWarning: Error: Unable to get browser page
    at Worker.<anonymous> (/usr/local/load_test/node_modules/puppeteer-cluster/dist/Worker.js:43:31)
    at Generator.next (<anonymous>)
    at fulfilled (/usr/local/load_test/node_modules/puppeteer-cluster/dist/Worker.js:4:58)
    at process._tickCallback (internal/process/next_tick.js:68:7)
krunalshah68 commented 5 years ago

any update on this @thomasdondorf ?

thomasdondorf commented 5 years ago

No. I'm not using this library on AWS Lambda.

In case you want to investigate:

krunalshah68 commented 5 years ago

Hey, @thomasdondorf Thanks for the reply, Actually, I have moved my project to EC2 due to a deadline but will post try and post updates about your questions soon.

mapi33 commented 5 years ago

Hi @krunalshah68, I would love to know if you've managed to run puppeteer cluster on Lambda

cmaycumber commented 5 years ago

I was able to get it working in a lambda with this forked version of puppeteer-cluster: https://github.com/NikolaiT/puppeteer-cluster/ and https://github.com/alixaxel/chrome-aws-lambda

mapi33 commented 5 years ago

Awesome! Thanks, I'll try it out

thomasdondorf commented 5 years ago

That commit/repo you are linking to does not change anything related to running on AWS, it just adds "per browser options". So, if you got it running on AWS with that repo, I am very positive, that you should be able to get it running with this repository. @cmaycumber Maybe you can share your code?

Kikobeats commented 5 years ago

Hello,

I've recently discovered this library and I'm using it with AWS Lambda.

That's my configuration:

const { Cluster } = require('puppeteer-cluster')
const HEADLESS = !!process.env.LAMBDA_TASK_ROOT
const REQ_TIMEOUT = 8000

return Cluster.launch({
    concurrency: Cluster.CONCURRENCY_PAGE,
    maxConcurrency: 2,
    puppeteer: isLambda ? require('puppeteer-core') : require('puppeteer'),
    puppeteerOptions: {
      args: [
        ...chromium.args,
        '--disable-web-security',
        '--disable-dev-profile',
        '--font-render-hinting=medium', // could be 'none', 'medium'
        '--enable-font-antialiasing'
      ],
      defaultViewport: chromium.defaultViewport,
      executablePath: await chromium.executablePath,
      headless: HEADLESS,
      timeout: REQ_TIMEOUT,
      userDataDir: '/dev/null'
    }
  })

Also, I'm the author of browserless, a top-level wrapper around puppeteer.

I was thinking could be very great to adopt puppeteer-cluster inside browserless. Right now, I have a generic pool package, but this library is a better solution since you are handling errors, etc.

The premise of browserless is to provide sensible good defaults (e.g., the library has a builtin adblocker by default) so just I need to expose maxConcurrency and concurrency as part of the shared interface and any user can setup a cluster of instances automagically 🙂

thomasdondorf commented 5 years ago

Nice, thank you for sharing your config.

vidyanand commented 4 years ago

Hi @thomasdondorf , I have been able to get puppeteer-cluster running on AW Lmabda with the intrsuctions provided above by @Kikobeats . However it works only when I set the concurrent to Cluster.CONCURRENCY_PAGE. IN the cases of both Cluster.CONCURRENCY_CONTEXT and Cluster.CONCURRENCY_BROWSER , it doesnt seem to work. It looks like the worker(s) are initialized but fails when it trys to get the browser page

puppeteer-cluster: Worker Error getting browser page (try: 5), message: Protocol error (Target.createTarget): Target closed.

I ran the lambda with DEBUG for puppeteer* and here is the log file from running it.

Can you advise on what I might be doing wrong ?

deldrid1 commented 4 years ago

@vidyanand - I'm running into the same problem with Cluster.CONCURRENCY_CONTEXT. Did you ever get this resolved?

vidyanand commented 4 years ago

@deldrid1 Nope. I did not. I dont think you can do this with AWS Lambda right now with this package. This article (though not related to pupetter at all) may shed some light on this.

eezdev commented 3 years ago

Hi, did anyone succeeded to run this using Cluster.CONCURRENCY_CONTEXT ? I confirm @Kikobeats solution is working fine with Cluster.CONCURRENCY_PAGE

Uchihinha commented 8 months ago

The same here, it's not working with CONCURRENCY_CONTEXT, any updates on it?

johnwheeler commented 1 day ago

related: https://github.com/puppeteer/puppeteer/issues/1947

johnwheeler commented 1 day ago

The reason it is not possible on AWS Lambda is because you're likely using @sparticuz/chromium which passes in --single-process. For some reason, Chrome requires that switch be excluded if you are using incognito, which CONCURRENCY_BROWSER depends on.

https://github.com/alixaxel/chrome-aws-lambda/issues/15