thomasdondorf / puppeteer-cluster

Puppeteer Pool, run a cluster of instances in parallel
MIT License
3.2k stars 307 forks source link

cluster concurrent seems not work #527

Closed xlnhaha closed 1 year ago

xlnhaha commented 1 year ago

Hi, I used puppeteer-cluster in my program, and run it with execute, launch it with concurrent=15 then i print the time of execute function used, it looks like need longer time for the later task. in my opinion, the concurrent is 15, every 15 tasks's spending time should be close to each other, but the result is not. Detail see below image, is there something wrong i used this package? Thank you very much. image

` const cluster = await Cluster.launch({ concurrency: Cluster.CONCURRENCY_PAGE, maxConcurrency: 16, monitor: false, puppeteerOptions: { args: [ "--no-sandbox", "--disable-dev-shm-usage", "--disable-setuid-sandbox" ], executablePath: "C:\Program Files\Google\Chrome\Application\chrome.exe", headless: true, }, });

await cluster.task(async ({ page, data, worker }) => { var s1 = new Date().getTime() // make a screenshot await page.setContent(data.html); const contentSize = await page.evaluate(() => { const { width, height } = document.body.getBoundingClientRect(); return { width, height }; }); // Set viewport dimensions to match content size await page.setViewport(contentSize); await page.waitForSelector('body'); const screen = await page.screenshot({ type: "jpeg", quality: 100, deviceScaleFactor: 1 }); });

// setup server app.post("/html2img", async function (req, res) { // console.log('get request', new Date().getTime()); const data = req.body; res.header('Connection', 'keep-alive'); res.header('Keep-Alive', 'timeout=20'); if (!data) { return res.send("no html"); } try { var s1 = new Date().getTime();

  const screen = await cluster.execute(data);
  console.log('********', process.pid, new Date().getTime(), '*****spend time:', new Date().getTime() - s1)

  res.send(screen);
} catch (err) {
  // catch error
  res.send({ success: false, error: err.message });
}

}) `

xlnhaha commented 1 year ago

i think i have known the reason, it's because of the server single thread.