thomasdondorf / puppeteer-cluster

Puppeteer Pool, run a cluster of instances in parallel
MIT License
3.2k stars 307 forks source link

Error detection super slow with new Puppeteer versions #521

Closed MarioCodarin closed 1 year ago

MarioCodarin commented 1 year ago

Hi guys, I use a lot Puppeteer-Cluster and it is an amazing library but, when i use new Puppeteer-core versions (I'm actually using 19.5.2 and there is no problem) the node script get stuck when there is an error and it takes up to 2 minutes to print the error in console and start a new job. I've recognized this problem in several scripts, complicated and simple ones. I use cluster.task and cluster.queue. Do someone had the same problem?

(async () => {
    await processLineByLine('releases.txt', songs)
    await createInstances()
    const cluster = await Cluster.launch({
        maxConcurrency: threads,
        concurrency: Cluster.CONCURRENCY_BROWSER,
        workerCreationDelay: 12000,
        timeout: 360000000,
        retryLimit: 20,
        puppeteer,
        perBrowserOptions: perBrowserOptions
    })
    cluster.on('taskerror', (err, data) => {
        console.log(` Failed to crawl ${data}: ${err.message}`)
    })
    await cluster.task(async ({ page }) => {
        const pages = await page.browser().pages()
        if (pages.length > 1) await pages[0].close()
        await page.setViewport({ width: 1366, height: 768 })
        await signin(page)
        for (let i = 0; songs.length; i++) {
            await release(page, songs[i])
        }
    })
    cluster.queue()
    await cluster.idle()
    await cluster.close()
})()
MarioCodarin commented 1 year ago

Was not library's fault