search

thomasdondorf / puppeteer-cluster

Puppeteer Pool, run a cluster of instances in parallel
MIT License
3.2k stars 307 forks source link

CPU limiting with cpuset_cpus not working #497

Closed diego95root closed 1 year ago

diego95root commented 1 year ago

Hi,

Thank you for the great library! I was playing with it on docker on one of my VPSes and wanted to limit the CPU resources that each container can use. For that I used cpuset_cpus with one container and set it to 0.

Using Cluster.CONCURRENCY_PAGE, when I set maxConcurrency to 1 everything works fine and it runs as usual. However, when I increase it to something else like 2 then it seems that everything breaks down:

2022-11-29T12:44:43.020Z puppeteer-cluster: Worker Starting #0
2022-11-29T12:44:43.046Z puppeteer-cluster: Worker Starting #1
2022-11-29T12:44:48.038Z puppeteer-cluster: Worker Error getting browser page (try: 0), message: Timeout hit: 5000
2022-11-29T12:44:48.058Z puppeteer-cluster: Worker Error getting browser page (try: 0), message: Timeout hit: 5000

The machine is idle and there are no other processes, so I'm not sure what the issue is. I would appreciate it if you could help me look into it.

Kind regards,

diego95root commented 1 year ago

Looks like I had accidentally removed --disable-gpu, after adding it back it works fine. I wonder why it is required though... Thanks!