disinfoRG / ZeroScraper

Web scraper made by 0archive.
https://0archive.tw
MIT License
10 stars 2 forks source link

Chromedriver still creates zombie processes on middle2 #118

Open pm5 opened 4 years ago

pm5 commented 4 years ago

Same symptom. A while ago we verified with FbScraper that the same code and settings running on GCP does not result in zombie process. Looks like we need an alternative.

The overall scheme should be to move chromedriver processes out of docker containers. Middle2, being a SaaS, is never meant to be used to run complex multiprocess tasks. It is better to treat headless browsers as a service accessed through an API, similar to how we use database services.

Some candidates: