search

scrapy / scrapyd

A service daemon to run Scrapy spiders
https://scrapyd.readthedocs.io/en/stable/
BSD 3-Clause "New" or "Revised" License
2.92k stars 569 forks source link

High frequency scheduling #466

Closed pspsdev closed 1 year ago

pspsdev commented 1 year ago

In a scenario where new job is added multiple times every second on a strong server what is the best polling rate? It seems if the polling rate is too high it will overload the cpu just because of the polling rate, if the polling rate is too low then the jobs are not starting as fast and server is wasting resources.

pspsdev commented 1 year ago

Maybe setting a queue to be rabbitmq would fix the high cpu usage in high frequency polling?