Decide frequency to run for each scraper

ShelterApp / AddResources

http://shelterapp.org/

11 stars 10 forks source link

Decide frequency to run for each scraper #41

Closed prabhushrikant closed 3 years ago

prabhushrikant commented 3 years ago

We have many scrapers which will run as azure function (scheduled) . We need to decide which scraper to run when , because if they all run at the same time , it can bring down the mongoDB cluster

We need to make sure that the bigger ones are not run simultaneously

IRS and HUD shouldn't run simultaneously.

ShelterApp commented 3 years ago

Full schedule is available here : https://github.com/ShelterApp/AddResources/wiki/Scraper-schedules