search

opinionated / scraper-core

1 stars 0 forks source link

throttle requests #4

Open ConnorFoody opened 9 years ago

ConnorFoody commented 9 years ago

We should make request timing appear more human and make sure we don't ping a site too much. While this functionality should go on top of the scheduler, it should be configurable by source. This change should be mostly on the scheduler side, but effects the scraper. Any significant changes to the scraper code as a result of this issue should go here.

See scheduler issue here for more details

ConnorFoody commented 8 years ago

Sam did client side throttle and put it here