Small fixes for queue management

alephdata / memorious

Lightweight web scraping toolkit for documents and structured data.

MIT License

311 stars 59 forks source link

Closed pudo closed 5 years ago

pudo commented 5 years ago

After looking through the crawlers we have, I'd also propose getting rid of the stage-level rate limiting after all...