expires in request meta not considered when pop from RedisThrottledQueue

istresearch / scrapy-cluster

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

http://scrapy-cluster.readthedocs.io/

MIT License

1.18k stars 324 forks source link

Closed softwarevamp closed 6 years ago

softwarevamp commented 6 years ago

when the RedisThrottledQueue pops it does not check the expires meta in the request, so it will be over moderated if there are some expired requests.