search

TeamHG-Memex / scrapy-rotating-proxies

use multiple proxies with Scrapy
MIT License
736 stars 157 forks source link

Make backoff cap configurable #9

Closed petermoore14 closed 7 years ago

petermoore14 commented 7 years ago

The current backoff algorithm is quite effective, but I think it would benefit from supporting configuration of the 'cap' used for the time limit. It's very possible a cluster of proxies could go down for several days, but the fixed cap used at present ensures they'll be checked around once every half hour at minimum, which isn't terribly sensible in this use case.

The fix would be to simply support the ROTATING_PROXY_BACKOFF_CAP setting, which would be used in the same way as the existing ROTATING_PROXY_BACKOFF_BASE to initialize the backoff function.

kmike commented 7 years ago

Yeah, a good suggestion. +1 to add this option.

kmike commented 7 years ago

Fixed by https://github.com/TeamHG-Memex/scrapy-rotating-proxies/pull/10 - thanks @petermoore14!