Only rotate proxy when banned or dead

TeamHG-Memex / scrapy-rotating-proxies

use multiple proxies with Scrapy

MIT License

733 stars 156 forks source link

Open Marow opened 4 years ago

Marow commented 4 years ago

Hi, i have a list with 1000 proxies and over 200 spiders running same time.

I have set CONCURRENT_REQUESTS = 1 so i want each proxy running one time overall.

But i want also that each proxy is use until it gets banned or its dead or something. Now its running and changing on each request