search

aivarsk / scrapy-proxies

Random proxy middleware for Scrapy
MIT License
1.66k stars 409 forks source link

Feature: Proxy mode: change proxy every N-th request #45

Open exactstat opened 6 years ago

exactstat commented 6 years ago

If it's possible, please, add a possibility to change proxy every N-th request. Add a variable (for setting N) and a new value for "Proxy mode" for this.

Lusitaniae commented 4 years ago

Some services like Luminati provide a single endpoint for all requests.

The right approach is to re establish the connection (after n connections, or in case of failure) to the same endpoint.

Currently, if the endpoint gets blocked, the crawl will be terminated because there's no more proxies available.

If I try to copy paste the proxy multiple times in the file, is still being aggregated in the code.