search

TeamHG-Memex / scrapy-rotating-proxies

use multiple proxies with Scrapy
MIT License
733 stars 156 forks source link

Insensible middleware priority recommendations #47

Open 3hhh opened 3 years ago

3hhh commented 3 years ago

In your README you recommend setting

    'rotating_proxies.middlewares.RotatingProxyMiddleware': 610,
    'rotating_proxies.middlewares.BanDetectionMiddleware': 620,

This however is insensible as the default settings have 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 590, which means that a custom response_is_ban(self, request, response) implementation receives a gzipped response and response.css('whatever') leads to exceptions.

To avoid that, something like

    'rotating_proxies.middlewares.RotatingProxyMiddleware': 582,
    'rotating_proxies.middlewares.BanDetectionMiddleware': 583,

makes more sense.

KostyaBovt commented 1 year ago

have the same issue @3hhh does changing original order break any functionality?

3hhh commented 1 year ago

None that I'm aware of.