TeamHG-Memex / scrapy-rotating-proxies

use multiple proxies with Scrapy
MIT License
738 stars 158 forks source link

Fix the setting of `Proxy-Authorization` header #33

Open tianhuil opened 4 years ago

tianhuil commented 4 years ago

Currently, the Proxy-Authorization header is not properly reset in between retries. This is because Proxy-Authorization is only set if the header is empty, which it is not as it was copied from a previous request. The fix is to always reset Proxy-Authorization when a new proxy is issued.

Specifically, line 153 of middlewares.py should be changed from

if creds and not request.headers.get('Proxy-Authorization'):

to

if creds:

@kmike: any thoughs?