wkeeling / selenium-wire

Extends Selenium's Python bindings to give you the ability to inspect requests made by the browser.
MIT License
1.9k stars 254 forks source link

Proxy headers leaking #451

Closed airled closed 2 years ago

airled commented 2 years ago

I figured out problem with proxy. Selenium-wire uses header Proxy-Connection=keep-alive. Detectors see this header and tell that ip is a fake. I manually delete this header via interceptor like this:

del request.headers['Proxy-Connection']

and now f.vision says that ip is ok. Seems like selenium-wire should not use this header since security systems can detect it.

I see that it was fixed in version 3.0.0 but my version 4.5.5 and detectors see this header.

wkeeling commented 2 years ago

Thanks for raising this.

Selenium Wire does use it's own internal proxy server for capturing requests, so this header could be related to that. Ideally Selenium Wire should be completely transparent to the upstream site. I'll look at filtering this header out.

wkeeling commented 2 years ago

I've just run a test against https://httpbin.org/headers and I don't see a Proxy-Connection header in the list:

{
  "headers": {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9", 
    "Accept-Encoding": "gzip, deflate, br", 
    "Accept-Language": "en-GB,en-US;q=0.9,en;q=0.8", 
    "Host": "httpbin.org", 
    "Sec-Ch-Ua": "\"Chromium\";v=\"94\", \"Google Chrome\";v=\"94\", \";Not A Brand\";v=\"99\"", 
    "Sec-Ch-Ua-Mobile": "?0", 
    "Sec-Ch-Ua-Platform": "\"Linux\"", 
    "Sec-Fetch-Dest": "document", 
    "Sec-Fetch-Mode": "navigate", 
    "Sec-Fetch-Site": "none", 
    "Sec-Fetch-User": "?1", 
    "Upgrade-Insecure-Requests": "1", 
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36", 
    "X-Amzn-Trace-Id": "Root=1-619d4386-06b72ff52da1cba11161b6a5"
  }
}

The code I used:

import time

driver = webdriver.Chrome()
driver.get('https://httpbin.org/headers')

time.sleep(300)  # So I have time to view the browser output

Are you passing any seleniumwire_options in your version?

airled commented 2 years ago

Sorry for being unclear. Sure, I meant that I use proxy and pass proxy options with seleniumwire_options parameter to driver. And then via interceptor I see this header (and target server also sees it).

wkeeling commented 2 years ago

I've modified Selenium Wire to filter out the Proxy-Connection header. It will do this after capture (so you'll still see it in an interceptor and/or driver.requests) but it won't get passed to the target server. It could be filtered out before capture, but I wonder whether it may be useful to have some visibility of it.

wkeeling commented 2 years ago

Now available in v4.5.6