wkeeling / selenium-wire

Extends Selenium's Python bindings to give you the ability to inspect requests made by the browser.
MIT License
1.89k stars 249 forks source link

Requests empty with corporate proxy #593

Open AntonioVentilii opened 1 year ago

AntonioVentilii commented 1 year ago

Hi, I'm trying to use selenium-wire at my firm, but I must be setting something wrong, because the requests come back empty. Here below a test code that should return something but isn't.

from seleniumwire import webdriver

GECKODRIVER_PATH = XXXXXXXXXXXXXX
PROXIES = {
XXXXXXXXXXXXXX
}

wire_options = {
    'auto_config': False,
    'proxy': PROXIES,
}

driver = webdriver.Firefox(executable_path=GECKODRIVER_PATH, seleniumwire_options=wire_options)

driver.get('https://www.google.com')

for request in driver.requests:
    if request.response:
        print(
            request.url,
            request.response.status_code,
            request.response.headers['Content-Type']
        )

It doesn't print anything. If i try to look into driver.requests i receive an empty list. Both GECKODRIVER_PATH and PROXIES are correct, I use them in another code smoothly.

wkeeling commented 1 year ago

You've set auto_config to False which means you'll need to manually configure Firefox to point at Selenium Wire. Without that, Selenium Wire won't capture any requests.

Firstly specify the hostname and port that Selenium Wire should listen on.

wire_options = {
    'auto_config': False,
    'addr': '127.0.0.1',  # Specify an address
    'port': 9999,         # Specify a port
    'proxy': PROXIES,
}

Then you'll need to configure Firefox's proxy settings to point at 127.0.0.1:9999. This StackOverflow post describes how to configure a proxy in Firefox.

Note that configuring Firefox's proxy settings will not override your PROXIES values. These are automatically applied by Selenium Wire once it is operational.

AntonioVentilii commented 1 year ago

Hi @wkeeling

Thanks for answering!

However I cannot change the proxy settings on my local environment. They are set by default by my firm to use the local settings. They are forcefully changed back when I try to modify them.

I have a proxy of this type http://{username}:{password}@{server}:{port}

Can i pass it to wire_options? If yes, how?

wkeeling commented 1 year ago

However I cannot change the proxy settings on my local environment. They are set by default by my firm to use the local settings. They are forcefully changed back when I try to modify them.

Unfortunately I think this may prevent you from using Selenium Wire. Just to clarify, there are 2 proxies involved. Selenium Wire works by redirecting browser traffic through a hidden proxy it starts up in the background. This means that Selenium Wire must be able to configure the browser settings to point at the hidden proxy. Once that part is working, then Selenium Wire will forward requests to your upstream corporate proxy defined in the seleniumwire_options.

So it's essential that the browser can be configured with the first proxy which sounds like it could be an issue. Are you able to try a different browser, e.g. Chrome - would that make any difference in your environment?