search

wkeeling / selenium-wire

Extends Selenium's Python bindings to give you the ability to inspect requests made by the browser.
MIT License
1.9k stars 254 forks source link

SeleniumWire forcing HTTP? #217

Closed Azocki closed 3 years ago

Azocki commented 3 years ago

Hey, Selenium itself works fine but when using Selenium-Wire I'm getting no internet connection, forcing http.. i was thinking that some sites FORCE HTTPS And selenium wire uses http instead?

Azocki commented 3 years ago

site has a subdomain aswell buy.blabla.com (loading normal site without subdomain works)

wkeeling commented 3 years ago

Thanks for raising this issue. Are you able to post some example code that demonstrates the problem?

Azocki commented 3 years ago

Yes i can.

Code:

from seleniumwire import webdriver

driver = webdriver.Chrome() driver.get('https://bla.bla.com') # Some random insurance company, blabla.. any site that forces HTTPS..

Basically, the first time the page is completely white, if you refresh it gives a NO INTERNET CONNECTION error.

wkeeling commented 3 years ago

I think there's some broken Javascript on the "buy" page which causes it to be blank. That happens in Chrome and Firefox outside of running Selenium/Selenium Wire.

You'll get a NO INTERNET CONNECTION message if you try and use the page after Selenium Wire has finished executing. This is due to how Selenium Wire works. It works by sending all browser traffic through an internal proxy server where it can decrypt and capture it. Once Selenium Wire finishes executing, the proxy server is shutdown, so if you attempt to use the same browser instance it won't work - because it will try to connect to a proxy that has gone.

To demonstrate, after calling driver.get() add a time.sleep(3000). This will prevent Selenium Wire from shutting down, and the browser should continue to work if you refresh the page.

But anyway, the underlying issue here I think is that the "buy" page itself is broken.

Azocki commented 3 years ago

the buy page works on https tho. not on http.

Azocki commented 3 years ago

basically selenium blocks https redirect or something like that from the page and thats why it doesnt load cause the page needs to be https.

wkeeling commented 3 years ago

I get a blank page whether I use https or http, it seems to make no difference. I'm trying the link in a browser directly outside of Selenium Wire.

Azocki commented 3 years ago

and btw i tried doing the time.sleep thing. same stuff.

Azocki commented 3 years ago

I might have fixed it. I connected to a public proxy server and tried it. works flawless

Solution: Use a proxy.