search

wkeeling / selenium-wire

Extends Selenium's Python bindings to give you the ability to inspect requests made by the browser.
MIT License
1.9k stars 254 forks source link

Request when with socks5 proxy #436

Closed survtur closed 2 years ago

survtur commented 2 years ago

I am using selenium-wire with socks5 proxy. This combination makes site opening very unstable. Sometimes i see this error:

502 Bad Gateway TlsProtocolException('Cannot establish TLS with amazon.com:443 (sni: amazon.com): TlsException("SSL handshake error: SysCallError(-1, \'Unexpected EOF\')")')

And sometimes, when i connect i get even 502 Bad Gateway ProtocolException('Server connection to (\'192.168.8.1\', 80) failed: Error connecting to "192.168.8.1": Socket error: 0x02: Connection not allowed by ruleset')

The error above i got when connecting to https://li.ru. I don't know why 192.168.8.1 appears. I think it is somehow about mimt proxy.

When I use my socks5 proxy with plain filefox (without selenium-wire). Everything works fine and quick.

But under selenium-wire a lot of requests just did not get response at all.

Any idea where to dig?

wkeeling commented 2 years ago

Thanks for raising this. I must admit I haven't seen those errors before. Are you able to post some example code and the options you're passing in?

survtur commented 2 years ago

My research convince me that my remote socks5 proxy together with selenium-wire just cant properly process many requests or connections at once.

I half-solved issue by decreasing Firefox connection limits.

I am parsing jsons from websites so i need only main page and some special request it does. So I add ublock extension to prevent more than 70% of requests and websockets.

These measures are not a solution. I still sometimes have to refresh page few times. First attempts put some files in cache and further refreshes let other stuff to be loaded.

Unfortunately I have only one socks5 proxy and I can't make a lot of tests. Maybe other proxies with another configuration everything will work fine.