search

wkeeling / selenium-wire

Extends Selenium's Python bindings to give you the ability to inspect requests made by the browser.
MIT License
1.9k stars 254 forks source link

Selenium wire proxy error: Connection Closed Unexpectedly #182

Closed oracle1987 closed 3 years ago

oracle1987 commented 3 years ago

Hi,

iam writing a simple code to connect to website through proxy using selenium wire.

Iam using Ubuntu 20.04 and python3.8, using pycharm IDE.

CODE START:

from seleniumwire import webdriver

options = webdriver.ChromeOptions()

options = {
    'proxy': {
        'http': 'socks5://localhost:4422',
        'https': 'socks5://localhost:4422',
    },

}

driver = webdriver.Chrome(seleniumwire_options=options,executable_path="/usr/bin/chromedriver")

driver.get('https://ifconfig.me')

CODE END

note that if i exclude the driver.get('https:/...') i donot get an error. if i replace driver.get with time.sleep(10), surprisingly the same error appears although im not fetching a url through driver.get(...ifconfig.me)

I have tried to run the same proxy using selenium (not seleniumwire) and it works fine. also, I have run the above exact code in another pc and it seems to work fine.

note that the proxy is established through 'ssh -D 4422 '

when selenium-wire attempts to fetch the url, the ssh terminal establishing the proxy displays the following:

channel 5: open failed: connect failed: Network is unreachable channel 3: open failed: connect failed: Network is unreachable

the connection between the browser and the proxy seems to close immediately after its established.

Iam thankful to your suggestions.

the error is as follows:

Error making request Traceback (most recent call last): File "/home/username/PycharmProjects/testfolder/venv/lib/python3.8/site-packages/seleniumwire/proxy/socks.py", line 816, in connect negotiate(self, dest_addr, dest_port) File "/home/username/PycharmProjects/testfolder/venv/lib/python3.8/site-packages/seleniumwire/proxy/socks.py", line 450, in _negotiate_SOCKS5 self.proxy_peername, self.proxy_sockname = self._SOCKS5_request( File "/home/username/PycharmProjects/testfolder/venv/lib/python3.8/site-packages/seleniumwire/proxy/socks.py", line 531, in _SOCKS5_request resp = self._readall(reader, 3) File "/home/username/PycharmProjects/testfolder/venv/lib/python3.8/site-packages/seleniumwire/proxy/socks.py", line 285, in _readall raise GeneralProxyError("Connection closed unexpectedly") seleniumwire.proxy.socks.GeneralProxyError: Connection closed unexpectedly

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/username/PycharmProjects/testfolder/venv/lib/python3.8/site-packages/seleniumwire/proxy/proxy2.py", line 91, in proxy_request conn.request(self.command, path, req_body, dict(req.headers)) File "/usr/lib/python3.8/http/client.py", line 1255, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib/python3.8/http/client.py", line 1301, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib/python3.8/http/client.py", line 1250, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib/python3.8/http/client.py", line 1010, in _send_output self.send(msg) File "/usr/lib/python3.8/http/client.py", line 950, in send self.connect() File "/home/username/PycharmProjects/testfolder/venv/lib/python3.8/site-packages/seleniumwire/proxy/proxy2.py", line 360, in connect self.sock = _socks_connection( File "/home/username/PycharmProjects/testfolder/venv/lib/python3.8/site-packages/seleniumwire/proxy/proxy2.py", line 409, in _socks_connection return socks.create_connection( File "/home/username/PycharmProjects/testfolder/venv/lib/python3.8/site-packages/seleniumwire/proxy/socks.py", line 213, in create_connection raise err File "/home/username/PycharmProjects/testfolder/venv/lib/python3.8/site-packages/seleniumwire/proxy/socks.py", line 203, in create_connection sock.connect((remote_host, remote_port)) File "/home/username/PycharmProjects/testfolder/venv/lib/python3.8/site-packages/seleniumwire/proxy/socks.py", line 46, in wrapper return function(*args, **kwargs) File "/home/username/PycharmProjects/testfolder/venv/lib/python3.8/site-packages/seleniumwire/proxy/socks.py", line 821, in connect raise GeneralProxyError("Socket error", error) seleniumwire.proxy.socks.GeneralProxyError: Socket error: Connection closed unexpectedly Error making request Traceback (most recent call last): File "/home/username/PycharmProjects/testfolder/venv/lib/python3.8/site-packages/seleniumwire/proxy/socks.py", line 816, in connect negotiate(self, dest_addr, dest_port) File "/home/username/PycharmProjects/testfolder/venv/lib/python3.8/site-packages/seleniumwire/proxy/socks.py", line 450, in _negotiate_SOCKS5 self.proxy_peername, self.proxy_sockname = self._SOCKS5_request( File "/home/username/PycharmProjects/testfolder/venv/lib/python3.8/site-packages/seleniumwire/proxy/socks.py", line 531, in _SOCKS5_request resp = self._readall(reader, 3) File "/home/username/PycharmProjects/testfolder/venv/lib/python3.8/site-packages/seleniumwire/proxy/socks.py", line 285, in _readall raise GeneralProxyError("Connection closed unexpectedly") seleniumwire.proxy.socks.GeneralProxyError: Connection closed unexpectedly

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/username/PycharmProjects/testfolder/venv/lib/python3.8/site-packages/seleniumwire/proxy/proxy2.py", line 91, in proxy_request conn.request(self.command, path, req_body, dict(req.headers)) File "/usr/lib/python3.8/http/client.py", line 1255, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib/python3.8/http/client.py", line 1301, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib/python3.8/http/client.py", line 1250, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib/python3.8/http/client.py", line 1010, in _send_output self.send(msg) File "/usr/lib/python3.8/http/client.py", line 950, in send self.connect() File "/home/username/PycharmProjects/testfolder/venv/lib/python3.8/site-packages/seleniumwire/proxy/proxy2.py", line 360, in connect self.sock = _socks_connection( File "/home/username/PycharmProjects/testfolder/venv/lib/python3.8/site-packages/seleniumwire/proxy/proxy2.py", line 409, in _socks_connection return socks.create_connection( File "/home/username/PycharmProjects/testfolder/venv/lib/python3.8/site-packages/seleniumwire/proxy/socks.py", line 213, in create_connection raise err File "/home/username/PycharmProjects/testfolder/venv/lib/python3.8/site-packages/seleniumwire/proxy/socks.py", line 203, in create_connection sock.connect((remote_host, remote_port)) File "/home/username/PycharmProjects/testfolder/venv/lib/python3.8/site-packages/seleniumwire/proxy/socks.py", line 46, in wrapper return function(*args, **kwargs) File "/home/username/PycharmProjects/testfolder/venv/lib/python3.8/site-packages/seleniumwire/proxy/socks.py", line 821, in connect raise GeneralProxyError("Socket error", error) seleniumwire.proxy.socks.GeneralProxyError: Socket error: Connection closed unexpectedly Error making request Traceback (most recent call last): File "/home/username/PycharmProjects/testfolder/venv/lib/python3.8/site-packages/seleniumwire/proxy/socks.py", line 816, in connect negotiate(self, dest_addr, dest_port) File "/home/username/PycharmProjects/testfolder/venv/lib/python3.8/site-packages/seleniumwire/proxy/socks.py", line 450, in _negotiate_SOCKS5 self.proxy_peername, self.proxy_sockname = self._SOCKS5_request( File "/home/username/PycharmProjects/testfolder/venv/lib/python3.8/site-packages/seleniumwire/proxy/socks.py", line 531, in _SOCKS5_request resp = self._readall(reader, 3) File "/home/username/PycharmProjects/testfolder/venv/lib/python3.8/site-packages/seleniumwire/proxy/socks.py", line 285, in _readall raise GeneralProxyError("Connection closed unexpectedly") seleniumwire.proxy.socks.GeneralProxyError: Connection closed unexpectedly

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/username/PycharmProjects/testfolder/venv/lib/python3.8/site-packages/seleniumwire/proxy/proxy2.py", line 91, in proxy_request conn.request(self.command, path, req_body, dict(req.headers)) File "/usr/lib/python3.8/http/client.py", line 1255, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib/python3.8/http/client.py", line 1301, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib/python3.8/http/client.py", line 1250, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib/python3.8/http/client.py", line 1010, in _send_output self.send(msg) File "/usr/lib/python3.8/http/client.py", line 950, in send self.connect() File "/home/username/PycharmProjects/testfolder/venv/lib/python3.8/site-packages/seleniumwire/proxy/proxy2.py", line 360, in connect self.sock = _socks_connection( File "/home/username/PycharmProjects/testfolder/venv/lib/python3.8/site-packages/seleniumwire/proxy/proxy2.py", line 409, in _socks_connection return socks.create_connection( File "/home/username/PycharmProjects/testfolder/venv/lib/python3.8/site-packages/seleniumwire/proxy/socks.py", line 213, in create_connection raise err File "/home/username/PycharmProjects/testfolder/venv/lib/python3.8/site-packages/seleniumwire/proxy/socks.py", line 203, in create_connection sock.connect((remote_host, remote_port)) File "/home/username/PycharmProjects/testfolder/venv/lib/python3.8/site-packages/seleniumwire/proxy/socks.py", line 46, in wrapper return function(*args, **kwargs) File "/home/username/PycharmProjects/testfolder/venv/lib/python3.8/site-packages/seleniumwire/proxy/socks.py", line 821, in connect raise GeneralProxyError("Socket error", error) seleniumwire.proxy.socks.GeneralProxyError: Socket error: Connection closed unexpectedly

wkeeling commented 3 years ago

Thanks for raising this. Could you switch the protocol to socks5h and see if that makes a difference? So:

options = {
    'proxy': {
        'http': 'socks5h://localhost:4422',
        'https': 'socks5h://localhost:4422',
    },
}
oracle1987 commented 3 years ago

Yes, it worked. Would it be possible to clarify as to why that resolved the issue. Thanks alot.

wkeeling commented 3 years ago

It was a bit of a shot in the dark. My thinking was that if something was failing immediately (indicated by the "connect failed: Network is unreachable"), then the DNS lookup could be the culprit. Using socks5h sends the DNS query down the proxy with everything else. Bit of a guess, but glad it's working.

oracle1987 commented 3 years ago

Thanks alot. Much appreciated your support.