socket.timeout: _ssl.c:1059: The handshake operation timed out

[Imran109]

Whenever I open any website using get, the next moment I am getting this error and its so annoying because the browser just stops loading! Traceback (most recent call last): File "/Users/opt/anaconda3/lib/python3.7/site-packages/seleniumwire/proxy/proxy2.py", line 91, in proxy_request conn.request(self.command, path, req_body, dict(req.headers)) File "/Users/opt/anaconda3/lib/python3.7/http/client.py", line 1252, in request self._send_request(method, url, body, headers, encode_chunked) File "/Users/opt/anaconda3/lib/python3.7/http/client.py", line 1298, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/Users/opt/anaconda3/lib/python3.7/http/client.py", line 1247, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/Users/opt/anaconda3/lib/python3.7/http/client.py", line 1026, in _send_output self.send(msg) File "/Users/opt/anaconda3/lib/python3.7/http/client.py", line 966, in send self.connect() File "/Users/opt/anaconda3/lib/python3.7/site-packages/seleniumwire/proxy/proxy2.py", line 368, in connect super().connect() File "/Users/opt/anaconda3/lib/python3.7/http/client.py", line 1422, in connect server_hostname=server_hostname) File "/Users/opt/anaconda3/lib/python3.7/ssl.py", line 423, in wrap_socket session=session File "/Users/opt/anaconda3/lib/python3.7/ssl.py", line 870, in _create self.do_handshake() File "/Users/opt/anaconda3/lib/python3.7/ssl.py", line 1139, in do_handshake self._sslobj.do_handshake() socket.timeout: _ssl.c:1059: The handshake operation timed out

[wkeeling]

Thanks for the issue. Could you post your code including any options you're passing to the webdriver instance. Also, what version of Selenium Wire are you using?

[Imran109]

@wkeeling Name: selenium-wire Version: 2.1.0 Summary: Extends Selenium to give you the ability to inspect requests made by the browser. Home-page: https://github.com/wkeeling/selenium-wire Author: Will Keeling Author-email: will@zifferent.com License: MIT

from selenium.webdriver.chrome.options import Options  
chrome_options = Options()
chrome_options.binary_location = "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"
driver = webdriver.Chrome(executable_path = os.path.abspath("drivers/chromedriver"), chrome_options = chrome_options)

I was using the same arguments only all these days. Today only I am facing with this issue

[Imran109]

@wkeeling When this issue will be resolved. Because we are trying to use this in production level and it causes failure!

[wkeeling]

If this had previously been working and has stopped working, and nothing has changed in your environment, then I can only assume there is some issue with the remote site you are connecting to, or perhaps the network? The error seems to be an SSL handshake timing out. Can you access the same URL if you just use Selenium itself?

[wkeeling]

@Imran109 one other thing to try - try increasing the timeout:

from selenium.webdriver.chrome.options import Options  
chrome_options = Options()
chrome_options.binary_location = "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"

sw_options = {
    'connection_timeout': 60
}

driver = webdriver.Chrome(
    executable_path=os.path.abspath("drivers/chromedriver"), 
    chrome_options=chrome_options,
    seleniumwire_options=sw_options
)

[Imran109]

@wkeeling Tried this. If I set the connection timeout the page simply keeps loading and times out eventually and still get the same error .

[wkeeling]

Do you see timeouts if you switch the backend to mitmproxy?

[Imran109]

@wkeeling I tried with normal selenium. It's working as expected without any errors. I was doing the same task only for the past one week, suddenly I face this issue these two days. I don't know anything about mitmproxy. The site I open with normal selenium works fine and the same site gives me the error with selenium wire.

socket.timeout: _ssl.c:1059: The handshake operation timed out
Error making request

[wkeeling]

@Imran109 is the site publicly accessible? If I can reproduce I can try to debug.

Switching to the mitmproxy backend is a workaround which may get you past the problem. It's fairly simple to set up.

[Imran109]

@wkeeling Yes, its just Instagram. Try to login into the page using selenium wire. The error is also not consistent. 1/10 times I don't see the error. But I get it almost every time I run the code.

[wkeeling]

Ok thanks will try and reproduce.

[Imran109]

@wkeeling Or, could you help me how I can achieve the same using mitmproxy. What I want to do is, get a particular request header, get the response of it and store it in a json or any format while inside selenium. Seleniumwire was perfect for me.

[wkeeling]

Yes you can do that with the mitmproxy backend.

To switch the backend, first install mitmproxy:

pip install mitmproxy

Then specify the backend in the options you pass to the webdriver:

from selenium.webdriver.chrome.options import Options  
chrome_options = Options()
chrome_options.binary_location = "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"

sw_options = {
    'backend': 'mitmproxy'
}

driver = webdriver.Chrome(
    executable_path=os.path.abspath("drivers/chromedriver"), 
    chrome_options=chrome_options,
    seleniumwire_options=sw_options
)

You can then load the page:

driver.get(...)

Retrieve the request you are interested in, e.g the last request captured:

request = driver.last_request

(Or perhaps locate the request with request = driver.wait_for_request('http://myserver.com/some/path?foo=bar').)

Store the header you require together with the response:

header, response =  request.headers['Some-Header'], request.response

mitmproxy runs on port 9950. If you have any issues with "address already in use" you can switch the port in the options:

sw_options = {
    'backend': 'mitmproxy',
    'port': 9951
}

[Imran109]

@wkeeling Still facing the exact same issue! For the first two times, it ran without error

[wkeeling]

Can you post the traceback you see when using mitmproxy.

[wkeeling]

Some further investigation on this indicates that the timeouts may be down to OpenSSL which ssl.py uses under the covers. Can you post the version of OpenSSL you are using:

openssl version

I am using version OpenSSL 1.1.0g 2 Nov 2017 (Library: OpenSSL 1.1.1 11 Sep 2018). I have not experienced any timeout issues with this version.

[JasonCrowe]

I am getting some socket.timeout issues with LibreSSL 2.8.3. I don't know if it's connected, but I thought it can't hurt to share the info.

[JasonCrowe]

I have the same script running on another computer with the same ssl version and no issues.

[regisfrais]

I have the same problem, many times I get this message, sometimes the application stops, I tried to catch the exception of the request but without success, I believe it is a problem with SSL, the version I'm running here is LibreSSL 2.2.7

[wkeeling]

@JasonCrowe @regisfrais thanks for the info. I'll see if I can reproduce using LibreSSL. What OS are you using?

[regisfrais]

@wkeeling For development and testing I use macOS High Sierra 10.13.6, but I get the same error message on a Windows Server 2012 machine installed with all the latest updates available through the datacenter.

[JasonCrowe]

MacOS python 3.6 - No issues MacOS python 3.8 - Doesn't work

Edit: This is on two different Macbook Pros

[JasonCrowe]

Is there any other information I can provide?

[wkeeling]

@JasonCrowe apologies for the delay on this - I'm a little stumped. It might be useful to know whether you get the same issue if you switch to the mitmproxy backend on the machine that has the problem. That might yield some further clues.

To switch, first install mitmproxy:

pip install mitmproxy

and then specify the backend in the options you pass to the webdriver:

options = {
    'backend': 'mitmproxy'
}
driver = webdriver.Firefox(seleniumwire_options=options)  # Or Chrome

[JasonCrowe]

When I switch to mitmproxy, I no longer have the issue.

Also, when I use the default, it appears to work. The browser goes through the steps it's supposed to without any visible browser issues. BUT... My logfile is a bloodbath of red.

I'll use mitmproxy as a workaround. Thanks for the suggestion.

[wkeeling]

The core of Selenium Wire has recently been reworked to improve connection handling with upstream proxies. If you are using an older version of Selenium Wire and are still experiencing issues, try upgrading to the latest version.