Avoid exception when trying to send error to client

vbanos commented 5 years ago

When a request fails for any reason, we use MitmProxyHandler.send_error to send an error response to the client.

The problem is that in many cases, there isn't an open socket connection with the client any more, so we get a BrokenPipeError send_error(%r, %r, %r) raised exception.

We just add a check to make sure that the socket connection with the client is not None before trying to send the error.

nlevitt commented 5 years ago

Can you paste the full logging thing that this avoids?

vbanos commented 5 years ago

Example command:

export https_proxy=http://127.0.0.1:8000/; curl -X GET https://cm.mgid.com/

raises the following exception:

2019-04-09 20:28:37,971 26974 ERROR MitmProxyHandler(tid=13830,started=2019-04-09T20:28:37.963770,client=127.0.0.1:54188) warcprox.warcprox.WarcProxyHandler.send_error(mitmproxy.py:420) send_error(%r, %r, %r) raised exception
Traceback (most recent call last):
  File "/opt/spn/lib/python3.5/site-packages/warcprox-2.4.2-py3.5.egg/warcprox/mitmproxy.py", line 330, in do_CONNECT
    self._transition_to_ssl()
  File "/opt/spn/lib/python3.5/site-packages/warcprox-2.4.2-py3.5.egg/warcprox/mitmproxy.py", line 302, in _transition_to_ssl
    self.connection, server_side=True, certfile=certfile)
  File "/home/vbanos/.pyenv/versions/3.5.2/lib/python3.5/ssl.py", line 1069, in wrap_socket
    ciphers=ciphers)
  File "/home/vbanos/.pyenv/versions/3.5.2/lib/python3.5/ssl.py", line 752, in __init__
    self.do_handshake()
  File "/home/vbanos/.pyenv/versions/3.5.2/lib/python3.5/ssl.py", line 988, in do_handshake
    self._sslobj.do_handshake()
  File "/home/vbanos/.pyenv/versions/3.5.2/lib/python3.5/ssl.py", line 633, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:645)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/spn/lib/python3.5/site-packages/warcprox-2.4.2-py3.5.egg/warcprox/mitmproxy.py", line 417, in send_error
    self, code, message, explain)
  File "/home/vbanos/.pyenv/versions/3.5.2/lib/python3.5/http/server.py", line 474, in send_error
    self.end_headers()
  File "/home/vbanos/.pyenv/versions/3.5.2/lib/python3.5/http/server.py", line 524, in end_headers
    self.flush_headers()
  File "/home/vbanos/.pyenv/versions/3.5.2/lib/python3.5/http/server.py", line 528, in flush_headers
    self.wfile.write(b"".join(self._headers_buffer))
  File "/home/vbanos/.pyenv/versions/3.5.2/lib/python3.5/socket.py", line 593, in write
    return self._sock.send(b)
OSError: [Errno 9] Bad file descriptor

These exceptions no longer occur with my fix, you only get:

2019-04-09 20:31:25,882 17772 ERROR MitmProxyHandler(tid=19176,started=2019-04-09T20:31:25.856578,client=127.0.0.1:54190) warcprox.warcprox.WarcProxyHandler.do_CONNECT(mitmproxy.py:334) problem handling 'CONNECT cm.mgid.com:443 HTTP/1.1': SSLError(1, '[SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:645)')

vbanos commented 5 years ago

Also note that this is quite common. There are more than 1k exceptions like that in production for the last few days.

nlevitt commented 5 years ago

Closing in favor of #122

internetarchive / warcprox

Avoid exception when trying to send error to client #121