internetarchive / warcprox

WARC writing MITM HTTP/S proxy
378 stars 54 forks source link

Check if connection is still open when trying to close #135

Closed vbanos closed 5 years ago

vbanos commented 5 years ago

When an exception is raised during network communication with the remote host, we handle it and we close the socket.

Some times, the socket is already closed due to the exception and we get an extra OSError [Errno 107] Transport endpoint is not connected when trying to shutdown the socket.

We add a check to avoid that.

vbanos commented 5 years ago

Example

ConnectionResetError: [Errno 104] Connection reset by peer
  File "warcprox/mitmproxy.py", line 536, in _inner_proxy_request
    prox_rec_res.begin(extra_response_headers=extra_response_headers)
  File "warcprox/mitmproxy.py", line 187, in begin
    http_client.HTTPResponse.begin(self)  # reads status line, headers
  File "http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "warcprox/mitmproxy.py", line 142, in readline
    hunk = self.fp.readline(size)
  File "python3.5/socket.py", line 575, in readinto
    return self._sock.recv_into(b)
OSError: [Errno 107] Transport endpoint is not connected
  File "warcprox/mitmproxy.py", line 439, in do_COMMAND
    return self._proxy_request()
  File "warcprox/warcproxy.py", line 210, in _proxy_request
    self, extra_response_headers=extra_response_headers)
  File "warcprox/mitmproxy.py", line 479, in _proxy_request
    return self._inner_proxy_request(extra_response_headers)
  File "warcprox/mitmproxy.py", line 584, in _inner_proxy_request
    self._remote_server_conn.sock.shutdown(socket.SHUT_RDWR)