search

internetarchive / warcprox

WARC writing MITM HTTP/S proxy
380 stars 54 forks source link

Any known issues using warcprox with SSLv3? #115

Open anjackson opened 5 years ago

anjackson commented 5 years ago

I've been rendering http://acid.matkelly.com/ in a Google Chrome Puppeteer session running behind warcprox and it seems to work but I noticed some errors in the warcprox logs:

2019-03-04 13:25:03,595 1 ERROR MitmProxyHandler(tid=48,started=2019-03-04T13:25:03.270923,client=172.19.0.16:44824) warcprox.warcprox.WarcProxyHandler.do_CONNECT(mitmproxy.py:308) problem handling 'CONNECT www.cs.odu.edu:443 HTTP/1.1': SSLError(1, '[SSL: SSLV3_ALERT_CERTIFICATE_UNKNOWN] sslv3 alert certificate unknown (_ssl.c:1056)')
2019-03-04 13:25:03,596 1 WARNING MitmProxyHandler(tid=48,started=2019-03-04T13:25:03.270923,client=172.19.0.16:44824) warcprox.warcprox.WarcProxyHandler.log_error(mitmproxy.py:444) code 500, message [SSL: SSLV3_ALERT_CERTIFICATE_UNKNOWN] sslv3 alert certificate unknown (_ssl.c:1056)
2019-03-04 13:25:03,596 1 WARNING MitmProxyHandler(tid=48,started=2019-03-04T13:25:03.270923,client=172.19.0.16:44824) warcprox.warcprox.WarcProxyHandler.do_CONNECT(mitmproxy.py:314) failed to send error response ([SSL: SSLV3_ALERT_CERTIFICATE_UNKNOWN] sslv3 alert certificate unknown (_ssl.c:1056)) to proxy client: [Errno 9] Bad file descriptor
2019-03-04 13:25:03,598 1 ERROR MitmProxyHandler(tid=49,started=2019-03-04T13:25:03.502744,client=172.19.0.16:44840) warcprox.warcprox.WarcProxyHandler.do_CONNECT(mitmproxy.py:308) problem handling 'CONNECT www.cs.odu.edu:443 HTTP/1.1': SSLError(1, '[SSL: SSLV3_ALERT_CERTIFICATE_UNKNOWN] sslv3 alert certificate unknown (_ssl.c:1056)')
2019-03-04 13:25:03,598 1 WARNING MitmProxyHandler(tid=49,started=2019-03-04T13:25:03.502744,client=172.19.0.16:44840) warcprox.warcprox.WarcProxyHandler.log_error(mitmproxy.py:444) code 500, message [SSL: SSLV3_ALERT_CERTIFICATE_UNKNOWN] sslv3 alert certificate unknown (_ssl.c:1056)
2019-03-04 13:25:03,598 1 WARNING MitmProxyHandler(tid=49,started=2019-03-04T13:25:03.502744,client=172.19.0.16:44840) warcprox.warcprox.WarcProxyHandler.do_CONNECT(mitmproxy.py:314) failed to send error response ([SSL: SSLV3_ALERT_CERTIFICATE_UNKNOWN] sslv3 alert certificate unknown (_ssl.c:1056)) to proxy client: [Errno 9] Bad file descriptor
2019-03-04 13:30:04,321 1 ERROR MitmProxyHandler(tid=63,started=2019-03-04T13:30:04.171524,client=172.19.0.1:44850) warcprox.warcprox.WarcProxyHandler.do_CONNECT(mitmproxy.py:308) problem handling 'CONNECT www.cs.odu.edu:443 HTTP/1.1': SSLError(1, '[SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:1056)')
2019-03-04 13:30:04,322 1 WARNING MitmProxyHandler(tid=63,started=2019-03-04T13:30:04.171524,client=172.19.0.1:44850) warcprox.warcprox.WarcProxyHandler.log_error(mitmproxy.py:444) code 500, message [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:1056)
2019-03-04 13:30:04,322 1 WARNING MitmProxyHandler(tid=63,started=2019-03-04T13:30:04.171524,client=172.19.0.1:44850) warcprox.warcprox.WarcProxyHandler.do_CONNECT(mitmproxy.py:314) failed to send error response ([SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:1056)) to proxy client: [Errno 9] Bad file descriptor
2019-03-04 13:32:30,118 1 ERROR MitmProxyHandler(tid=71,started=2019-03-04T13:32:29.910298,client=172.19.0.13:58436) warcprox.warcprox.WarcProxyHandler.do_CONNECT(mitmproxy.py:308) problem handling 'CONNECT www.cs.odu.edu:443 HTTP/1.1': SSLError(1, '[SSL: SSLV3_ALERT_CERTIFICATE_UNKNOWN] sslv3 alert certificate unknown (_ssl.c:1056)')
2019-03-04 13:32:30,118 1 WARNING MitmProxyHandler(tid=71,started=2019-03-04T13:32:29.910298,client=172.19.0.13:58436) warcprox.warcprox.WarcProxyHandler.log_error(mitmproxy.py:444) code 500, message [SSL: SSLV3_ALERT_CERTIFICATE_UNKNOWN] sslv3 alert certificate unknown (_ssl.c:1056)
2019-03-04 13:32:30,118 1 WARNING MitmProxyHandler(tid=71,started=2019-03-04T13:32:29.910298,client=172.19.0.13:58436) warcprox.warcprox.WarcProxyHandler.do_CONNECT(mitmproxy.py:314) failed to send error response ([SSL: SSLV3_ALERT_CERTIFICATE_UNKNOWN] sslv3 alert certificate unknown (_ssl.c:1056)) to proxy client: [Errno 9] Bad file descriptor
2019-03-04 13:32:30,120 1 ERROR MitmProxyHandler(tid=72,started=2019-03-04T13:32:30.019054,client=172.19.0.13:58440) warcprox.warcprox.WarcProxyHandler.do_CONNECT(mitmproxy.py:308) problem handling 'CONNECT www.cs.odu.edu:443 HTTP/1.1': SSLError(1, '[SSL: SSLV3_ALERT_CERTIFICATE_UNKNOWN] sslv3 alert certificate unknown (_ssl.c:1056)')
2019-03-04 13:32:30,120 1 WARNING MitmProxyHandler(tid=72,started=2019-03-04T13:32:30.019054,client=172.19.0.13:58440) warcprox.warcprox.WarcProxyHandler.log_error(mitmproxy.py:444) code 500, message [SSL: SSLV3_ALERT_CERTIFICATE_UNKNOWN] sslv3 alert certificate unknown (_ssl.c:1056)
2019-03-04 13:32:30,120 1 WARNING MitmProxyHandler(tid=72,started=2019-03-04T13:32:30.019054,client=172.19.0.13:58440) warcprox.warcprox.WarcProxyHandler.do_CONNECT(mitmproxy.py:314) failed to send error response ([SSL: SSLV3_ALERT_CERTIFICATE_UNKNOWN] sslv3 alert certificate unknown (_ssl.c:1056)) to proxy client: [Errno 9] Bad file descriptor

(I think the URLs in question are https://www.cs.odu.edu/~mkelly/acid/externalScript.js and https://www.cs.odu.edu/~mkelly/acid/pixel.png)

Does warcprox support SSLv3?

anjackson commented 5 years ago

OTOH It looks like the server in question doesn't support SSLv3 so perhaps that's just how it looks in that case? https://www.ssllabs.com/ssltest/analyze.html?d=www.cs.odu.edu

nlevitt commented 5 years ago

That looks similar to the errors I see when my client (browser, curl, ...) is configured to validate certificates, except that I also see a big stack trace from WarcProxyHandler.send_error. Did you leave anything out of the logs you pasted?

2019-03-06 11:38:34,464 36187 ERROR MitmProxyHandler(tid=n/a,started=2019-03-06T19:38:34.292188,client=127.0.0.1:56899) warcprox.warcprox.WarcProxyHandler.do_CONNECT(mitmproxy.py:329) problem handling 'CONNECT www.cs.odu.edu:443 HTTP/1.1': SSLError(1, '[SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:1051)')
2019-03-06 11:38:34,464 36187 WARNING MitmProxyHandler(tid=n/a,started=2019-03-06T19:38:34.292188,client=127.0.0.1:56899) warcprox.warcprox.WarcProxyHandler.log_error(mitmproxy.py:524) code 500, message [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:1051)
2019-03-06 11:38:34,465 36187 ERROR MitmProxyHandler(tid=n/a,started=2019-03-06T19:38:34.292188,client=127.0.0.1:56899) warcprox.warcprox.WarcProxyHandler.send_error(mitmproxy.py:415) send_error(%r, %r, %r) raised exception
Traceback (most recent call last):
  File "/Users/nlevitt/workspace/warcprox/warcprox/mitmproxy.py", line 325, in do_CONNECT
    self._transition_to_ssl()
  File "/Users/nlevitt/workspace/warcprox/warcprox/mitmproxy.py", line 297, in _transition_to_ssl
    self.connection, server_side=True, certfile=certfile)
  File "/usr/local/Cellar/python/3.7.1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 1222, in wrap_socket
    suppress_ragged_eofs=suppress_ragged_eofs
  File "/usr/local/Cellar/python/3.7.1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 412, in wrap_socket
    session=session
  File "/usr/local/Cellar/python/3.7.1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 853, in _create
    self.do_handshake()
  File "/usr/local/Cellar/python/3.7.1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 1117, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:1051)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/nlevitt/workspace/warcprox/warcprox/mitmproxy.py", line 412, in send_error
    self, code, message, explain)
  File "/usr/local/Cellar/python/3.7.1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/server.py", line 478, in send_error
    self.end_headers()
  File "/usr/local/Cellar/python/3.7.1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/server.py", line 528, in end_headers
    self.flush_headers()
  File "/usr/local/Cellar/python/3.7.1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/server.py", line 532, in flush_headers
    self.wfile.write(b"".join(self._headers_buffer))
  File "/usr/local/Cellar/python/3.7.1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/socketserver.py", line 796, in write
    self._sock.sendall(b)
OSError: [Errno 9] Bad file descriptor

Warcprox logs that error ☝️when I try curl --proxy localhost:8000 https://www.cs.odu.edu/ (or e.g. https://example.com/). The error goes away when I run curl with -k.

I'm not sure what to make of the fact that you see "SSLv3" in your logs whereas I see "TLSv1". Hmm.

anjackson commented 5 years ago

I'm running warcprox like this:

warcprox -b 0.0.0.0 -d /output/warcs --base32 -z --rollover-idle-time 3600 --dedup-db-file /dev/null --stats-db-file /dev/null --quiet

So maybe the --quiet is limiting the output?

And perhaps I just need to make sure Chrome is relaxed about such things?

nlevitt commented 5 years ago

I still see the stack trace with --quiet and the curl command I was running. Could be that the browser is doing something a little different.

nclarkekb commented 5 years ago

I am getting SSL3 errors everywhere on a site I am trying to capture. Since SSL3 is almost in the grave these days I was wondering if SSL3 or TLS1.2 is used for the connection from warcprox to the destination server? Assuming the browser will be a bit more lenient if warcprox uses SSL3 to bind for incoming proxy connections?