Failing tests in test_ssl.py

[jkbrzt]

There are some failing test in test_ssl.py.

All the failures seem to be caused by some SSL-related issue pytest-httpbin as they only occur when the httpbin_secure fixture is used.

The HTTPS connection times out on Python3 and Pypy:

• https://travis-ci.org/jakubroztocil/httpie/jobs/49870580
• https://travis-ci.org/jakubroztocil/httpie/jobs/49870583#L394

I can reproduce the timeouts also when running the test via py.test … from terminal even on Python 2.7.9, but strangely enough, they don't occur when the test suite is being run from PyCharm.

// cc @kevin1024, @msabramo

[kevin1024]

Taking a quick look at this tonight. It looks to me like it's isolated to pypy and pypy3. I wasn't able to reproduce the problem in python3 (and it looks like Travis is fine on Python 3).

[kevin1024]

Looks like pypy is doing some fancy new SSL error handling that is causing problems for pytest-httpbin.

Here's what I think is happening:

1. Pytest-httpbin starts a server in a new thread
2. One of httpie's tests that makes a HTTPS request with a broken cert causes an SSLError exception in the server's thread. These fancy new SSLErrors seem to only be raised in pypy.
3. Subsequent tests hang waiting for the server thread to respond to their request.

[kevin1024]

I opened an issue on pytest-httpbin. I'll update over there with progress.

[jkbrzt]

@kevin1024 thanks for looking into this. I'm able to consistently reproduce the failures and freezing with the tests in test_ssl.py also on Python 2.7.9:

$ time py.test --debug --verbose tests/test_ssl.py
writing pytestdebug information to /Users/jakub/Code/httpie/httpie/pytestdebug.log
================================================================= test session starts ==================================================================
platform darwin -- Python 2.7.9 -- py-1.4.26 -- pytest-2.6.4 -- /Users/jakub/.virtualenvs/httpie/bin/python2.7
using: pytest-2.6.4 pylib-1.4.26
setuptools registered plugins:
  pytest-cov-1.8.1 at /Users/jakub/.virtualenvs/httpie/lib/python2.7/site-packages/pytest_cov.pyc
  pytest-httpbin-0.0.5 at /Users/jakub/.virtualenvs/httpie/lib/python2.7/site-packages/pytest_httpbin/plugin.pyc
plugins: cov, httpbin
collected 10 items

tests/test_ssl.py::TestClientSSLCertHandling::test_cert_file_not_found PASSED
tests/test_ssl.py::TestClientSSLCertHandling::test_cert_file_invalid FAILED
tests/test_ssl.py::TestClientSSLCertHandling::test_cert_ok_but_missing_key FAILED
tests/test_ssl.py::TestClientSSLCertHandling::test_cert_and_key FAILED
tests/test_ssl.py::TestClientSSLCertHandling::test_cert_pem FAILED
tests/test_ssl.py::TestServerSSLCertHandling::test_self_signed_server_cert_by_default_raises_ssl_error FAILED
tests/test_ssl.py::TestServerSSLCertHandling::test_verify_no_OK FAILED
tests/test_ssl.py::TestServerSSLCertHandling::test_verify_custom_ca_bundle_path FAILED
tests/test_ssl.py::TestServerSSLCertHandling::test_verify_custom_ca_bundle_invalid_path FAILED
tests/test_ssl.py::TestServerSSLCertHandling::test_verify_custom_ca_bundle_invalid_bundle FAILED^C^\Quit: 3

real    1m40.591s
user    0m0.301s
sys 0m0.079s

Notice I had to kill it with SIGQUIT (^\).

[kevin1024]

OK I think this is fixed upstream. Bump your version to 0.0.6 and give it a try. (fingers crossed)

[kevin1024]

$ time py.test --debug --verbose tests/test_ssl.py
writing pytestdebug information to /Users/kevin/work/httpie/pytestdebug.log
================================================================ test session starts =================================================================
platform darwin -- Python 2.7.9 -- py-1.4.26 -- pytest-2.6.4 -- /Users/kevin/.virtualenvs/httpie/bin/python2.7
using: pytest-2.6.4 pylib-1.4.26
setuptools registered plugins:
  pytest-cov-1.8.1 at /Users/kevin/.virtualenvs/httpie/lib/python2.7/site-packages/pytest_cov.pyc
  pytest-httpbin-0.0.6 at /Users/kevin/.virtualenvs/httpie/lib/python2.7/site-packages/pytest_httpbin/plugin.pyc
plugins: cov, httpbin
collected 10 items

tests/test_ssl.py::TestClientSSLCertHandling::test_cert_file_not_found PASSED
tests/test_ssl.py::TestClientSSLCertHandling::test_cert_file_invalid PASSED
tests/test_ssl.py::TestClientSSLCertHandling::test_cert_ok_but_missing_key PASSED
tests/test_ssl.py::TestClientSSLCertHandling::test_cert_and_key PASSED
tests/test_ssl.py::TestClientSSLCertHandling::test_cert_pem PASSED
tests/test_ssl.py::TestServerSSLCertHandling::test_self_signed_server_cert_by_default_raises_ssl_error PASSEDTraceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 295, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 321, in process_request

tests/test_ssl.py::TestServerSSLCertHandling::test_verify_no_OK     self.finish_request(request, client_address)
  File "/Users/kevin/.virtualenvs/httpie/lib/python2.7/site-packages/pytest_httpbin/serve.py", line 68, in finish_request
    server_side=True
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 891, in wrap_socket
PASSED
tests/test_ssl.py::TestServerSSLCertHandling::test_verify_custom_ca_bundle_path PASSED
tests/test_ssl.py::TestServerSSLCertHandling::test_verify_custom_ca_bundle_invalid_path PASSED
tests/test_ssl.py::TestServerSSLCertHandling::test_verify_custom_ca_bundle_invalid_bundle ----------------------------------------
PASSED

============================================================= 10 passed in 0.56 seconds ==============================================================
wrote pytestdebug information to /Users/kevin/work/httpie/pytestdebug.log

real    0m1.195s
user    0m0.904s
sys 0m0.211s
$ python --version
Python 2.7.9

Here it is succeeding on my machine on 2.7.9.

There is an annoying exception that gets barfed to stdout by the server thread. Not sure if there's a better way to handle this but it makes the output kinda ugly. I'm open to suggestions :)