Spurious acme coverage failures

[bmw]

I've seen this a couple times now. We should fix this.

Name                           Stmts   Miss  Cover   Missing
------------------------------------------------------------
acme.py                            0      0   100%
acme/challenges.py               194      0   100%
acme/challenges_test.py          282      0   100%
acme/client.py                   233      0   100%
acme/client_test.py              347      0   100%
acme/crypto_util.py               92      2    98%   134-135
acme/crypto_util_test.py          79      0   100%
acme/dns_resolver.py              16      0   100%
acme/dns_resolver_test.py         36      0   100%
acme/errors.py                    29      0   100%
acme/errors_test.py               28      0   100%
acme/fields.py                    32      0   100%
acme/fields_test.py               41      0   100%
acme/jose.py                       8      0   100%
acme/jose/b64.py                  15      0   100%
acme/jose/b64_test.py             38      0   100%
acme/jose/errors.py               12      0   100%
acme/jose/errors_test.py           8      0   100%
acme/jose/interfaces.py           39      0   100%
acme/jose/interfaces_test.py      71      0   100%
acme/jose/json_util.py           170      0   100%
acme/jose/json_util_test.py      214      0   100%
acme/jose/jwa.py                 105      0   100%
acme/jose/jwa_test.py             58      0   100%
acme/jose/jwk.py                 114      0   100%
acme/jose/jwk_test.py            101      0   100%
acme/jose/jws.py                 205      0   100%
acme/jose/jws_test.py            146      0   100%
acme/jose/util.py                114      0   100%
acme/jose/util_test.py           128      0   100%
acme/jws.py                       17      0   100%
acme/jws_test.py                  27      0   100%
acme/messages.py                 174      0   100%
acme/messages_test.py            193      0   100%
acme/standalone.py                94      0   100%
acme/standalone_test.py           93      3    97%   145-147
acme/test_util.py                 33      0   100%
acme/util.py                      11      0   100%
acme/util_test.py                 17      0   100%
------------------------------------------------------------
TOTAL                           3614      5    99%
nose.plugins.cover: ERROR: TOTAL Coverage did not reach minimum required: 100%

In case line number changes, here's the relevant lines with some extra padding around them for additional context:

$ sed -n 129,140p crypto_util.py
        'source_address': source_address}

    try:
        # pylint: disable=star-args
        sock = socket.create_connection((host, port), **socket_kwargs)
    except socket.error as error:
        raise errors.Error(error)

    with contextlib.closing(sock) as client:
        client_ssl = OpenSSL.SSL.Connection(context, client)
        client_ssl.set_connect_state()
        client_ssl.set_tlsext_host_name(name)  # pyOpenSSL>=0.13

$ sed -n 140,152p standalone_test.py
        while max_attempts:
            max_attempts -= 1
            try:
                cert = crypto_util.probe_sni(
                    b'localhost', b'0.0.0.0', self.port)
            except errors.Error:
                self.assertTrue(max_attempts > 0, "Timeout!")
                time.sleep(1)  # wait until thread starts
            else:
                self.assertEqual(jose.ComparableX509(cert),
                                 test_util.load_comparable_cert('cert.pem'))
                break

[bmw]

This just happened in Travis 3 times in a row: log

[schuyler1d]

working on this at PyCon2017. A more recent failure is here: https://travis-ci.org/certbot/certbot/jobs/232619237

[SwartzCr]

@schuyler1d did this ever make it's way into a PR? iirc @paulproteus was worked on this with you, is that right?

[schuyler1d]

Yes, it's here though just saw the feedback. Can fix later today

[schuyler1d]

https://github.com/certbot/certbot/pull/4712