TLS Canary regression scans fail with "OneCRL sanity check failed, aborting run"

[cr]

Steps to reproduce

tlscanary regression -l 100

What's going on?

Initial investigation shows that the sanity check at https://github.com/mozilla/tls-canary/blob/master/tlscanary/modes/regression.py#L40 fails to detect any difference between enabling and disabling OCSP for connecting to its test host https://revoked.badssl.com .

The self test naively asserts that enabling OCSP shall return an SSL error, as the server's cert is supposed to be revoked, and disabling OCSP shall return no error at all, as otherwise the certificate is supposed to be fine.

The reason for the check failing is that for unknown reasons the BadSSL server has started delivering an invalid fallback certificate which produces an error in both cases:

• With OCSP: SEC_ERROR_REVOKED_CERTIFICATE
• Without OCSP: SEC_ERROR_EXPIRED_CERTIFICATE

The certificate itself does not even have a CN:

        Validity
            Not Before: Aug  8 21:17:05 2016 GMT
            Not After : Aug  8 21:17:05 2018 GMT
        Subject: C=US, ST=California, L=San Francisco, O=BadSSL Fallback. Unknown subdomain or no SNI., CN=badssl-fallback-unknown-subdomain-or-no-sni

[cr]

There are three general approaches for dealing with the situation:

• Temporarily disabling one_crl_sanity_check() by making it always return True until the server-side problem at BadSSL is resolved.
• Adding a something like a --nocheck argument (to be discussed) to the argument parser that bypasses the failing sanity check.
• Amending one_crl_sanity_check() to fully evaluate the SSL state, allowing it to deal with the new certificate situation while ensuring that the check still works when the BadSSL server returns to its previous behavior.

EDIT:

There is a highly preferable fourth option as outlined in the following comment.

[cr]

Apparently this is a known issue (chromium/badssl.com/issues/404) and the old cert expired on September 11th. The BadSSL team is working on the issue, but the process of issuing and revoking a new certificate seems to take a little longer than anticipated.

A commenter there suggested to switch to https://revoked.grc.com/ as a temporary workaround which is what I will try right now.

[cr]

Aw, no game. revoked.grc.com fails with an revocation error both with and without OCSP enabled. Looks like its revocation might be hardcoded in NSS's internal certificate database.

[cr]

Deployments should be working again due to temporary workaround landed in #207, but leaving issue open until BadSSL sorts out their cert and the workaround is removed.

[cr]

BadSSL team reports their revoked certificate fixed, but the sanity check is still seeing an SSL error in either case (default profile vs. profile updated against OneCRL). I'm suspecting that it's in the way the new cert is handled in OneCRL, as I vaguely remember that the old cert enjoyed some special treatment from our side.

@mwobensmith, since this was your project, perhaps you already get what's going on and needs to be done?

[mwobensmith]

@cr I have been following this thread casually, but not digging in to the details. If I recall, we just needed a way to determine that OneCRL worked, and I vaguely remember we needed the BadSSL use case to confirm that. I apologize that I am not of more help here, it has been a while.

[cr]

I think I finally understand what's going on. The key to the sanity check is that default Firefox minus OCSP accepts the certificate, but rejects it after OneCRL update. The issue should be that the revocation hasn't yet made it into OneCRL production stage and should resolve itself once it does. In fact Canary is seeing no error in either case, so all we need to do is wait.