Inclusion of approximately 190000 email protection certificates into the test corpus

[christopher-henderson]

The ZLint project has begun writing lints for email protection certificates (https://github.com/zmap/zlint/pull/713) so it would appropriate to have a wide corpus for testing.

These certs were pulled from Censys using the following query.

SELECT fingerprint_sha256, raw FROM `censys-io.certificates_v2.certificates` 
WHERE (parsed.extensions.extended_key_usage.any = true 
or parsed.extensions.extended_key_usage.email_protection = true
or parsed.extensions.extended_key_usage is null)
and validation.nss.ever_valid = true

Aside from reformatting and sanitization, I have also gone through the task of deduplicating these certificates against the corpus because (as it turns out) there are about 1500 certs that are both email protection certs and server auth certs.

[cardonator]

No concerns with this having been merged. I am curious if the corpus that was pulled from Censys is compatible with the SMIME BRs that go into effect on Sept 1. Is there a good way to determine that?

[christopher-henderson]

@cardonator does this address the compatibility concerns you are referring to?

This query was constructed based off the boolean logic that was agreed upon in util.IsEmailProtectionCert

func IsEmailProtectionCert(cert *x509.Certificate) bool {
    if len(cert.ExtKeyUsage) == 0 {
        return true
    }
    for _, eku := range cert.ExtKeyUsage {
        if eku == x509.ExtKeyUsageAny || eku == x509.ExtKeyUsageEmailProtection {
            return true
        }
    }
    return false
}

That is..

• No EKUs present at all, or...
• The presence of EKU Any ({2, 5, 29, 37, 0}), or...
• The presence of EKU Email Protection ({1, 3, 6, 1, 5, 5, 7, 3, 4})

[cardonator]

@christopher-henderson logically I think what you have done is sound. The concern I have is that a significant corpus of these additions probably would not be compliant once zlint has a full set of SMIME checks. Perhaps that is what we would expect, but newly generated SMIME certs posted to CT post-Sept 1 will need to conform to the BR profiles where they don't currently need to.