x509 module without sha1 dependency

[securitykernel]

Our high-level goal is to be able to use the BSI module policy and additionally enable the tls module, but leave the sha1 module disabled. This is currently not possible, as tls depends on x509, which depends on sha1. From looking at the code, it seems the x509 module already handles non-existence of the sha1 module by throwing, e.g., at runtime in X509Cert::subject_public_key_bitstring_sha1(). This function seems to be used by Flatfile_Certificate_Store, which we could potentially replace. Also, ocsp relies on subject_public_key_bitstring_sha1(). Additionally, the CertificateStore interface and implementations rely on it via CertificateStore::find_cert_by_pubkey_sha1(), I am not sure what the consequences would be here.

Temporarily removing sha1 from x509's info.txt builds just fine, but unsurprisingly some tests fail:

$ ./configure.py --module-policy=bsi --enable-modules=tls
...
make check
...
Tests complete ran 171854 tests in 23.68 sec 129 tests failed (in ecdsa_unit ocsp x509_path_name_constraint x509_path_nist x509_path_rsa_pss x509_path_with_ocsp x509_path_x509test x509_v1_ca)

Probably we can replace most of the test certificates using SHA-1 with SHA-256 ones, that depends on the source. I will check these.

[randombit]

I think one issue is that OCSP relies on SHA-1 as a key identifier, though I don't recall the details at the moment.

Absolutely in favor of any changes which allow disabling SHA-1 in more configurations.

Possibly for OCSP we can use some different key identifier mechanism instead.

[securitykernel]

So from what I can see, the following things need to be done to fix the tests:

• [x] Fix ecdsa_unit test: Check whether SHA-1 is available
• [ ] Fix ocsp tests: Possibly we can use SHA-256 here, need to check
• [ ] Fix x509_path_name_constraint: xxx
• [ ] Fix x509_path_nist: ~Hard without NIST's help, "All certificates are signed with RSA PKCS#1 v1.5 and SHA-1", can we get NIST to update their test suite to use SHA-256?~ Update to NIST PKITS (#1047)
• [ ] Fix x509_path_rsa_pss: Certificates from different sources, some already use SHA-256, some still SHA-1
• [ ] Fix x509_path_with_ocsp: See ocsp
• [ ] x509_path_x509test certificates with SHA-256: https://github.com/yymax/x509test hasn't been updated since 8 years ago and does not work out of the box with python 3.9, maybe we can use an older Ubuntu container image
• [ ] Fix x509_v1_ca: I am skeptical that we find a root that signs with SHA-256 but uses a v1 certificate, but I managed to create one using OpenSSL command line tool, so we could replace the Verisign Root currently used with a custom one

[securitykernel]

Re OCSP (via Feisty Duck TLS News):

The CA/Browser Forum has decided to deprecate the use of SHA-1 signatures in OCSP signing. SHA-1 signatures for certificates were deprecated in the web PKI a long time ago, but in OCSP they were still allowed. Some further discussion around this issue can be found in the bug tracker of the Go x509 module. The module’s developer, Filippo Valsorda, recently tried to disable SHA-1 signatures and had to re-enable them for OCSP.

[randombit]

It's not just the signatures either - IIRC, OCSP uses the SHA-1 of the certificate as a key identifier.

Oh, or looking at this code again, perhaps it just was "required" by the ecosystem at the time I wrote the OCSP code? It may be we can start using SHA-256 here, given that this comment dates back to 2012.