search

Mbed-TLS / mbedtls

An open source, portable, easy to use, readable and flexible TLS library, and reference implementation of the PSA Cryptography API. Releases are on a varying cadence, typically around 3 - 6 months between releases.
https://www.trustedfirmware.org/projects/mbed-tls/
Other
5.21k stars 2.55k forks source link

tests fail after 2027-06-23 #6978

Open bmwiedemann opened 1 year ago

bmwiedemann commented 1 year ago

Summary

While working on reproducible builds for openSUSE, I found that in our mbedtls-2.28.2 package, similar to #2357 , SSL tests start to fail after 2023-08-07.

System information

Mbed TLS version (number or commit id): 2.28.2 Operating system and version: openSUSE-Tumbleweed-20230126 Configuration (if not default, please attach mbedtls_config.h): Compiler and options (if you used a pre-built binary, please indicate how you obtained it): https://code.opensuse.org/package/mbedtls/blob/master/f/mbedtls.spec#_97 Additional environment information:

Expected behavior

Tests should continue to pass at least 16 years into the future.

Actual behavior

98% tests passed, 2 tests failed out of 103
 The following tests FAILED:
         98 - ssl-suite (Failed)
        101 - x509parse-suite (Failed)

Steps to reproduce

On Debian or openSUSE do:

osc co openSUSE:Factory/mbedtls && cd $_
osc build --vm-type=kvm --noservice --build-opt=--vm-custom-opt="-rtc base=2023-08-08T12:00:00" standard

or set the system clock or use libfaketime to run the tests.

Additional information

tests/data_files/test-ca2_cat-present-future.crt has expiry of Aug 7 09:17:03 2023 GMT and many more expire in September.

Background: As part of my work on reproducible builds for openSUSE, I check that software still gives identical build results in the future. The usual offset is +16 years, because that is how long I expect some software will be used in some places. This showed up failing tests in our package build. See https://reproducible-builds.org/ for why this matters.

gilles-peskine-arm commented 1 year ago

Thank you for reporting this!

The build of the library and the test programs should be reproducible. (Do please let us know if it isn't — we are not currently testing this in our CI.) However, the tests are unavoidably not fully reproducible because there are dates in the test data. We cannot generate the test data based on the current date (it's far too complex), and we cannot always force a date when testing (it's not possible in all environments).

Goal of this issue: fix the 2023 expiry. We'll test manually. I've filed a separate issue to address the requirement of having tests pass 16 years in the future.

gilles-peskine-arm commented 1 year ago

The list of test cases that will fail on 07 August 2023 in the full config as of fd13a0f85195cc5cf7e0d650713935b88a28ac52:

From test_suite_pkcs7:

PKCS7 Signed Data Verification Pass SHA256 #9 ..................... FAILED
PKCS7 Signed Data Verification Pass SHA256 #9.1 ................... FAILED
PKCS7 Signed Data Verification Pass SHA1 #10 ...................... FAILED
PKCS7 Signed Data Verification Pass SHA512 #11 .................... FAILED
PKCS7 Signed Data Verification Fail because of different certifica  FAILED
PKCS7 Signed Data Verification Fail because of different data hash  FAILED
PKCS7 Signed Data Verify with multiple signers #16 ................ FAILED
PKCS7 Signed Data Hash Verify with multiple signers #17 ........... FAILED

From test_suite_ssl:

Handshake, ECDHE-ECDSA-WITH-AES-256-CCM ........................... FAILED
Handshake, ECDH-ECDSA-WITH-CAMELLIA-256-CBC-SHA384 ................ FAILED
DTLS Handshake, ECDHE-ECDSA-WITH-AES-256-CCM ...................... FAILED
DTLS Handshake, ECDH-ECDSA-WITH-CAMELLIA-256-CBC-SHA384 ........... FAILED
Handshake, select ECDHE-ECDSA-WITH-AES-256-CCM, non-opaque ........ FAILED
Handshake, select ECDHE-ECDSA-WITH-AES-256-CCM, opaque, PSA_ALG_AN  FAILED
Handshake, select ECDHE-ECDSA-WITH-AES-256-CCM, opaque, PSA_ALG_SH  FAILED
Handshake, select ECDH-RSA-WITH-AES-256-CBC-SHA384, non-opaque .... FAILED
Handshake, select ECDH-RSA-WITH-AES-256-CBC-SHA384, opaque ........ FAILED
Handshake, select ECDH-ECDSA-WITH-CAMELLIA-256-CBC-SHA384, non-opa  FAILED
Handshake, select ECDH-ECDSA-WITH-CAMELLIA-256-CBC-SHA384, opaque,  FAILED
Handshake, select ECDH-ECDSA-WITH-CAMELLIA-256-CBC-SHA384, opaque,  FAILED
Raw key agreement: nominal ........................................ FAILED
Raw key agreement: bad server key ................................. FAILED

From test_suite_x509parse:

X509 Time Future #2 ............................................... FAILED
X509 CRT verification #1a (Revoked Cert, Future CRL, no CN) ....... FAILED
X509 CRT verification #2a (Revoked Cert, Future CRL) .............. FAILED
X509 CRT verification #3a (Revoked Cert, Expired CRL, CN Mismatch)  FAILED
X509 CRT verification #4a (Revoked Cert, Future CRL) .............. FAILED
X509 CRT verification #8 (Valid Cert) ............................. FAILED
X509 CRT verification #8a (Expired Cert) .......................... FAILED
X509 CRT verification #8b (Future Cert) ........................... FAILED
X509 CRT verification #8c (Expired Cert, longer chain) ............ FAILED
X509 CRT verification #8d (Future Cert, longer chain) ............. FAILED
X509 CRT verification #30 (domain matching multi certificate witho  FAILED
X509 CRT verification #31 (domain not matching multi certificate w  FAILED
X509 CRT verification #32 (Valid, EC cert, RSA CA) ................ FAILED
X509 CRT verification #33 (Valid, RSA cert, EC CA) ................ FAILED
X509 CRT verification #34 (Valid, EC cert, EC CA) ................. FAILED
X509 CRT verification #35 (Revoked, EC CA) ........................ FAILED
X509 CRT verification #36 (Valid, EC CA, SHA1 Digest) ............. FAILED
X509 CRT verification #37 (Valid, EC CA, SHA224 Digest) ........... FAILED
X509 CRT verification #38 (Valid, EC CA, SHA384 Digest) ........... FAILED
X509 CRT verification #39 (Valid, EC CA, SHA512 Digest) ........... FAILED
X509 CRT verification #43 (Depth 0, not CA, EC) ................... FAILED
X509 CRT verification #44 (Corrupted signature, EC) ............... FAILED
X509 CRT verification #45b (Corrupted signature, intermediate CA) . FAILED
X509 CRT verification #46 (Valid, depth 2, EC-RSA-EC) ............. FAILED
X509 CRT verification #47 (Untrusted, depth 2, EC-RSA-EC) ......... FAILED
X509 CRT verification #48 (Missing intermediate CA, EC-RSA-EC) .... FAILED
X509 CRT verification #49 (Valid, depth 2, RSA-EC-RSA) ............ FAILED
X509 CRT verification #52 (CA keyUsage valid) ..................... FAILED
X509 CRT verification #53 (CA keyUsage missing cRLSign) ........... FAILED
X509 CRT verification #54 (CA keyUsage missing cRLSign, no CRL) ... FAILED
X509 CRT verification #55 (CA keyUsage missing keyCertSign) ....... FAILED
X509 CRT verification #56 (CA keyUsage plain wrong) ............... FAILED
X509 CRT verification #57 (Valid, RSASSA-PSS, SHA-1) .............. FAILED
X509 CRT verification #58 (Valid, RSASSA-PSS, SHA-224) ............ FAILED
X509 CRT verification #59 (Valid, RSASSA-PSS, SHA-256) ............ FAILED
X509 CRT verification #60 (Valid, RSASSA-PSS, SHA-384) ............ FAILED
X509 CRT verification #61 (Valid, RSASSA-PSS, SHA-512) ............ FAILED
X509 CRT verification #62 (Revoked, RSASSA-PSS, SHA-1) ............ FAILED
X509 CRT verification #63 (Revoked, RSASSA-PSS, SHA-1, CRL badsign  FAILED
X509 CRT verification #64 (Valid, RSASSA-PSS, SHA-1, not top) ..... FAILED
X509 CRT verification #65 (RSASSA-PSS, SHA1, bad cert signature) .. FAILED
X509 CRT verification #66 (RSASSA-PSS, SHA1, no RSA CA) ........... FAILED
X509 CRT verification #67 (Valid, RSASSA-PSS, all defaults) ....... FAILED
X509 CRT verification #68 (RSASSA-PSS, wrong salt_len, USE_PSA) ... FAILED
X509 CRT verification #69 (RSASSA-PSS, wrong mgf_hash) ............ FAILED
X509 CRT verification #70 (v1 trusted CA) ......................... FAILED
X509 CRT verification #71 (v1 trusted CA, other) .................. FAILED
X509 CRT verification #72 (v1 chain) .............................. FAILED
X509 CRT verification #73 (selfsigned trusted without CA bit) ..... FAILED
X509 CRT verification #74 (signed by selfsigned trusted without CA  FAILED
X509 CRT verification #76 (multiple CRLs, not revoked) ............ FAILED
X509 CRT verification #77 (multiple CRLs, revoked) ................ FAILED
X509 CRT verification #78 (multiple CRLs, revoked by second) ...... FAILED
X509 CRT verification #79 (multiple CRLs, revoked by future) ...... FAILED
X509 CRT verification #82 (Not yet valid CA and valid CA) ......... FAILED
X509 CRT verification #83 (valid CA and Not yet valid CA) ......... FAILED
X509 CRT verification #84 (valid CA and Not yet valid CA) ......... FAILED
X509 CRT verification #85 (Not yet valid CA and valid CA) ......... FAILED
X509 CRT verification #86 (Not yet valid CA and invalid CA) ....... FAILED
X509 CRT verification #87 (Expired CA and invalid CA) ............. FAILED
X509 CRT verification #88 (Spurious cert in the chain) ............ FAILED
X509 CRT verification #89 (Spurious cert later in the chain) ...... FAILED
X509 CRT verification #90 (EE with same name as trusted root) ..... FAILED
X509 CRT verification #93 (Suite B invalid, EC cert, RSA CA) ...... FAILED
X509 CRT verification #94 (Suite B invalid, RSA cert, EC CA) ...... FAILED
X509 CRT verification #95 (Suite B Valid, EC cert, EC CA) ......... FAILED
X509 CRT verification callback: bad name .......................... FAILED
X509 CRT verification callback: trusted EE cert ................... FAILED
X509 CRT verification callback: simple, root expired .............. FAILED
X509 CRT verification callback: intermediate ca ................... FAILED
X509 CRT verification callback: intermediate ca, root included .... FAILED
X509 CRT verification callback: intermediate ca trusted ........... FAILED
X509 CRT verification callback: intermediate ca, EE expired ....... FAILED
X509 CRT verification callback: intermediate ca, int expired ...... FAILED
X509 CRT verification callback: intermediate ca, root expired ..... FAILED
X509 CRT verification callback: two intermediates ................. FAILED
X509 CRT verification callback: two intermediates, root included .. FAILED
X509 CRT verification callback: two intermediates, top int trusted  FAILED
X509 CRT verification callback: two intermediates, low int trusted  FAILED
X509 CRT verification callback: no intermediate, bad signature .... FAILED
X509 CRT verification callback: one intermediate, bad signature ... FAILED
X509 CRT verify long chain (max intermediate CA, trusted) ......... FAILED
X509 CRT verify long chain (max intermediate CA, untrusted) ....... FAILED
X509 CRT verify chain #12 (suiteb profile, RSA root) .............. FAILED
X509 CRT verify chain #13 (RSA only profile, EC root) ............. FAILED
X509 CRT verify chain #13 (RSA only profile, EC trusted EE) ....... FAILED
X509 CRT verify chain #15 (suiteb profile, rsa intermediate) ...... FAILED
X509 CRT verify chain #16 (RSA-only profile, EC intermediate) ..... FAILED
X509 CRT verify chain #17 (SHA-512 profile) ....................... FAILED
X509 CRT verify restart: trusted EE, max_ops=0 (disabled) ......... FAILED
X509 CRT verify restart: trusted EE, max_ops=1 .................... FAILED
X509 CRT verify restart: no intermediate, max_ops=0 (disabled) .... FAILED
X509 CRT verify restart: no intermediate, max_ops=1 ............... FAILED
X509 CRT verify restart: no intermediate, max_ops=40000 ........... FAILED
X509 CRT verify restart: no intermediate, max_ops=500 ............. FAILED
X509 CRT verify restart: no intermediate, badsign, max_ops=0 (disa  FAILED
X509 CRT verify restart: no intermediate, badsign, max_ops=1 ...... FAILED
X509 CRT verify restart: no intermediate, badsign, max_ops=40000 .. FAILED
X509 CRT verify restart: no intermediate, badsign, max_ops=500 .... FAILED
X509 CRT verify restart: one int, max_ops=0 (disabled) ............ FAILED
X509 CRT verify restart: one int, max_ops=1 ....................... FAILED
X509 CRT verify restart: one int, max_ops=30000 ................... FAILED
X509 CRT verify restart: one int, max_ops=500 ..................... FAILED
X509 CRT verify restart: one int, EE badsign, max_ops=0 (disabled)  FAILED
X509 CRT verify restart: one int, EE badsign, max_ops=1 ........... FAILED
X509 CRT verify restart: one int, EE badsign, max_ops=30000 ....... FAILED
X509 CRT verify restart: one int, EE badsign, max_ops=500 ......... FAILED
X509 CRT verify restart: one int, int badsign, max_ops=0 (disabled  FAILED
X509 CRT verify restart: one int, int badsign, max_ops=1 .......... FAILED
X509 CRT verify restart: one int, int badsign, max_ops=30000 ...... FAILED
X509 CRT verify restart: one int, int badsign, max_ops=500 ........ FAILED

Many of them are copies of the same few certificates (one RSA and one ECC?) used in different context. But quite a few are manually modified to exercise error detection in parsers.

bmwiedemann commented 1 year ago

For other test-suites we created certs that expire in 2999, because these are not meant to expire (unlike regular production-grade certification). Would that work for you?

gnutls' certtool -u can update certs to an expiry of 2049-12-31 (but not beyond yet because of some format change there)

gilles-peskine-arm commented 1 year ago

Changing to make our certs expire in 2999 would be great if it works. I don't know if it's possible: I don't know if everything we're using supports dates that far in the future. (We don't even have tests for Y2038!) This includes interop tests with old versions of OpenSSL and GnuTLS.

The difficulty for us with updating certificates is that since we're an X.509 library, we have a lot of negative test cases, many of them crafted by manually tweaking some nominal data. Updating the files containing nominal data is only the tip of the iceberg.

bmwiedemann commented 1 year ago

An alternative solution could be to provide a date to the validation functions that is used instead of the current date. Thus such a test would be about "a cert that expires in 2023 is valid in 2022" independent of the system clock, so it keeps passing even after the current year moves to 2024. This has the advantage that you do not need to recreate hand-crafted certs.

A date-override could even have advantages in production builds. E.g. when you run as non-root on a system with incorrect system time and still want SSL validation.

bmwiedemann commented 1 year ago

Hi. What is the status of the fix for this issue? Only one week left until 2023-08-07

gilles-peskine-arm commented 1 year ago

We've regenerated all the test certificates that expired in 2023. We're keeping this issue open because we're still working on an alert mechanism and on regenerating certificates that will expire in the next few years.

gilles-peskine-arm commented 1 year ago

For the sake of continuing to have passing tests in third-party packages that ship Mbed TLS such as Linux distributions and BSP, we're going to make official releases today or tomorrow (right now we're making some final preparations):

gilles-peskine-arm commented 1 year ago

The releases are out: 3.4.1 and 2.28.4. Their tests should be good until January 2024, and in a month or two we're planning a new release with tests that are good until at least mid-2027. Incremental progress!

bmwiedemann commented 1 year ago

Thanks a lot. I still have hope, that some day, tests will continue to pass forever.

gilles-peskine-arm commented 10 months ago

As of 2.28.5 and 3.5.0, the next certificate expiry is on 2027-06-23 (dir-maxpath/*.crt).