Failed to verify Letsencrypt cert with ISRG Root X1 CA (IDFGH-11039)

[jpmeijers]

Summary

On the ESP32 (I'm using C3), when trying to use SSL/TLS over WiFi to connect to a server - in my case using a Letsencrypt certificate - the certificate validation fails when using a certificate bundle.

This error is also mentioned by other users, for example here.

[E][esp_crt_bundle.c:147] esp_crt_verify_callback(): Failed to verify certificate followed by [E][ssl_client.cpp:37] _handle_error(): [start_ssl_client():273]: (-12288) X509 - A fatal error occurred, eg the chain is too long or the vrfy callback failed

The problem seems to be in this function: https://github.com/espressif/esp-idf/blob/3640dc86bb4b007da0c53500d90e318f0b7543ef/components/mbedtls/esp_crt_bundle/esp_crt_bundle.c#L84

I'm using the Arduino WiFiClientSecure to test. Two servers: Letsencrypt valid cert test server, and my own server.

Using client.setCACert(isrgx1ca);

First test hardcoding the ISRG Root X1 CA pem in the sketch.

On https://valid-isrgrootx1.letsencrypt.org/

[  3369][V][ssl_client.cpp:290] start_ssl_client(): Verifying peer X.509 certificate...
[  3369][V][ssl_client.cpp:298] start_ssl_client(): Certificate verified.
[  3372][V][ssl_client.cpp:313] start_ssl_client(): Free internal heap after TLS 188492
Connected to server!

On my server

[  3781][V][ssl_client.cpp:290] start_ssl_client(): Verifying peer X.509 certificate...
[  3781][V][ssl_client.cpp:298] start_ssl_client(): Certificate verified.
[  3784][V][ssl_client.cpp:313] start_ssl_client(): Free internal heap after TLS 184632
Connected to server!

Using client.setCACertBundle(rootca_crt_bundle_start);

On https://valid-isrgrootx1.letsencrypt.org/

[  2393][D][esp_crt_bundle.c:108] esp_crt_verify_callback(): 141 certificates in bundle
[  2559][I][esp_crt_bundle.c:142] esp_crt_verify_callback(): Certificate validated
[  3497][V][ssl_client.cpp:290] start_ssl_client(): Verifying peer X.509 certificate...
[  3497][V][ssl_client.cpp:298] start_ssl_client(): Certificate verified.
[  3500][V][ssl_client.cpp:313] start_ssl_client(): Free internal heap after TLS 187900
Connected to server!

On my server

[  2868][D][esp_crt_bundle.c:108] esp_crt_verify_callback(): 141 certificates in bundle
[  2868][E][esp_crt_bundle.c:147] esp_crt_verify_callback(): Failed to verify certificate
[  2874][E][ssl_client.cpp:37] _handle_error(): [start_ssl_client():273]: (-12288) X509 - A fatal error occurred, eg the chain is too long or the vrfy callback failed
[  2887][E][WiFiClientSecure.cpp:144] connect(): start_ssl_client: -12288
[  2894][V][ssl_client.cpp:321] stop_ssl_socket(): Cleaning SSL connection.
Connection failed!

Cert bundle

Following the steps here. Cert bundle downloaded from curl. It's the Mozilla bundle that contains the ISRG Root X1 cert.

Server certs

Both the test server and my server report to have certificates that are signed by ISRG Root X1, with the cert serial numbers matching the screenshot above.

Test server:

My server:

[dhalbert]

You may also need the (expired) DST Root CA X3 cert in your roots list. We just went through this same issue: https://github.com/adafruit/certificates/pull/1.

This problem is Let's Encrypt specific. See https://community.letsencrypt.org/t/production-chain-changes/150739 for background.

[jpmeijers]

@dhalbert Thanks for the response and your clear explanation in the linked Github issue. I am pretty sure you are correct, but what is still confusing me is why does https://valid-isrgrootx1.letsencrypt.org/ work, even though it uses the same chain as I am using on my server?

The original issue that I tried to understand that mentions the DST Root X3 is here: https://github.com/espressif/esp-idf/issues/11077

Because of the above issue I have updated to the latest platform-espressif32@6.4.0, that uses ESP IDF 5.1.1. That didn't fix the problem by itself. I will try and understand how the cacrt_local.pem file should be used and report back.

@Alvin1Zhang because you transfered the issue: At this point my feeling is still that the function in ESP-IDF that I linked in my original post (esp_crt_verify_callback) is the root cause for all these issues. It's not an Arduino issue. Shouldn't this function correctly verify the intermediate certificate with the ISRG Root X1 in the cert bundle? And then stop there and not try and verify ISRG Root X1 itself, because it was in the list of trusted certs.

[Alvin1Zhang]

@jpmeijers Thanks for sharing the updates, would you please help share IDF based sample code to recreate the issue? Thanks.

[dhalbert]

what is still confusing me is why does https://valid-isrgrootx1.letsencrypt.org/ work, even though it uses the same chain as I am using on my server?

The cert chain used by https://valid-isrgrootx1.letsencrypt.org/ does not use the cross-signed ISRG Root X1. Instead it uses this chain, with the self-signed ISRG Root X1.:

• server-cert <- Let's Encrypt R3 intermediate <- self-signed ISRG Root X1 (as supplied by your local roots list, not returned by the server)

This contrasts with a typical Let's Encrypt chain, which uses the cross-signed version:

• server-cert <- Let's Encrypt R3 intermediate <- cross-signed ISRG Root X1, signed with DST Root CA X3

You can see the difference by looking at the output of these two commands:

• uses the self-sign ISRG Root X1:

  openssl s_client -showcerts -connect valid-isrgrootx1.letsencrypt.org:443 < /dev/null

• vs this typical Let's Encrypt site (your site would be the same, I think), which uses the cross-signed DST root CA X3:

  openssl s_client -showcerts -connect site.api.espn.com:443 < /dev/null

[dhalbert]

I'd like to note also that it seems to be an mbedtls issue that the ISRG Root X1 is not handled correctly as a trust anchor when it is cross-signed and also in the local roots cert list. It should not be necessary to include DST Root CA X3 in the roots list. For instance, that expired root cert is not in the Mozilla list (see https://curl.se/docs/caextract.html), because most TLS implementations do the right thing.

We needed to include DST Root CA X3 in our root cert list both for Espressif and also for the Pi Pico W. Both use mbedtls.

[lodemo]

I encountered the exact same issue today (on Espressif32 5.2.0) with WifiClientSecure.

After checking the server certificate chain it was also using the cross-signed chain. Adding the DST Root CA X3 cert to the bundle fixed it. Thanks for that!

Wouldn't it be a good idea to include both the self signed ISRG Root X1 and the cross-signed ISRG Root X1 + DST Root CA X3 in the bundle, to be prepared when Lets Encrypt switches to the self-signed chain (at the server) at some point in the future? (at least as long as the mbedtls issue isnt fixed)

[dhalbert]

@lodemo The special cross-signed ISRG Root X1 is supplied by the server. It's not in any root certs list that I know of. The self-signed ISRG Root X1 should be in the roots cert list. https://letsencrypt.org/2023/07/10/cross-sign-expiration describes what Let's Encrypt will do as the cross-signing expiration date (September 2024) approaches.

Quoting:

• On Thursday, Feb 8th, 2024, we will stop providing the cross-sign by default in requests made to our /acme/certificate API endpoint. For most Subscribers, this means that your ACME client will configure a chain which terminates at ISRG Root X1, and your webserver will begin providing this shorter chain in all TLS handshakes. The longer chain, terminating at the soon-to-expire cross-sign, will still be available as an alternate chain which you can configure your client to request.
• On Thursday, June 6th, 2024, we will stop providing the longer cross-signed chain entirely. This is just over 90 days (the lifetime of one certificate) before the cross-sign expires, and we need to make sure subscribers have had at least one full issuance cycle to migrate off of the cross-signed chain.
• On Monday, September 30th, 2024, the cross-signed certificate will expire. This should be a non-event for most people, as any client breakages should have occurred over the preceding six months.

So after June 6, 2024, only certs supplied by the server will be the base server cert and the R3 intermediate cert. The DST Root CA X3 will no longer be mentioned at all in the chain. As long as the client has the self-signed ISRG Root X1 in its root list, the chain will work. At that point you could drop DST Root CA X3 from your roots list to save space, because no Let's Encrypt chains will be mentioning it.

[dzungpv]

I have the same problem, trying the sample pre_encrypted_ota with the letsenrypt cert not working. Try local self sign working fine. I still don't know how to fix, the sample working fine with server like https://www.howsmyssl.com, but not with letsencrypt.