Skip expired certs in partial chain hook

[voltone]

Fixes #327.

Opened as draft since I want to review the changes once more after clearing my head a bit, but it seems to work in testing...

BTW, note that on OTP 23.3.4.5 and 24.0.4 and later, partial_chain is no longer necessary at all. See https://blog.voltone.net/post/30. But I suppose it will be a good number of years before Mint can drop the partial_chain implementation.

[voltone]

Demonstrating the fix, starting by reproducing the problem described in #327:

$ docker run -it --rm hexpm/elixir:1.12.2-erlang-23.3.4-ubuntu-focal-20210325                Mon Aug 30 09:07:30 2021
root@1d8f3da3f4da:/# apt update && apt install -y faketime ca-certificates git-core
[...snip...]
done.
root@1d8f3da3f4da:/# faketime '2021-10-01 09:00' iex
Erlang/OTP 23 [erts-11.2.2] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe]

Interactive Elixir (1.12.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> Mix.install([:castore, :mint], force: true)
[...snip...]
:ok
iex(2)> isrg = File.read!("/etc/ssl/certs/ISRG_Root_X1.pem") |> :public_key.pem_decode() |> hd() |> elem(1)
<<48, 130, 5, 107, 48, 130, 3, 83, 160, 3, 2, 1, 2, 2, 17, 0, 130, 16, 207, 176,
  210, 64, 227, 89, 68, 99, 224, 187, 99, 130, 139, 0, 48, 13, 6, 9, 42, 134,
  72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 79, 49, ...>>
iex(3)> dst = File.read!("/etc/ssl/certs/DST_Root_CA_X3.pem") |> :public_key.pem_decode() |> hd() |> elem(1)
<<48, 130, 3, 74, 48, 130, 2, 50, 160, 3, 2, 1, 2, 2, 16, 68, 175, 176, 128,
  214, 163, 39, 186, 137, 48, 57, 134, 46, 248, 64, 107, 48, 13, 6, 9, 42, 134,
  72, 134, 247, 13, 1, 1, 5, 5, 0, 48, 63, 49, 36, ...>>
iex(4)> Mint.HTTP.connect(:https, "blog.voltone.net", 443, transport_opts: [reuse_sessions: false, cacerts: [isrg, dst]])

09:00:47.995 [info]  TLS :client: In state :certify at ssl_handshake.erl:1878 generated CLIENT ALERT: Fatal - Certificate Expired

{:error,
 %Mint.TransportError{
   reason: {:tls_alert,
    {:certificate_expired,
     'TLS client: In state certify at ssl_handshake.erl:1878 generated CLIENT ALERT: Fatal - Certificate Expired\n'}}
 }}

Here, Erlang/OTP selects the longest chain (with DST Root CA X3 as the root) and passes it to Mint's partial_chain. Because the public key of the first certificate matches a certificate in the trust store (the DST Root CA X3 certificate itself), Mint returns it as the trusted CA. Erlang/OTP then performs path validation with the entire chain, which fails on 23.3 or later because the first certificate in the chain has expired.

In the same container, starting a new IEx session with the branch from this PR:

root@1d8f3da3f4da:/# faketime '2021-10-01 09:00' iex
Erlang/OTP 23 [erts-11.2.2] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:1] [hipe]

Interactive Elixir (1.12.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> Mix.install([:castore, {:mint, git: "https://github.com/voltone/mint", branch: "partial-chain-skip-expired"}], force: true)
[...snip...]
:ok
iex(2)> isrg = File.read!("/etc/ssl/certs/ISRG_Root_X1.pem") |> :public_key.pem_decode() |> hd() |> elem(1)
<<48, 130, 5, 107, 48, 130, 3, 83, 160, 3, 2, 1, 2, 2, 17, 0, 130, 16, 207, 176,
  210, 64, 227, 89, 68, 99, 224, 187, 99, 130, 139, 0, 48, 13, 6, 9, 42, 134,
  72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 79, 49, ...>>
iex(3)> dst = File.read!("/etc/ssl/certs/DST_Root_CA_X3.pem") |> :public_key.pem_decode() |> hd() |> elem(1)
<<48, 130, 3, 74, 48, 130, 2, 50, 160, 3, 2, 1, 2, 2, 16, 68, 175, 176, 128,
  214, 163, 39, 186, 137, 48, 57, 134, 46, 248, 64, 107, 48, 13, 6, 9, 42, 134,
  72, 134, 247, 13, 1, 1, 5, 5, 0, 48, 63, 49, 36, ...>>
iex(4)> Mint.HTTP.connect(:https, "blog.voltone.net", 443, transport_opts: [reuse_sessions: false, cacerts: [isrg, dst]])
{:ok,
 %Mint.HTTP1{
  # ...snip...
 }}

Now Mint skips over the expired DST Root CA X3 in the chain, and starts looking for a trusted CA from the cross-signed ISRG Root X1 CA sent by the server. Since the public key of this certificate matches the ISRG Root X1 public key in the trust store, Mint returns it as the trusted CA. This time Erlang/OTP is happy with the (shortened) chain, since all certificates are valid.

Verification still works once the DST Root CA X3 certificate is removed from the trust store:

iex(5)> Mint.HTTP.connect(:https, "blog.voltone.net", 443, transport_opts: [reuse_sessions: false, cacerts: [isrg]]) {:ok,
 %Mint.HTTP1{
  # ...snip...
 }}

[ericmj]

This is great, thank you!

I have done an initial review and it looks good. Let us know when it's ready for final review.

💜💜💜💜💜💜

[voltone]

Let us know when it's ready for final review.

I suppose it is. I will run a few more manual tests like the one above on other OTP versions, just to be sure. Should be able to do that later today...

[voltone]

Ok, additional testing with the following OTP versions was successful:

• 23.2.7 (pre-23.3)
• 23.3.4.5 (with alternate chain support in OTP)
• 24.0.3 (the 24.x equivalent of 23.3.4 as tested above)
• 24.0.4 (with alternate chain support in OTP)

Of these versions, only 24.0.3 requires the fix in this PR. The other versions do not require the fix; they were tested for regressions.

[whatyouhide]

@voltone did you do the testing manually? Do you think it would be beneficial to add those tests on those versions to CI?

[voltone]

Yes, the end-to-end tests were done manually, using my blog as the server. Testing them in CI would require a stable endpoint that presents a suitable chain, and that will continue to do so indefinitely. Most servers with a Let's Encrypt certificate would probably work, at least for the time being, however:

1. For tests to work between now and Oct 1st, the use of faketime would be needed
2. If faketime is used after Oct 1st, at some point the server cert will become invalid because it will have been issued in the future

How about I create a PR after Oct 1st that will check for regressions, without faketime?

[ericmj]

How about I create a PR after Oct 1st that will check for regressions, without faketime?

That sounds like the most practical solution.

[voltone]

Hey @ericmj @whatyouhide, will you be publishing a new release of Mint before the end of the month, in case people need it to resolve issues as a result of the DST Root CA expiry?

[ericmj]

Done!

Thanks again for your PR and research into this issue! 💜