Unexpected Bad Certificate errors

[1player]

Apologies if this is not the right place for this issue. I'm using this library through Mint and Req, and some websites are throwing TLS client: In state wait_cert_cr at ssl_handshake.erl:2121 generated CLIENT ALERT: Fatal - Bad Certificate errors even when the certificate is known to work. Over the past few days I've collected two URLs that fail this way:

• https://www.kb.cert.org/vuls/id/836068
• https://people.bath.ac.uk

Example, using Req:

iex(8)> Req.get("https://www.kb.cert.org/vuls/id/836068")
[notice] TLS :client: In state :wait_cert_cr at ssl_handshake.erl:2121 generated CLIENT ALERT: Fatal - Bad Certificate

{:error,
 %Mint.TransportError{
   reason: {:tls_alert,
    {:bad_certificate,
     ~c"TLS client: In state wait_cert_cr at ssl_handshake.erl:2121 generated CLIENT ALERT: Fatal - Bad Certificate\n"}}
 }}

These URL and certificate are known to work on Firefox, and cURL. Who's at fault here? I tried pulling this repo and running mix certdata, and it's up-to-date, and of course I made sure all my deps updated as well.

[YgorCastor]

Have the same issue, with hackney without using castore it's ok, so i suspect its a CAStore problem.

[whatyouhide]

Rather than being a CAStore issue per se, I suspect this is something to do with Mint's SSL defaults. CAStore is just the certificate store provided by Mozilla. Moving this to Mint, and possibly pinging @voltone to see if he has any ideas?

[YgorCastor]

Rather than being a CAStore issue per se, I suspect this is something to do with Mint's SSL defaults. CAStore is just the certificate store provided by Mozilla. Moving this to Mint, and possibly pinging @voltone to see if he has any ideas?

Hmm, it can be, if i change Mint transport_opts verify to verify_none it works.

[voltone]

Both servers send their certificate chain in an incorrect order, e.g.:

$ openssl s_client -connect www.kb.cert.org:443 -servername www.kb.cert.org
CONNECTED(00000005)
depth=3 C = GB, ST = Greater Manchester, L = Salford, O = Comodo CA Limited, CN = AAA Certificate Services
verify return:1
depth=2 C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust ECC Certification Authority
verify return:1
depth=1 C = US, ST = MI, L = Ann Arbor, O = Internet2, OU = InCommon, CN = InCommon ECC Server CA
verify return:1
depth=0 C = US, ST = Pennsylvania, O = Carnegie Mellon University, CN = kb.cert.org
verify return:1
---
Certificate chain
 0 s:/C=US/ST=Pennsylvania/O=Carnegie Mellon University/CN=kb.cert.org
   i:/C=US/ST=MI/L=Ann Arbor/O=Internet2/OU=InCommon/CN=InCommon ECC Server CA
 1 s:/C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=AAA Certificate Services
   i:/C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=AAA Certificate Services
 2 s:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust ECC Certification Authority
   i:/C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=AAA Certificate Services
 3 s:/C=US/ST=MI/L=Ann Arbor/O=Internet2/OU=InCommon/CN=InCommon ECC Server CA
   i:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust ECC Certification Authority
 4 s:/C=US/ST=Pennsylvania/O=Carnegie Mellon University/CN=kb.cert.org
   i:/C=US/ST=MI/L=Ann Arbor/O=Internet2/OU=InCommon/CN=InCommon ECC Server CA
---

The "AAA Certificate Services" certificate is a root CA, it shouldn't be sent by the server at all. The server certificate itself appears twice, as both the first and the last certificate. The two intermediate certificate (#2 and #3) appear in the reverse order.

Many TLS clients are quite flexible in fixing up the CA certificate chains, ignoring unnecessary/duplicate certificates, reordering them and even fetching missing certificate using the URL in the "Authority Information Access (AIA)" extension if available. Unfortunately, Erlang/OTPs ssl application is more strict.

[voltone]

For comparison, here's what a proper certificate chain looks like:

Certificate chain
 0 s:/C=US/ST=California/L=San Francisco/O=GitHub, Inc./CN=github.com
   i:/C=US/O=DigiCert Inc/CN=DigiCert TLS Hybrid ECC SHA384 2020 CA1
 1 s:/C=US/O=DigiCert Inc/CN=DigiCert TLS Hybrid ECC SHA384 2020 CA1
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA

Note now each i-line matches the s-line of the next certificate in the chain, and how there are no self-signed certificates (with identical s- and i-lines).

[1player]

Rather than being a CAStore issue per se, I suspect this is something to do with Mint's SSL defaults. CAStore is just the certificate store provided by Mozilla. Moving this to Mint, and possibly pinging @voltone to see if he has any ideas?

Hmm, it can be, if i change Mint transport_opts verify to verify_none it works.

Yes, on my crawler I'm using :verify_none until this is fixed. It's a decent workaround, iff you can afford to skip TLS verification.

[whatyouhide]

Ah, thanks @voltone this makes a lot of sense. @1player we'll close this as this is fixable in Mint by configuring your own TLS options. After all, we can't have defaults that will work for everyone on every certificate, so it might be safer here to not alter the default behavior and instead let the user customize the options.

@YgorCastor if you're modifying hackney TLS options, then hackney doesn't do the "smart merging" of defaults (or at least didn't do this a while ago, not sure if it was fixed). So, you might be not checking the cert altogether.

Thank you both for the report and thanks @voltone for the brilliant (as usual) help 🙃

[1player]

What TLS options fix this? Turning off verification is no fix, it's just a work around.

If it works on other clients, such as browsers or curl, and not on Mint, it's a bug. I do not think it is very useful to have a HTTP client that is too strict and cannot deal with not exactly compliant servers.

https://en.wikipedia.org/wiki/Robustness_principle

[whatyouhide]

If it works on other clients, such as browsers or curl, and not on Mint, it's a bug

Disagree.

do not think it is very useful to have a HTTP client that is too strict and cannot deal with not exactly compliant servers

The robustness principle you cite also literally cites "assume that the network is filled with malevolent entities that will send in packets designed to have the worst possible effect" 🙃

What TLS options fix this?

This is not a Mint question, but an Erlang SSL question. I'd love to help out but my knowledge in this area is somewhat limited. For general help, you might want to look at the Elixir Forum instead.