DoT gives broken certificate (untrusted root)

[AlexFullmoon]

I issue new certificate through certbot, convert it to pkcs12 with openssl pkcs12 -export -out "dns.moonlightwell.ru.pfx" -inkey "privkey.pem" -in "cert.pem" -certfile "chain.pem" and add that .pfx along with password to Technitium.

Then several DNS lookup utilities fail with errors like: Error [tls]: A certificate chain processed, but terminated in a root certificate which is not trusted by the trust provider. (os error -2146762487) [fatal] Cannot make the DNS request: getting conn to tls://dns.moonlightwell.ru:853: connecting to dns.moonlightwell.ru: tls: failed to verify certificate: x509: certificate signed by unknown authority

I suspect that something is off in either conversion to pkcs12 or Technitium itself, because I can add the same certificate (specifically, fullchain.pem and privkey.pem) to AdGuardHome and DoT in it works without errors.

I tried running openssl pkcs12 -export -out "dns.moonlightwell.ru.pfx" -inkey "privkey.pem" -in "cert.pem" -certfile "fullchain.pem", but that didn't help.

[AlexFullmoon]

Update: here are results of openssl s_client -showcerts -servername dns.moonlightwell.ru -connect dns.moonlightwell.ru:853 As I understand, Technitium doesn't send cert chain correctly. If needed I can provide my certificate files, but ofc not in the open.

sslquery-adguardhome.txt sslquery-technitium.txt

[ShreyasZare]

Thanks for the post. I tested this with the DNS Client website and it seems to be working well. Check the query here.

Regarding the cert chain issue, I will get it tested locally and let you know.

[AlexFullmoon]

I tested this with the DNS Client website and it seems to be working well.

Well, some clients (like dig with +tls support) don't get errors, or ignore them, some (dnslookup, q, dog, and, apparently and most importantly, Android private DNS) do.

For comparison, I started AGH with same certificate on dns.moonlightwell.ru:854, feel free to test. Sorry, had to shut it down for a while.

[ShreyasZare]

I tested this with the DNS Client website and it seems to be working well.

Well, some clients (like dig with +tls support) don't get errors, or ignore them, some (dnslookup, q, dog, and, apparently and most importantly, Android private DNS) do.

I have tested it with Android private DNS and it works without issues. It could be issue with some phones though.

Its clear though that the DoT service is returning the cert + ca cert, but not the root cert. Since a lot of clients already have the ca cert in their list so they work without issue.

[Jburso]

I've been running into this exact issue as well, but with the web service. I have a root -> intermediate -> leaf and the pkcs12 file contains both the intermediate and leaf certificates, while the root is trusted on the client. It looks like technitium is only serving the leaf without the intermediate which is causes the connection to fail.

[ShreyasZare]

I've been running into this exact issue as well, but with the web service. I have a root -> intermediate -> leaf and the pkcs12 file contains both the intermediate and leaf certificates, while the root is trusted on the client. It looks like technitium is only serving the leaf without the intermediate which is causes the connection to fail.

Thanks for the feedback. This has been tested and it returns the cert + the ca-cert. Only part missing is the root cert which is expected to be installed on the client.

Which OS are you running the DNS server on? Run the following openssl command and post the response here.

openssl s_client -connect <server-ip>:<https-port>

[Jburso]

I am running Technitium DNS version 11.2 on Alpine Linux 3.18.2

openssl_sclient.txt cert.pem chain.pem

It looks like the s_client only shows one certificate in the chain, and that's the leaf cert. I'm pretty sure the intermediate should be there as well.

This is the command I am using to create the PKCS12 file:

#!/bin/sh

PFX_FILE=tls.pfx

openssl pkcs12 -export -out /etc/letsencrypt/live/dns.corp/${PFX_FILE} \
-inkey /etc/letsencrypt/live/dns.corp/privkey.pem \
-in /etc/letsencrypt/live/dns.corp/cert.pem \
-certfile /etc/letsencrypt/live/dns.corp/chain.pem

[ShreyasZare]

I am running Technitium DNS version 11.2 on Alpine Linux 3.18.2

openssl_sclient.txt cert.pem chain.pem

It looks like the s_client only shows one certificate in the chain, and that's the leaf cert. I'm pretty sure the intermediate should be there as well.

This is the command I am using to create the PKCS12 file:

#!/bin/sh

PFX_FILE=tls.pfx

openssl pkcs12 -export -out /etc/letsencrypt/live/dns.corp/${PFX_FILE} \
-inkey /etc/letsencrypt/live/dns.corp/privkey.pem \
-in /etc/letsencrypt/live/dns.corp/cert.pem \
-certfile /etc/letsencrypt/live/dns.corp/chain.pem

Thanks for the details. I have tested this on Debian and Ubuntu, and in both cases its working as expected. Not sure why you are seeing this issue.

[Jburso]

I did some further testing and I am getting the same behavior for Alpine 3.18, Debian 12, and Ubuntu 22.04 -- only the leaf certificate is being sent when activating the HTTPS webserver with a PKCS12 file that contains the leaf and the intermediate certificates. All of my testing is being done with a local, private certificate authority. This might explain the difference between my experience and your testing.

With that said, it looks like this bug is the result of a particularity with the .NET runtime and how Kestrel handles multiple certificates in the PKSC12 file when loaded with X509Certificate2: https://github.com/dotnet/aspnetcore/issues/36202. The intermediate is ignored and not sent if it's not trusted. I added the intermediate to the dns root trust store and the webserver started exhibiting the correct behavior where both certificates were being sent.

This obviously isn't a long term solution though because intermediate certs should not be in the root trust store.

While some workaround for .NET 6 are provided in the above issue, it looks like a fix was merged into .NET 7 (https://github.com/dotnet/aspnetcore/issues/21513) by using ServerCertificateChain with HttpsConnectionAdapterOptions to specify the full certificate chain. Documentation is here: https://learn.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.server.kestrel.https.httpsconnectionadapteroptions.servercertificatechain?view=aspnetcore-7.0#microsoft-aspnetcore-server-kestrel-https-httpsconnectionadapteroptions-servercertificatechain

TechnitiumDNS already depends on version 7 so hopefully this will be a quick fix.

[ShreyasZare]

I did some further testing and I am getting the same behavior for Alpine 3.18, Debian 12, and Ubuntu 22.04 -- only the leaf certificate is being sent when activating the HTTPS webserver with a PKCS12 file that contains the leaf and the intermediate certificates. All of my testing is being done with a local, private certificate authority. This might explain the difference between my experience and your testing.

With that said, it looks like this bug is the result of a particularity with the .NET runtime and how Kestrel handles multiple certificates in the PKSC12 file when loaded with X509Certificate2: dotnet/aspnetcore#36202. The intermediate is ignored and not sent if it's not trusted. I added the intermediate to the dns root trust store and the webserver started exhibiting the correct behavior where both certificates were being sent.

This obviously isn't a long term solution though because intermediate certs should not be in the root trust store.

While some workaround for .NET 6 are provided in the above issue, it looks like a fix was merged into .NET 7 (dotnet/aspnetcore#21513) by using ServerCertificateChain with HttpsConnectionAdapterOptions to specify the full certificate chain. Documentation is here: https://learn.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.server.kestrel.https.httpsconnectionadapteroptions.servercertificatechain?view=aspnetcore-7.0#microsoft-aspnetcore-server-kestrel-https-httpsconnectionadapteroptions-servercertificatechain

TechnitiumDNS already depends on version 7 so hopefully this will be a quick fix.

Thanks for the detailed research. Will test it out and let you know soon.

[ShreyasZare]

Technitium DNS Server v11.4 is now available that fixes this issue. The Kestrel web server will now always send the CA cert but wont send the root certificate and there seems to be no option to make it do it. The same goes for the DoT service since the SslStream also works the same way.

Do update and let me know your feedback.