DNSDist: Failed to load TLS server certificate

MachineITSvcs commented 2 years ago

I've been using DNSDist without issue for over a year now, but I just updated it and now it's having an issue with the Let's Encrypt/certbot fullchain certificate and fails to start. The error message provided is as follows:

40D074D4A37F0000:error:0A00018E:SSL routines:SSL_CTX_use_certificate:ca md too weak:../ssl/ssl_rsa.c:221: Fatal error: Error setting up TLS context for DoH listener on '66.70.208.168:443': An error occurred while trying to load the TLS server certificate file: /etc/dnsdist/fullchain.pem

From some research I did, I saw a similar issue existed for OpenVPN with MD5 certificates, and required certificate to be SHA256, but I confirmed my certificate has a Signature Algorithm of sha256WithRSAEncryption. Can someone assist me with this?

And yes, the files have the correct permissions: -rw-r--r-- 1 root _dnsdist 6254 Aug 13 17:06 fullchain.pem -rw-r----- 1 root _dnsdist 3272 Aug 13 17:06 privkey.pem

Thank you

Habbie commented 2 years ago

"I just updated it" - from what version to what version? What OS? Where did you get the packages?

MachineITSvcs commented 2 years ago

I believe it was from 1.7.2 to 1.8.0

Is there a log I can refer to?

rgacogne commented 2 years ago

If /etc/dnsdist/fullchain.pem is indeed a chain, it likely has more than one certificate. Did you check all of them for a weak digest?

MachineITSvcs commented 2 years ago

Yes, I did. There were only 2 in the chain. Both were sha256WithRSAEncryption.

Habbie commented 2 years ago

1.8.0 was not released yet, what are the exact from and to versions?
what distribution/OS? did you also upgrade other things at the same time, perhaps openssl?
can you share fullchain.pem?

MachineITSvcs commented 2 years ago

The package installed is 1.8.0~alpha0+master.1352.g38fd34c4a-1pdns.jammy (I do not know the exact from version. Just that it was 1.7.2, unless you know a log I can refer to?)
I'm running Ubuntu 22.04.1 LTS (jammy)
Please see below:

dwfreed commented 2 years ago

Please provide your full dnsdist config and the contents of /etc/ssl/openssl.cnf

MachineITSvcs commented 2 years ago

dnsdist config:

-- Set local listen address:port
setLocal("127.0.0.1:5353")

-- Set public listen address:port
--addLocal("<redacted-ip4-1>:53")
--addLocal("<redacted-ip4-2>:53")
--addLocal("<redacted-ip6-1>:53")
--addLocal("<redacted-ip6-2>:53")

-- Set downstream resolver
newServer({address="127.0.0.53:53", qps=250, name="resolver1"})

-- Set DoH listen address:port
addDOHLocal("<redacted-ip4-3>:443", "/etc/dnsdist/fullchain.pem", "/etc/dnsdist/privkey.pem", { "/dns-query" }, { doTCP=true, reusePort=true, tcpFastOpenSize=0 })

-- Allow query from all IP addresses
addACL('0.0.0.0/0')

I have redacted public IP addresses as these are private/not publicly listed. Again, this config worked perfectly before the upgrade.

Contents of /etc/ssl/openssl.conf

#
# OpenSSL example configuration file.
# See doc/man5/config.pod for more info.
#
# This is mostly being used for generation of certificate requests,
# but may be used for auto loading of providers

# Note that you can include other files from the main configuration
# file using the .include directive.
#.include filename

# This definition stops the following lines choking if HOME isn't
# defined.
HOME                    = .

 # Use this in order to automatically load providers.
openssl_conf = openssl_init

# Comment out the next line to ignore configuration errors
config_diagnostics = 1

# Extra OBJECT IDENTIFIER info:
# oid_file       = $ENV::HOME/.oid
oid_section = new_oids

# To use this configuration file with the "-extfile" option of the
# "openssl x509" utility, name here the section containing the
# X.509v3 extensions to use:
# extensions            =
# (Alternatively, use a configuration file that has only
# X.509v3 extensions in its main [= default] section.)

[ new_oids ]
# We can add new OIDs in here for use by 'ca', 'req' and 'ts'.
# Add a simple OID like this:
# testoid1=1.2.3.4
# Or use config file substitution like this:
# testoid2=${testoid1}.5.6

# Policies used by the TSA examples.
tsa_policy1 = 1.2.3.4.1
tsa_policy2 = 1.2.3.4.5.6
tsa_policy3 = 1.2.3.4.5.7

# For FIPS
# Optionally include a file that is generated by the OpenSSL fipsinstall
# application. This file contains configuration data required by the OpenSSL
# fips provider. It contains a named section e.g. [fips_sect] which is
# referenced from the [provider_sect] below.
# Refer to the OpenSSL security policy for more information.
# .include fipsmodule.cnf

[openssl_init]
providers = provider_sect
ssl_conf = ssl_sect

# List of providers to load
[provider_sect]
default = default_sect
# The fips section name should match the section name inside the
# included fipsmodule.cnf.
# fips = fips_sect

# If no providers are activated explicitly, the default one is activated implicitly.
# See man 7 OSSL_PROVIDER-default for more details.
#
# If you add a section explicitly activating any other provider(s), you most
# probably need to explicitly activate the default provider, otherwise it
# becomes unavailable in openssl.  As a consequence applications depending on
# OpenSSL may not work correctly which could lead to significant system
# problems including inability to remotely access the system.
[default_sect]
# activate = 1

####################################################################
[ ca ]
default_ca      = CA_default            # The default ca section

####################################################################
[ CA_default ]

dir             = ./demoCA              # Where everything is kept
certs           = $dir/certs            # Where the issued certs are kept
crl_dir         = $dir/crl              # Where the issued crl are kept
database        = $dir/index.txt        # database index file.
#unique_subject = no                    # Set to 'no' to allow creation of
                                        # several certs with same subject.
new_certs_dir   = $dir/newcerts         # default place for new certs.

certificate     = $dir/cacert.pem       # The CA certificate
serial          = $dir/serial           # The current serial number
crlnumber       = $dir/crlnumber        # the current crl number
                                        # must be commented out to leave a V1 CRL
crl             = $dir/crl.pem          # The current CRL
private_key     = $dir/private/cakey.pem# The private key

x509_extensions = usr_cert              # The extensions to add to the cert

# Comment out the following two lines for the "traditional"
# (and highly broken) format.
name_opt        = ca_default            # Subject Name options
cert_opt        = ca_default            # Certificate field options

# Extension copying option: use with caution.
# copy_extensions = copy

# Extensions to add to a CRL. Note: Netscape communicator chokes on V2 CRLs
# so this is commented out by default to leave a V1 CRL.
# crlnumber must also be commented out to leave a V1 CRL.
# crl_extensions        = crl_ext

default_days    = 365                   # how long to certify for
default_crl_days= 30                    # how long before next CRL
default_md      = default               # use public key default MD
preserve        = no                    # keep passed DN ordering

# A few difference way of specifying how similar the request should look
# For type CA, the listed attributes must be the same, and the optional
# and supplied fields are just that :-)
policy          = policy_match

# For the CA policy
[ policy_match ]
countryName             = match
stateOrProvinceName     = match
organizationName        = match
organizationalUnitName  = optional
commonName              = supplied
emailAddress            = optional

# For the 'anything' policy
# At this point in time, you must list all acceptable 'object'
# types.
[ policy_anything ]
countryName             = optional
stateOrProvinceName     = optional
localityName            = optional
organizationName        = optional
organizationalUnitName  = optional
commonName              = supplied
emailAddress            = optional

####################################################################
[ req ]
default_bits            = 2048
default_keyfile         = privkey.pem
distinguished_name      = req_distinguished_name
attributes              = req_attributes
x509_extensions = v3_ca # The extensions to add to the self signed cert

# Passwords for private keys if not present they will be prompted for
# input_password = secret
# output_password = secret

# This sets a mask for permitted string types. There are several options.
# default: PrintableString, T61String, BMPString.
# pkix   : PrintableString, BMPString (PKIX recommendation before 2004)
# utf8only: only UTF8Strings (PKIX recommendation after 2004).
# nombstr : PrintableString, T61String (no BMPStrings or UTF8Strings).
# MASK:XXXX a literal mask value.
# WARNING: ancient versions of Netscape crash on BMPStrings or UTF8Strings.
string_mask = utf8only

# req_extensions = v3_req # The extensions to add to a certificate request

[ req_distinguished_name ]
countryName                     = Country Name (2 letter code)
countryName_default             = AU
countryName_min                 = 2
countryName_max                 = 2

stateOrProvinceName             = State or Province Name (full name)
stateOrProvinceName_default     = Some-State

localityName                    = Locality Name (eg, city)

0.organizationName              = Organization Name (eg, company)
0.organizationName_default      = Internet Widgits Pty Ltd

# we can do this but it is not needed normally :-)
#1.organizationName             = Second Organization Name (eg, company)
#1.organizationName_default     = World Wide Web Pty Ltd

organizationalUnitName          = Organizational Unit Name (eg, section)
#organizationalUnitName_default =

commonName                      = Common Name (e.g. server FQDN or YOUR name)
commonName_max                  = 64

emailAddress                    = Email Address
emailAddress_max                = 64

# SET-ex3                       = SET extension number 3

[ req_attributes ]
challengePassword               = A challenge password
challengePassword_min           = 4
challengePassword_max           = 20

unstructuredName                = An optional company name

[ usr_cert ]

# These extensions are added when 'ca' signs a request.

# This goes against PKIX guidelines but some CAs do it and some software
# requires this to avoid interpreting an end user certificate as a CA.

basicConstraints=CA:FALSE

# This is typical in keyUsage for a client certificate.
# keyUsage = nonRepudiation, digitalSignature, keyEncipherment

# PKIX recommendations harmless if included in all certificates.
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer

# This stuff is for subjectAltName and issuerAltname.
# Import the email address.
# subjectAltName=email:copy
# An alternative to produce certificates that aren't
# deprecated according to PKIX.
# subjectAltName=email:move

# Copy subject details
# issuerAltName=issuer:copy

# This is required for TSA certificates.
# extendedKeyUsage = critical,timeStamping

[ v3_req ]

# Extensions to add to a certificate request

basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment

[ v3_ca ]

# Extensions for a typical CA

# PKIX recommendation.

subjectKeyIdentifier=hash

authorityKeyIdentifier=keyid:always,issuer

basicConstraints = critical,CA:true

# Key usage: this is typical for a CA certificate. However since it will
# prevent it being used as an test self-signed certificate it is best
# left out by default.
# keyUsage = cRLSign, keyCertSign

# Include email address in subject alt name: another PKIX recommendation
# subjectAltName=email:copy
# Copy issuer details
# issuerAltName=issuer:copy

# DER hex encoding of an extension: beware experts only!
# obj=DER:02:03
# Where 'obj' is a standard or added object
# You can even override a supported extension:
# basicConstraints= critical, DER:30:03:01:01:FF

[ crl_ext ]

# CRL extensions.
# Only issuerAltName and authorityKeyIdentifier make any sense in a CRL.

# issuerAltName=issuer:copy
authorityKeyIdentifier=keyid:always

[ proxy_cert_ext ]
# These extensions should be added when creating a proxy certificate

# This goes against PKIX guidelines but some CAs do it and some software
# requires this to avoid interpreting an end user certificate as a CA.

basicConstraints=CA:FALSE

# This is typical in keyUsage for a client certificate.
# keyUsage = nonRepudiation, digitalSignature, keyEncipherment

# PKIX recommendations harmless if included in all certificates.
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer

# This stuff is for subjectAltName and issuerAltname.
# Import the email address.
# subjectAltName=email:copy
# An alternative to produce certificates that aren't
# deprecated according to PKIX.
# subjectAltName=email:move

# Copy subject details
# issuerAltName=issuer:copy

# This really needs to be in place for it to be a proxy certificate.
proxyCertInfo=critical,language:id-ppl-anyLanguage,pathlen:3,policy:foo

####################################################################
[ tsa ]

default_tsa = tsa_config1       # the default TSA section

[ tsa_config1 ]

# These are used by the TSA reply generation only.
dir             = ./demoCA              # TSA root directory
serial          = $dir/tsaserial        # The current serial number (mandatory)
crypto_device   = builtin               # OpenSSL engine to use for signing
signer_cert     = $dir/tsacert.pem      # The TSA signing certificate
                                        # (optional)
certs           = $dir/cacert.pem       # Certificate chain to include in reply
                                        # (optional)
signer_key      = $dir/private/tsakey.pem # The TSA private key (optional)
signer_digest  = sha256                 # Signing digest to use. (Optional)
default_policy  = tsa_policy1           # Policy if request did not specify it
                                        # (optional)
other_policies  = tsa_policy2, tsa_policy3      # acceptable policies (optional)
digests     = sha1, sha256, sha384, sha512  # Acceptable message digests (mandatory)
accuracy        = secs:1, millisecs:500, microsecs:100  # (optional)
clock_precision_digits  = 0     # number of digits after dot. (optional)
ordering                = yes   # Is ordering defined for timestamps?
                                # (optional, default: no)
tsa_name                = yes   # Must the TSA name be included in the reply?
                                # (optional, default: no)
ess_cert_id_chain       = no    # Must the ESS cert id chain be included?
                                # (optional, default: no)
ess_cert_id_alg         = sha1  # algorithm to compute certificate
                                # identifier (optional, default: sha1)

[insta] # CMP using Insta Demo CA
# Message transfer
server = pki.certificate.fi:8700
# proxy = # set this as far as needed, e.g., http://192.168.1.1:8080
# tls_use = 0
path = pkix/

# Server authentication
recipient = "/C=FI/O=Insta Demo/CN=Insta Demo CA" # or set srvcert or issuer
ignore_keyusage = 1 # potentially needed quirk
unprotected_errors = 1 # potentially needed quirk
extracertsout = insta.extracerts.pem

# Client authentication
ref = 3078 # user identification
secret = pass:insta # can be used for both client and server side

# Generic message options
cmd = ir # default operation, can be overridden on cmd line with, e.g., kur

# Certificate enrollment
subject = "/CN=openssl-cmp-test"
newkey = insta.priv.pem
out_trusted = insta.ca.crt
certout = insta.cert.pem

[pbm] # Password-based protection for Insta CA
# Server and client authentication
ref = $insta::ref # 3078
secret = $insta::secret # pass:insta

[signature] # Signature-based protection for Insta CA
# Server authentication
trusted = insta.ca.crt # does not include keyUsage digitalSignature

# Client authentication
secret = # disable PBM
key = $insta::newkey # insta.priv.pem
cert = $insta::certout # insta.cert.pem

[ir]
cmd = ir

[cr]
cmd = cr

[kur]
# Certificate update
cmd = kur
oldcert = $insta::certout # insta.cert.pem

[rr]
# Certificate revocation
cmd = rr
oldcert = $insta::certout # insta.cert.pem

[ssl_sect]
system_default = system_default_sect

[system_default_sect]
CipherString = DEFAULT:@SECLEVEL=2

rgacogne commented 2 years ago

Did you upgrade anything else than dnsdist from 1.7 to master? The code loading TLS certificates did not change much since 1.7.x, but Jammy has OpenSSL 3.0.x which is a big change from 1.1.1x, so it would be useful to know if you were already using OpenSSL 3.0.x before, for example.

MachineITSvcs commented 2 years ago

Well, dnsdist was actually updated during an upgrade from Ubuntu Focal to Jammy, so it's entirely possible a new version of openssl among other updates were also installed. Do you think OpenSSL 3.0.X is the issue? Is there anything you'd like me to try?

rgacogne commented 2 years ago

It is very likely that the issue comes from OpenSSL 3.0.x being more strict, yes. That doesn't mean that we are doing everything right, but I'm pretty sure the issue does not come from a change we made since 1.7.x but from the switch to OpenSSL 3.0.x. I'm pretty sure re-compiling 1.7.2 on Jammy would yield the same issue you currently have with master. Now the question is how can we debug this.. Did you generate the chain on Jammy, or did you import it from Focal?

MachineITSvcs commented 2 years ago

It's a Let's Encrypt SSL certificate chain. The fullchain.pem was just combining the chain parts from the Let's Encrypt certificate directory. I believe the chain parts are served by LE, and not generated locally, but I could be mistaken. I know that I have tried forcing a renewal of the certificate from Let's Encrypt and recombining the parts into a new fullchain.pem and I get the same error.

rgacogne commented 2 years ago

It looks like commenting out our explicit call to OPENSSL_init_crypto solves the issue. So I guess OpenSSL decided to break the way they are loading ciphers and digests yet another time, without documenting it.

rgacogne commented 2 years ago

Indeed, in 3.0 ossl_init_ssl_base no longer loads the ciphers and digests required for TLS support, which seems insane to me.

rgacogne commented 2 years ago

https://github.com/openssl/openssl/commit/661595ca0933fe631faeadd14a189acd5d4185e0

git-noise commented 2 years ago

@rgacogne Hello, For what it's worth, I had an issue like the one described here, and it seems to be fixed for me now. Best,

PowerDNS / pdns

DNSDist: Failed to load TLS server certificate #11853