TLS tests fail on Mac OS

[tlater-famedly]

Apparently Mac OS has very strict requirements for SSL certificates, which we do not meet with the certificates in our test suite: https://support.apple.com/en-us/103769

[emgrav]

Which requirements are not met?

[tlater-famedly]

I believe:

• TLS server certificates must present the DNS name of the server in the Subject Alternative Name extension of the certificate. DNS names in the CommonName of a certificate are no longer trusted.
  • This one may actually be met, but I'm not 100% sure
• TLS server certificates must contain an ExtendedKeyUsage (EKU) extension containing the id-kp-serverAuth OID.
• TLS server certificates must have a validity period of 825 days or fewer (as expressed in the NotBefore and NotAfter fields of the certificate).

We can consider issuing the certificate in the past to fix that last bit, so we don't have to constantly regenerate certificates. Or call the certificate generation script during test setup.

[sirewix]

call the certificate generation script during test setup.

Would this be much trouble?

[tlater-famedly]

It requires having openssl (not just as a library) on the host running the tests, or adding another container to the test runs. So either require more host tooling - which I've avoided so far to simplify getting started as much as possible - or slow down tests a bit.

If we do regenerate the certs, we should at least add some logic to ensure we only do so when they are expired, as certificate generation also takes a non-negligible amount of time.

[sirewix]

I see, then maybe issuing a cetificate in the past is not such a bad idea. However I'm also fine with adding more tooling on the host or into the container (probably the latter is better). Regarding only-when-expired check, that could probably be done with some lock file where we store the issue date (to avoid calling openssl on each run to check the issue date).

[tlater-famedly]

that could probably be done with some lock file where we store the issue date (to avoid calling openssl on each run to check the issue date)

Maybe, if we have a requirement on openssl anyway that shouldn't really be a meaningful overhead, though, unless we prevent starting a container altogether, which I don't think is trivial?

[sirewix]

which I don't think is trivial?

I thought you meant having separate container for openssl:

or adding another container to the test runs

In that case it would be a matter of running docker run openssl -- keygen blabla or not running it.

If we add openssl to the container with the test suite, then yeah, no difference ofcourse

[emgrav]

We will generate issues before July 1st, 2019 with the server name in the server name extension and not just CommonName.

[jannden]

I suspect that there might be something else in play in addition to the aforementioned.

This Certificate Transparency Policy might be important as well. The other doubts I have is how the exact DNS name of the server certificate should be composed and how Docker influences the flow.

I provide a script bellow (assumes OpenSSL 1.1.1 or newer). By adjusting the system clock (safest bet + Mac command line openssl doesn't support the -startdate flag), the script creates what we said: the certificate is created before July 1st, 2019 and presents the DNS name of the server in the Subject Alternative Name extension of the server certificate.

The Outcomes


Dates correct: $ openssl x509 -in server.crt -noout -dates

Subject Alternative Name correct (assuming the DNS addresses make sense): $ openssl x509 -in server.crt -noout -ext subjectAltName

The Key Usage and Extended Key Usage: $ openssl x509 -in server.crt -noout -ext keyUsage,extendedKeyUsage

The subject (issuer for self-signed): $ openssl x509 -in server.crt -noout -subject -issuer

Public key algorithm and size: $ openssl x509 -in server.crt -noout -pubkey | openssl pkey -pubin -text -noout

The Verifications

Passes cert chain:
$ openssl verify -CAfile ca.crt server.crt

Passes cert verif, but with a warning: Unable to find at least 2 signed certificate timestamps (SCTs) from approved logs $ security verify-cert -c server.crt -p ssl

Optionally adding the certificate to the trusted ones in the Mac keychain doesn’t change much (expected since Docker): sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain ca.crt

The Script

Here is the certificate generation script:

#!/bin/bash
set -eux

# Create self-signed server certificate
openssl req -x509 -new -nodes -sha256 -newkey rsa:2048 \
    -keyout server.key \
    -out server.crt \
    -days 3650 \
    -subj "/C=DE/O=Example Organization/OU=testorg/CN=example.org" \
    -addext "subjectAltName = DNS:example.org, DNS:admin.example.org, DNS:zitadel, DNS:ldap, DNS:ldap-setup, DNS:test-setup, DNS:localhost" \
    -addext "keyUsage = critical, digitalSignature, keyEncipherment" \
    -addext "extendedKeyUsage = serverAuth" \
    -addext "subjectKeyIdentifier = hash" \
    -addext "authorityKeyIdentifier = keyid:always,issuer" \
    -addext "basicConstraints = critical, CA:true" \
    -set_serial 0x`openssl rand -hex 8`

# Set permissions for the server key
chmod go+r server.key

# Create client certificate
openssl req -x509 -new -nodes -sha256 -newkey rsa:2048 \
    -keyout client.key \
    -out client.crt \
    -days 3650 \
    -subj "/C=DE/O=Example Organization/OU=testorg/CN=admin.example.org" \
    -addext "keyUsage = critical, digitalSignature, keyEncipherment" \
    -addext "extendedKeyUsage = clientAuth" \
    -set_serial 0x`openssl rand -hex 8`

# Create CA certificate (optional, if needed)
cp server.crt ca.crt

[tlater-famedly]

As @mzaniolo spotted in 49bb8dc4824453383a5a3b528a7b01d60bebe384, looks like openssl doesn't want to generate certificates without an actual expiration date, not specifying one results in a ~1 month default. Given that, maybe regenerating the certs during test setup is the only reasonable option after all.

[tlater-famedly]

We should consider something like rcgen or certstrap for this, since openssl is notoriously hard to use.

[mzaniolo]

I tried to use the same certificates on a simple webserver in python and validate it with curl --cacert. It worked locally and inside a docker.

I also made a minimum example to reproduce the problem, a very simple program to query ldap. I tried to change two different settings, set_starttls and set_no_tls_verify. The set_no_tls_verify doesn't have any influence on the problem but making set_starttls true causes the following error: NativeTLS { source: Error { code: -67843, message: "The certificate was not trusted." } }

Since the error is from NativeTLS it makes sense that only macos is failing.

I read that I could get more info from what was causing the the error looking at the macos logs. I tried it using the command sudo log show --last 20s --predicate 'process=="trustd"' --debug --info I could be something related with EKUs since I see some massage related to that but no sure. The logs:

As next step it would be good to try to add the following parameters to the generation script: -addext "extendedKeyUsage = serverAuth, clientAuth" -addext "keyUsage = digitalSignature,keyAgreement"

[mzaniolo]

Adding the only the extendedKeyUsage solved the problem. I also ended up adding the KeyUsage to the server cert. This caused the client cert to don't be trusted because the keyUsage of the CA was more restricted then the keyUsage from the client. Adding the same keyUsage to the client cert or removing it from the server cert solved the problem