MQTT / TLS support for leaf certificates

The problem

esphome is unable to connect to MQTT servers that have self signed (leaf/end entity) certs that include only a portion of the distinguished name (as is common on BambuLabs 3d printers).

The certs can be viewed with openssl

IP=192.168.1.42
PORT=8883
openssl s_client -connect $IP:$PORT -showcerts < /dev/null 2>/dev/null

✅ Connection successful

Certificate chain
 0 s:C=US, ST=State, L=City, O=3DPrinters, CN=printer1
   i:C=US, ST=State, L=City, O=Example, CN=example.com
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Oct  4 21:10:34 2024 GMT; NotAfter: Oct  2 21:10:34 2034 GMT

❌ Connection fails

 Certificate chain
 0 s:CN=01P00A4xxxxxxxx
   i:C=CN, O=BBL Technologies Co., Ltd, CN=BBL CA
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Oct 11 16:44:40 2024 GMT; NotAfter: Oct  9 16:44:40 2034 GMT

If you have a BambuLabs printer, you can also reproduce yourself by connecting to port 8883 of your printer. Observe that the leaf certificate provides only CN for the distinguished name

openssl s_client -connect 192.168.0.2:8883 -showcerts < /dev/null 2>/dev/null

Which version of ESPHome has the issue?

2024.9.2

What type of installation are you using?

Docker

Which version of Home Assistant has the issue?

What platform are you using?

ESP32-IDF

Board

wemos d1 mini S2

Component causing the issue

mqtt

Example YAML snippet

esphome config

```yaml esp32: board: lolin_s2_mini variant: esp32s2 framework: type: esp-idf mqtt: broker: 192.168.81.145 # IP address of workstation where docker is running port: 9993 client_id: ${name} username: bblp password: 12345678 log_topic: discover_ip: false discovery: false discovery_retain: false discovery_prefix: use_abbreviations: false topic_prefix: testtopic/1234 birth_message: topic: # Dont post update to MQTT payload: # Dont post update to MQTT will_message: topic: # Dont post update to MQTT payload: # Dont post update to MQTT skip_cert_cn_check: true certificate_authority: | -----BEGIN CERTIFICATE----- MIIDGTCCAgGgAwIBAgIBATANBgkqhkiG9w0BAQsFADBCMQswCQYDVQQGEwJDTjEi MCAGA1UECgwZQkJMIFRlY2hub2xvZ2llcyBDby4sIEx0ZDEPMA0GA1UEAwwGQkJM IENBMB4XDTI0MTAxMTE2NDQ0MFoXDTM0MTAwOTE2NDQ0MFowGjEYMBYGA1UEAwwP MDFQMDBBNDQyMDAwMjc2MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA 4/0hXnlg1WAubzurfTEaZX2eqH+U+AXFVT6lTI65Hh1ykYhJoKrppEcwXb+c/v67 abCuXzdJAuOudcD+MziR5YChb2OUZ2U86Fp8lMcnkNNYoiP+609wTdibZ2kAijBC de/37NSQ2gmUrvrKKFSoSfy2ZzTJGg6J+oLzmXChyLNsxmulzNoAKSzccS/6u/hH taD5fFjR6He8tiXcCAalu+L7yNXatxES3TL3n78iNw9bLETE/Wy3T3fKRIpmig4e AzEGkbK3P4mttLzOUCh0dDKWUDJNaHq/yQMfWS4ZzAStBB3E2dCzTSaFrsFgj7eU LzawG93cWEwPS46enl3FvwIDAQABo0IwQDAdBgNVHQ4EFgQUn5hZ22xX3TrMJgvu /Hcyl3GBM7swHwYDVR0jBBgwFoAUfWjdttstezNFdUdOq2SE34+Gej4wDQYJKoZI hvcNAQELBQADggEBAI1UtjxfGYcW+0j0v+HkLL52CHK+t6M4aur2KhJ8KkUZnU3W FTotKCNwmuorHMB1DW66NZimBvr3O/UhxdW9u32mi4GigEQMkHdN6ef2xqFU3JTU wNmQfiNrSuzLct7ldVqNet/G4KS3afP56z8/x71kAJbzSj7bEjSbaKjwJsx0wBk/ 6drJM8iK38voAThyocN1OscWPgplBVuyW3Vsgdsg9oJ6/4Yk27tNdq2cfGaZtJoQ 9iubtuTKrCJDoJhg2GLxaDBTO0UCgOgLE2IX7Z32h6I4VznZxRU8rD/+OXo4ZFen +RcncGZMk9063LqgBkLEhR5eVlby2B0arZiRnxg= -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- MIIDZTCCAk2gAwIBAgIUKWdx3jaID5w19t9WtPta0dCdyw4wDQYJKoZIhvcNAQEL BQAwQjELMAkGA1UEBhMCQ04xIjAgBgNVBAoMGUJCTCBUZWNobm9sb2dpZXMgQ28u LCBMdGQxDzANBgNVBAMMBkJCTCBDQTAeFw0yNDEwMTExNjQ0MzlaFw0zNDEwMDkx NjQ0MzlaMEIxCzAJBgNVBAYTAkNOMSIwIAYDVQQKDBlCQkwgVGVjaG5vbG9naWVz IENvLiwgTHRkMQ8wDQYDVQQDDAZCQkwgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IB DwAwggEKAoIBAQDVwBwxkAcnaW2n4bHPFgeK9hR8OpSXb0WGohs6LyA89HjUS+Zp t8jsuKlNWgVM7ueE9sPWNV5EOO/F2YNH1NHgq51AjKnllArNd52E439Vo/fFlXSk uzI1OZ+HkLPuI/4BXTBSVuNnSWRvbWfByQ0Tf//UH7qCZq/s8TwwZwWiDZONK8SH SMd3sEaNCfrfZeael3IS0TcDl9xC8Q1G1kuPk11ioAoU1VRX1uVi5wLnRIOwfvF+ TxN64t8lGi2A62stcU/9hOs++7tSpWRWEzzp0oDj/HaF2WKo+8qUbbBcH2GLmqyv dVEYJkSN9SV7fDmOFi8maV2/5jRGnlrNyCFhAgMBAAGjUzBRMB0GA1UdDgQWBBR9 aN222y17M0V1R06rZITfj4Z6PjAfBgNVHSMEGDAWgBR9aN222y17M0V1R06rZITf j4Z6PjAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQBMO7A/CwhO pCOKeh7cUQNEtlGSqiXnDv7YDX2ltMM/lei9ebfaxqkUgqk2tVvrDhusNmIypyAX zZtH+MQvh7G0JjETXiIVcC4Lx0sErImntef0n/UvJTq/ynnSCqZVTA5yh8gC4vHy 0GnfXV/VZtC7ayqF7ykUux1dGbUBPhGILUehDR0HEeiFkZSLStdUH5kZJld+fzZw oXJhs5pxzdkGJ0ls65MduiocCbBVH/WkyHhHAt2yFawV4aYBK3iTZ9YXfWAl9fVZ tFuJv81M6Usv/rjLApoz2XYg6kZbI4Ja+KnNu0AlYjhwKZrs3XszwCnkrj+zVxog Af1z4aUP7/PC -----END CERTIFICATE----- ```

I've created a docker compose that can reproduce this problem. The docker container includes 3 mqtt servers running on localhost with various certs. The mqtt broker listening on port 8883 works correctly, however the mqtt broker listening on port 9993 reproduces the error.

Steps to reproduce

Clone git repo

git clone git@github.com:spuder/mqtt-demo.git
cd mqtt-demo

spin up docker containers using Podman Compose or Docker Compose
```
podman compose up
```
Observe 3 mqtt brokers on the following ports

port	result
1883	works (non tls)
8883	works (tls)
9993	throws error `0x8004`

Expected outcome

esp32 should be able to connect to servers regardless of what the distinguised name is

Actual outcome

esp32 is unable to connect to mqtt broker and produces the following logs

Logs

``` [13:39:21][V][esp-idf:000][mqtt_task]: E (2980149) esp-tls: [sock=57] delayed connect error: Connection reset by peer [13:39:21] [13:39:21][V][esp-idf:000][mqtt_task]: E (2980151) esp-tls: Failed to open new connection [13:39:21] [13:39:21][V][esp-idf:000][mqtt_task]: E (2980154) TRANSPORT_BASE: Failed to open a new connection [13:39:21] [13:39:21][V][esp-idf:000][mqtt_task]: E (2980156) MQTT_CLIENT: Error transport connect [13:39:21] [13:39:21][V][mqtt.idf:121]: Event dispatched from event loop event_id=0 [13:39:21][E][mqtt.idf:162]: MQTT_EVENT_ERROR [13:39:21][E][mqtt.idf:164]: Last error code reported from esp-tls: 0x8004 [13:39:21][E][mqtt.idf:165]: Last tls stack error number: 0x0 [13:39:21][E][mqtt.idf:167]: Last captured errno : 104 (Connection reset by peer) [13:39:21][V][mqtt.idf:121]: Event dispatched from event loop event_id=2 [13:39:21][V][mqtt.idf:133]: MQTT_EVENT_DISCONNECTED [13:39:21][W][mqtt:333]: MQTT Disconnected: TCP disconnected. [13:39:21][I][mqtt:244]: Connecting to MQTT... ```

Anything in the logs that might be useful for us?

The actual commands used to generate the SSL certs are located in certs/generate_certs.sh of the sample git repo.

Ignore the fact that one was created using a CSR and the other was created by signing the leaf cert directly. The key line is the following -subj "/C=US/ST=State/L=City/O=Example/CN=example.com" vs -subj "/CN=testprinter1"

Valid Cert

#!/bin/bash

# Variables
CA_KEY="ca-key.pem"
CA_CERT="ca-cert.crt"
PRINTER_KEY="printer1-key.pem"
PRINTER_CERT="printer1-cert.crt"
DAYS_VALID=3650

# Step 1: Generate CA private key
openssl genpkey -algorithm RSA -out $CA_KEY

# Step 2: Create CA certificate
openssl req -new -x509 -key $CA_KEY -sha256 -days $DAYS_VALID -out $CA_CERT \
    -subj "/C=US/ST=State/L=City/O=Example/CN=example.com"

# Step 3: Generate printer1 private key
openssl genpkey -algorithm RSA -out $PRINTER_KEY

# Step 4: Directly sign the printer1 key with the CA, creating a certificate
openssl req -new -key $PRINTER_KEY -out /dev/null -x509 -days $DAYS_VALID \
    -subj "/C=US/ST=State/L=City/O=3DPrinters/CN=printer1" \
    -set_serial 01 -CA $CA_CERT -CAkey $CA_KEY -out $PRINTER_CERT

echo "CA and Printer1 certificate generation complete!"

Rejected Cert

#!/bin/bash

# Variables
CA_KEY="ca.pem"
CA_CERT="ca.crt"
SERVER_KEY="server.pem"
SERVER_CERT="server.crt"
CERT_CHAIN="cert_chain.pem"
CA_DAYS_VALID=3650  # 10 years for CA
SERVER_DAYS_VALID=3650  # 10 years for the server cert
SERIAL=01

# Step 1: Generate CA private key and certificate
openssl genpkey -algorithm RSA -out $CA_KEY -pkeyopt rsa_keygen_bits:2048
openssl req -new -x509 -key $CA_KEY -sha256 -days $CA_DAYS_VALID -out $CA_CERT \
-subj "/C=CN/O=BBL Technologies Co., Ltd/CN=BBL CA" \
-sigopt rsa_padding_mode:pkcs1

# Step 2: Generate server private key
openssl genpkey -algorithm RSA -out $SERVER_KEY -pkeyopt rsa_keygen_bits:2048

# Step 3: Create the server certificate request (CSR)
openssl req -new -key $SERVER_KEY -out server.csr \
-subj "/CN=testprinter1"

# Step 4: Sign the server certificate with the CA using RSA-SHA256 (PKCS#1)
openssl x509 -req -in server.csr -CA $CA_CERT -CAkey $CA_KEY -CAcreateserial \
-out $SERVER_CERT -days $SERVER_DAYS_VALID -sha256 -set_serial $SERIAL \
-sigopt rsa_padding_mode:pkcs1

# Step 5: Create the certificate chain
cat $SERVER_CERT $CA_CERT > $CERT_CHAIN

# Clean up the CSR as it is not needed anymore
rm server.csr

echo "Certificate chain generation complete!"

Additional information

The esphome mqtt documentation states:

Support for esp-idf is still experminental. Please report issues you have with mqtt using the esp-idf framework.

Theory 1

The underlying esp-if/mbedtls have a parameter called common_name https://github.com/espressif/esp-mqtt/blob/master/include/mqtt_client.h#L267-L271

Theory 2

The certs provided by BambuLabs do not provide a S.A.N. The industry is moving away from CN and towards SAN as best practice on certs. Perhaps mbedtls requires a SAN

esphome / issues