Self-signed mosquitto + Tasmota + MQTT TLS no longer working

PROBLEM DESCRIPTION

After following the instructions curently outlined in the Self-signed Mosquitto docs I am unable to provision a device to connect to my private broker. The error returned on the web console is as follows:

23:53:54.758 MQT: Attempting connection...
23:53:54.821 MQT: TLS connection error: 296
23:53:54.823 MQT: Connect failed to mqtt.<my server>.io:8883, rc -2. Retry in 100 sec

Error 296 is a SSL3_ALERT_HANDSHAKE_FAILURE. The server logs state the following:

1707778768: OpenSSL Error[0]: error:0A0000C1:SSL routines::no shared cipher
1707778768: Client <unknown> disconnected: Protocol error.

This implies, to me, that either the server or the device do not have the right cipher suite. The likely culprit is the tasmota device as it is configured to use a singular cipher suite here

When I manually use the ca.crt, device.crt and device.pem to try to connect to the server ./BearSSL/build/brssl client mqtt.<my server>.io:8883 \ -CA easy-rsa/easyrsa3/pki/ca.crt \ -cert easy-rsa/easyrsa3/pki/issued/device.crt \ -key easy-rsa/easyrsa3/pki/private/device.pem We succeed! But the cipher used is ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, here is the output.

Algorithms:
   RNG:           rdrand
   AES/CBC (enc): x86ni
   AES/CBC (dec): x86ni
   AES/CTR:       x86ni
   AES/CCM:       x86ni
   DES/CBC (enc): ct
   DES/CBC (dec): ct
   GHASH (GCM):   pclmul
   ChaCha20:      sse2
   Poly1305:      ctmulq
   EC:            all_m31
   ECDSA:         i31_asn1
   RSA (vrfy):    i62
Server requests a client certificate.
--- anchor DN list start ---
--- anchor DN list end ---
supported: RSA signatures: SHA-224 SHA-256 SHA-384 SHA-512
supported: ECDSA signatures: SHA-224 SHA-256 SHA-384 SHA-512
server key curve: secp256r1 (23)
using ECDSA, hash = 4 (SHA-256)
Handshake completed
   version:               TLS 1.2
   cipher suite:          ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
   ECDHE curve:           Curve25519
   secure renegotiation:  yes
SSL closed normally

According to the tasmota source code, the only supported cipher is BR_TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256. This implies that the device should be using an RSA cert/private key BUT the documentation only talks about EC. Based on my reading of previous issues, my suspicion is that the source was adapted to support AWS IOT which uses an RSA based cert and the self-signed support was not updated?

A few other things I've tried:

If I do switch the EASYRSA_ALGO to rsa, the key generated by the tls key script is more that 32 bytes which causes the device to reject it.
If I set the cipher suite here in the source to BR_TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (which is what bearSSL said worked for the cert/keys I have), this alsodoes not work. Presumably because there are other configuration options that I missed.
MQTT explorer is happy to connect to my server and use the cert/keys I have generate.

The documentation here https://tasmota.github.io/docs/TLS/#implementation-notes says the following:

EC private key: 
AWS IoT requires the client to authenticate with its own Private Key and Certificate. By default AWS IoT will generate an RSA 2048 bit private key. In Tasmota, we moved to an EC (Elliptic Curve) Private Key of 256 bits. EC keys are much smaller, and handshake is significantly faster. Note: the key being 256 bits does not mean it's less secure than RSA 2048, it's actually the opposite.

Single Cipher: to reduce code size, we only support a single TLS cipher and embed only the code strictly necessary. When using TLS (e.g. LetsEncrypt on Mosquitto) the supported cipher is ECDHE_RSA_WITH_AES_128_GCM_SHA256. Additionally, ECDHE offers Perfect Forward Secrecy which means extra security.

It's very unclear to my how we can force EasyRSA generate a 2048 bit RSA cert but use an EC private key.

So I'm at a bit of a loss, it seems like something switched to RSA at some point but the "Self-signed Mosquitto" documentation still refers to EC? I don't know enough about TLS to find where things are going wrong in the source.

Here is my user_config_override.h

/*
  user_config_override.h - user configuration overrides my_user_config.h for Tasmota
*/

#ifndef _USER_CONFIG_OVERRIDE_H_
#define _USER_CONFIG_OVERRIDE_H_

// disable
//#ifdef USE_DOMOTICZ
//#undef USE_DOMOTICZ                              
//#endif 

// TODO: REMOVE - just for local testing
#undef  STA_SSID1
#define STA_SSID1         "<removed>"             // [Ssid1] Wifi SSID

#undef  STA_PASS1
#define STA_PASS1         "<removed>"     // [Password1] Wifi password

#undef  MQTT_HOST
#define MQTT_HOST "mqtt.metermaid.io"   // [MqttHost]

#undef  MQTT_PORT
#define MQTT_PORT 8883 // [MqttPort] MQTT port (10123 on CloudMQTT)

#ifndef USE_MQTT_TLS
#define USE_MQTT_TLS
#endif

#ifdef MQTT_TLS_ENABLED
#undef MQTT_TLS_ENABLED
#endif
#define MQTT_TLS_ENABLED true

/*  
    Force full CA validation instead of fingerprints, slower, 
    but simpler to use. 
    (+2.2k code, +1.9k mem during connection handshake)
*/
#define USE_MQTT_TLS_CA_CERT   

/*
    This will include LetsEncrypt CA, as well as our CA, in tasmota_ca.ino 
    for verifying server certificates
*/
#define USE_MQTT_AWS_IOT

// We want to use our own certs
#define OMIT_AWS_CERT
#define OMIT_LETS_ENCRYPT_CERT

// This is deprecated, no longer used in the code
// #define USE_MQTT_TLS_FORCE_EC_CIPHER           // Force Elliptic Curve cipher (higher security) required by some servers (automatically enabled with USE_MQTT_AWS_IOT) (+11.4k code, +0.4k mem)

#define INCLUDE_LOCAL_CERT

#endif  // _USER_CONFIG_OVERRIDE_H_

REQUESTED INFORMATION

Make sure your have performed every step and checked the applicable boxes before submitting your issue. Thank you!

[x] Read the Contributing Guide and Policy and the Code of Conduct
[x] Searched the problem in issues
[x] Searched the problem in discussions
[x] Searched the problem in the docs
[x] Searched the problem in the chat
[x] Device used (e.g., Sonoff Basic): Sonoff S31
[x] Tasmota binary firmware version number used: v13.3.0
- [ ] Pre-compiled
- [x] Self-compiled
[x] Flashing tools used: esptool.py

[x] Provide the output of command: Backlog Template; Module; GPIO 255:

Configuration output here:
00:26:28.720 CMD: Backlog Template; Module; GPIO 255
00:26:28.741 RSL: RESULT = {"NAME":"Generic","GPIO":[1,1,1,1,1,1,1,1,1,1,1,1,1,1],"FLAG":0,"BASE":18}
00:26:28.970 RSL: RESULT = {"Module":{"1":"Sonoff Basic"}}
00:26:29.175 RSL: RESULT = {"GPIO0":{"32":"Button1"},"GPIO1":{"0":"None"},"GPIO2":{"0":"None"},"GPIO3":{"0":"None"},"GPIO4":{"0":"None"},"GPIO5":{"0":"None"},"GPIO9":{"0":"None"},"GPIO10":{"0":"None"},"GPIO12":{"224":"Relay1"},"GPIO13":{"320":"Led_i1"},"GPIO14":{"0":"None"},"GPIO15":{"0":"None"},"GPIO16":{"0":"None"},"GPIO17":{"0":"None"}}
0

[x] If using rules, provide the output of this command: Backlog Rule1; Rule2; Rule3:
```
Not using rules
```

[x] Provide the output of this command: Status 0:


STATUS 0 output here:
00:27:38.665 CMD: Status 0
00:27:38.671 RSL: STATUS = {"Status":{"Module":1,"DeviceName":"Tasmota","FriendlyName":["Tasmota"],"Topic":"tasmota_93CA6D","ButtonTopic":"0","Power":0,"PowerOnState":3,"LedState":1,"LedMask":"FFFF","SaveData":1,"SaveState":1,"SwitchTopic":"0","SwitchMode":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"ButtonRetain":0,"SwitchRetain":0,"SensorRetain":0,"PowerRetain":0,"InfoRetain":0,"StateRetain":0,"StatusRetain":0}}
00:27:38.702 RSL: STATUS1 = {"StatusPRM":{"Baudrate":115200,"SerialConfig":"8N1","GroupTopic":"tasmotas","OtaUrl":"http://ota.tasmota.com/tasmota/release/tasmota.bin.gz","RestartReason":"Power On","Uptime":"0T00:01:42","StartupUTC":"2024-02-12T23:25:56","Sleep":50,"CfgHolder":4617,"BootCount":3,"BCResetTime":"2024-02-12T23:45:27","SaveCount":7,"SaveAddress":"F5000"}}
00:27:38.734 RSL: STATUS2 = {"StatusFWR":{"Version":"13.3.0(tasmota)","BuildDateTime":"2024-02-12T17:43:42","Boot":31,"Core":"2_7_4_9","SDK":"2.2.2-dev(38a443e)","CpuFrequency":80,"Hardware":"ESP8266EX","CR":"382/699"}}
00:27:38.753 RSL: STATUS3 = {"StatusLOG":{"SerialLog":2,"WebLog":2,"MqttLog":0,"SysLog":0,"LogHost":"","LogPort":514,"SSId":["smokey",""],"TelePeriod":300,"Resolution":"558180C0","SetOption":["00008009","2805C80001000600003C5A0A192800000000","00000080","00206000","00004000","00000000"]}}
00:27:38.785 RSL: STATUS4 = {"StatusMEM":{"ProgramSize":697,"Free":304,"Heap":18,"ProgramFlashSize":1024,"FlashSize":4096,"FlashChipId":"16405E","FlashFrequency":40,"FlashMode":"DOUT","Features":["0809","8F9AC7C7","04368001","000000CF","010013C0","C000F981","00004004","00001000","54000020","00000080","00000000"],"Drivers":"1,2,3,4,5,6,7,8,9,10,12,16,18,19,20,21,22,24,26,27,29,30,35,37,45,62,68","Sensors":"1,2,3,4,5,6","I2CDriver":"7"}}
00:27:38.818 RSL: STATUS5 = {"StatusNET":{"Hostname":"tasmota-93CA6D-2669","IPAddress":"192.168.7.113","Gateway":"192.168.7.1","Subnetmask":"255.255.255.0","DNSServer1":"192.168.7.1","DNSServer2":"0.0.0.0","Mac":"C8:C9:A3:93:CA:6D","Webserver":2,"HTTP_API":1,"WifiConfig":4,"WifiPower":17.0}}
00:27:38.842 RSL: STATUS6 = {"StatusMQT":{"MqttHost":"mqtt.<my server>.io","MqttPort":8883,"MqttClientMask":"DVES_%06X","MqttClient":"DVES_93CA6D","MqttUser":"DVES_USER","MqttCount":0,"MAX_PACKET_SIZE":1200,"KEEPALIVE":30,"SOCKET_TIMEOUT":4}}
00:27:38.867 RSL: STATUS7 = {"StatusTIM":{"UTC":"2024-02-12T23:27:38","Local":"2024-02-13T00:27:38","StartDST":"2024-03-31T02:00:00","EndDST":"2024-10-27T03:00:00","Timezone":"+01:00","Sunrise":"08:02","Sunset":"18:05"}}
00:27:38.884 RSL: STATUS10 = {"StatusSNS":{"Time":"2024-02-13T00:27:38"}}
00:27:38.891 RSL: STATUS11 = {"StatusSTS":{"Time":"2024-02-13T00:27:38","Uptime":"0T00:01:42","UptimeSec":102,"Heap":18,"SleepMode":"Dynamic","Sleep":50,"LoadAvg":19,"MqttCount":0,"POWER":"OFF","Wifi":{"AP":1,"SSId":"smokey","BSSId":"3C:7C:3F:E6:61:B8","Channel":2,"Mode":"11n","RSSI":100,"Signal":-37,"LinkCount":1,"Downtime":"0T00:00:03"}}}

- [x] Set `weblog` to 4 and then, when you experience your issue, provide the output of the Console log:
```lua
  Console output here:
00:32:36.464 CMD: weblog 4
00:32:36.468 RSL: RESULT = {"WebLog":4}
00:32:36.601 CFG: Saved to flash at F4, Count 8, Bytes 4096
00:32:39.788 WIF: Checking connection...
00:32:57.409 WIF: Sending Gratuitous ARP
00:32:59.763 WIF: Checking connection...
00:33:19.774 WIF: Checking connection...
00:33:39.771 WIF: Checking connection...
00:33:40.025 MQT: Attempting connection...
00:33:40.028 WIF: DNS resolved 'mqtt.<my server>.io' (my server ip) in 0 ms
00:33:40.088 MQT: TLS connection error: 296
00:33:40.090 MQT: Connect failed to mqtt.<my server>.io:8883, rc -2. Retry in 100 sec

TO REPRODUCE

Follow the procedure as outlined here: https://tasmota.github.io/docs/Self-signed-Mosquitto/
See problem description for details about why this is going wrong

EXPECTED BEHAVIOUR

A successful MQTT TLS session with a mosquitto broker using a tasmota device

Additional Context

Here is an issue that is similar and recent BUT does not work for me in this case as they are using Let's Encrypt and Home Assistant https://github.com/arendst/Tasmota/discussions/20192

(Please, remember to close the issue when the problem has been addressed)

arendst / Tasmota