Closed wz2b closed 5 months ago
dhalbert
commented
5 months ago Could you try 9.1.0-beta.2 and see if there is any difference in behavior?
Which TLS versions is your broker supporting? I am surprised, because I thought we supported at least TLSv1.2, if not also TLSv1.3 and we'd try the "best" one first.
There are test hosts here for TLSv1.0, v1.1, and v1.2: https://badssl.com/. You can just try a connect or a request from those.
wz2b
commented
5 months ago Yes, I would be happy to try the latest version! My broker is set for tls 1.3 and later, I looked at mosquitto.conf yesterday - when I get back to my office I will post an updated mosquitto.conf to confirm this, and I can try beta.2.
On Tue, May 21, 2024 at 12:14 PM Dan Halbert @.***> wrote:
Could you try 9.1.0-beta.2 and see if there is any difference in behavior?
Which TLS versions is your broker supporting? I am surprised, because I thought we supported at least TLSv1.2, if not also TLSv1.3 and we'd try the "best" one first.
There are test hosts here for TLSv1.0, v1.1, and v1.2: https://badssl.com/. You can just try a connect or a request from those.
— Reply to this email directly, view it on GitHub https://github.com/adafruit/circuitpython/issues/9265#issuecomment-2122980343, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALTMOPJS3WYCMQE5S5FAQTZDNXGVAVCNFSM6AAAAABIB4ELY2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRSHE4DAMZUGM . You are receiving this because you authored the thread.Message ID: @.***>
dhalbert
commented
5 months ago Can you try setting it to v1.2? Is it v1.3 only? v1.3 may not be supported.
wz2b
commented
5 months ago Setting the broker to use v1.2 and even v1.1 still didn't work, but now I'm thinking maybe the TLS version isn't the issue:
1716314886: OpenSSL Error[0]: error:1402542E:SSL routines:ACCEPT_SR_CLNT_HELLO:tlsv1 alert protocol version
1716314886: Client <unknown> disconnected: Protocol error.I don't know. Mosquitto is pretty standard - it's probably the #1 MQTT broker used by people who use CircuitPython (I would imagine) so I'm mystified. Trying beta.2 now.
dhalbert
commented
5 months ago The M5Dial has an M5Stamp inside, which is 8MB flash, but no PSRAM. We've seen problems with running out of memory on similar configurations when setting up HTTPS with your own certificates. Is the logging you're showing here from mosquitto? What is being printed on the REPL serial port in CircuitPython?
If you could come up with a minimal example, that would be great. And show us the (readacted as needed) mosquitto config.
dhalbert
commented
5 months ago Also update the libraries to the latest as of today. There have been changes even today that will not be in the bundle until tonight.
wz2b
commented
5 months ago Here's a really simple example:
import time
import board
import busio
from digitalio import DigitalInOut
import adafruit_minimqtt.adafruit_minimqtt as MQTT
import wifi
import socketpool
import ssl
print("Connecting WiFi")
r = wifi.radio
r.connect("RIT-WiFi")
print(r.connected)
# Define callback methods which are called when events occur
def connected(client, userdata, flags, rc):
print("Connected to MQTT broker!")
def disconnected(client, userdata, rc):
print("Disconnected from MQTT broker!")
def publish(client, userdata, topic, pid):
print("Published to {0} with PID {1}".format(topic, pid))
# Set up a socket pool
pool = socketpool.SocketPool(wifi.radio)
ssl_context = ssl.create_default_context()
ssl_context.check_hostname = False
# ssl_context.options |= ssl.OP_NO_TLSv1 | ssl.OP_NO_TLSv1_1 # Disable TLS 1.0 and 1.1
# ssl_context.verify_mode = ssl.CERT_NONE
# Create a MiniMQTT client
mqtt_client = MQTT.MQTT(
broker="192.168.0.99",
port=8883,
username="user",
password="password",
socket_pool=pool,
ssl_context=ssl_context
)
# Connect callback handlers to client
mqtt_client.on_connect = connected
mqtt_client.on_disconnect = disconnected
mqtt_client.on_publish = publish
# Connect to the MQTT broker
mqtt_client.connect()
# Publish a message
mqtt_client.publish("test/topic", "Hello from CircuitPython!")
# Disconnect from the MQTT broker
mqtt_client.disconnect()
# Loop forever doing nothing, as an example
while True:
pass# mosquitto.conf
per_listener_settings true
persistence true
persistence_location /mosquitto/data
listener 1883
password_file /mosquitto/passwd
listener 8883
# You may want to leave this out entirely
password_file /mosquitto/passwd
certfile /run/secrets/server_certificate
keyfile /run/secrets/server_certificate_key
# This can be tlsv1.2 or tlsv1.1, too. This key specifies
# the MINIMUM level, so if you specify 1.2, then 1.3 will work
# as well.
tls_version tlsv1.3
# Only provide support for GCM cipher modes - disabled while we try to figure this out
# ciphers_tls1.3 TLS_CHACHA20_POLY1305_SHA256:TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256Notice I commented out all the requirements on things like cipher suites to make things a little looser, just until we can figure this out. This is mosquitto 2.0.18 which is the most recent eclipse-mosquitto image in dockerhub.
wz2b
commented
5 months ago Just for the fun of it, I converted umqtt.simple to work with the socketpool, and got the same exact results:
import struct
import socketpool
import wifi
import ssl
class MQTTException(Exception):
pass
class MQTTClient:
def __init__(self, client_id, server, port=0, user=None, password=None, keepalive=0,
ssl=False, ssl_params={}):
if port == 0:
port = 8883 if ssl else 1883
self.client_id = client_id
self.sock = None
self.server = server
self.port = port
self.ssl = ssl
self.ssl_params = ssl_params
self.pid = 0
self.cb = None
self.user = user
self.pswd = password
self.keepalive = keepalive
self.lw_topic = None
self.lw_msg = None
self.lw_qos = 0
self.lw_retain = False
def _send_str(self, string):
self.sock.send(len(string).to_bytes(2, 'big'))
self.sock.send(string.encode('utf-8'))
def _recv_len(self):
n = 0
sh = 0
while True:
b = self.sock.recv(1)[0]
n |= (b & 0x7f) << sh
if not b & 0x80:
return n
sh += 7
def set_callback(self, f):
self.cb = f
def set_last_will(self, topic, msg, retain=False, qos=0):
assert 0 <= qos <= 2
assert topic
self.lw_topic = topic
self.lw_msg = msg
self.lw_qos = qos
self.lw_retain = retain
def connect(self, clean_session=True):
pool = socketpool.SocketPool(wifi.radio)
self.sock = pool.socket(pool.AF_INET, pool.SOCK_STREAM)
# Resolve address and connect
addr_info = pool.getaddrinfo(self.server, self.port)
addr = addr_info[0][-1]
self.sock.connect(addr)
if self.ssl:
context = ssl.create_default_context()
self.sock = context.wrap_socket(self.sock, server_hostname=self.server, **self.ssl_params)
premsg = bytearray(b"\x10\0\0\0\0\0")
msg = bytearray(b"\x04MQTT\x04\x02\0\0")
sz = 10 + 2 + len(self.client_id)
msg[6] = clean_session << 1
if self.user is not None:
sz += 2 + len(self.user) + 2 + len(self.pswd)
msg[6] |= 0xC0
if self.keepalive:
assert self.keepalive < 65536
msg[7] |= self.keepalive >> 8
msg[8] |= self.keepalive & 0x00FF
if self.lw_topic:
sz += 2 + len(self.lw_topic) + 2 + len(self.lw_msg)
msg[6] |= 0x4 | (self.lw_qos & 0x1) << 3 | (self.lw_qos & 0x2) << 3
msg[6] |= self.lw_retain << 5
i = 1
while sz > 0x7f:
premsg[i] = (sz & 0x7f) | 0x80
sz >>= 7
i += 1
premsg[i] = sz
self.sock.send(premsg[:i + 2])
self.sock.send(msg)
self._send_str(self.client_id)
if self.lw_topic:
self._send_str(self.lw_topic)
self._send_str(self.lw_msg)
if self.user is not None:
self._send_str(self.user)
self._send_str(self.pswd)
resp = self.sock.recv(4)
assert resp[0] == 0x20 and resp[1] == 0x02
if resp[3] != 0:
raise MQTTException(resp[3])
return resp[2] & 1
def disconnect(self):
self.sock.send(b"\xe0\0")
self.sock.close()
def ping(self):
self.sock.send(b"\xc0\0")
def publish(self, topic, msg, retain=False, qos=0):
pkt = bytearray(b"\x30\0\0\0")
pkt[0] |= qos << 1 | retain
sz = 2 + len(topic) + len(msg)
if qos > 0:
sz += 2
assert sz < 2097152
i = 1
while sz > 0x7f:
pkt[i] = (sz & 0x7f) | 0x80
sz >>= 7
i += 1
pkt[i] = sz
#print(hex(len(pkt)), hexlify(pkt, ":"))
self.sock.send(pkt[:i + 1])
self._send_str(topic)
if qos > 0:
self.pid += 1
pid = self.pid
struct.pack_into("!H", pkt, 0, pid)
self.sock.send(pkt[:2])
self.sock.send(msg)
if qos == 1:
while True:
op = self.wait_msg()
if op == 0x40:
sz = self.sock.recv(1)
assert sz == b"\x02"
rcv_pid = self.sock.recv(2)
rcv_pid = rcv_pid[0] << 8 | rcv_pid[1]
if pid == rcv_pid:
return
elif qos == 2:
assert 0
def subscribe(self, topic, qos=0):
assert self.cb is not None, "Subscribe callback is not set"
pkt = bytearray(b"\x82\0\0\0")
self.pid += 1
struct.pack_into("!BH", pkt, 1, 2 + 2 + len(topic) + 1, self.pid)
#print(hex(len(pkt)), hexlify(pkt, ":"))
self.sock.send(pkt)
self._send_str(topic)
self.sock.send(qos.to_bytes(1, "little"))
while True:
op = self.wait_msg()
if op == 0x90:
resp = self.sock.recv(4)
#print(resp)
assert resp[1] == pkt[2] and resp[2] == pkt[3]
if resp[3] == 0x80:
raise MQTTException(resp[3])
return
# Wait for a single incoming MQTT message and process it.
# Subscribed messages are delivered to a callback previously
# set by .set_callback() method. Other (internal) MQTT
# messages processed internally.
def wait_msg(self):
res = self.sock.recv(1)
self.sock.setblocking(True)
if res is None:
return None
if res == b"":
raise OSError(-1)
if res == b"\xd0": # PINGRESP
sz = self.sock.recv(1)[0]
assert sz == 0
return None
op = res[0]
if op & 0xf0 != 0x30:
return op
sz = self._recv_len()
topic_len = self.sock.recv(2)
topic_len = (topic_len[0] << 8) | topic_len[1]
topic = self.sock.recv(topic_len)
sz -= topic_len + 2
if op & 6:
pid = self.sock.recv(2)
pid = pid[0] << 8 | pid[1]
sz -= 2
msg = self.sock.recv(sz)
self.cb(topic, msg)
if op & 6 == 2:
pkt = bytearray(b"\x40\x02\0\0")
struct.pack_into("!H", pkt, 2, pid)
self.sock.send(pkt)
elif op & 6 == 4:
assert 0
# Checks whether a pending message from server is available.
# If not, returns immediately with None. Otherwise, does
# the same processing as wait_msg.
def check_msg(self):
self.sock.setblocking(False)
return self.wait_msg()It's odd that the SSL module doesn't support options like verify certificate (to false) and TLS version - is this to save code space?
dhalbert
commented
5 months ago We are using mbedtls under the covers. It may or may not provide some of this functionality. We implemented a subset to cover most use cases. Additions are welcome via PR.
wz2b
commented
5 months ago I would love to but I feel like I'd be over my head when it comes to TLS ....
wz2b
commented
5 months ago Is this possibly related? https://forums.mbed.com/t/tls-version-number-in-a-client-hello-packet/4863
dhalbert
commented
5 months ago Looking at the compilations settings, the Espressif boards are compiled to support TLSv1.2. The link you posted above is not what the source code looks like any more. It enforces a minimum TLS version based on the compilation options.
wz2b
commented
5 months ago This is really stretching what I know about TLS but you're right. I wrote a small shim around circuitpython's socket class so I could inject some logging. The first thing it sends is:
16 03 03 00 d3 01 00 00 cf 03 03 00 01 05 f1 fa ee 98 08 fe 52 73 ee 93 3f 46 62 a0 6b a0 8b a8 49 1e 6d 6a db b0 b4 8a 18 7e 62 00 00 5a c0 2c c0 30 c0 ad c0 24 c0 28 c0 0a c0 14 c0 af c0 5d c0 61 c0 49 c0 4d c0 2b c0 2f c0 ac c0 23 c0 27 c0 09 c0 13 c0 ae c0 5c c0 60 c0 48 c0 4c c0 32 c0 2a c0 0f c0 2e c0 26 c0 05 c0 5f c0 63 c0 4b c0 4f c0 31 c0 29 c0 0e c0 2d c0 25 c0 04 c0 5e c0 62 c0 4a c0 4e 00 ff 01 00 00 4c 00 00 00 18 00 16 00 00 13 74 65 73 74 62 65 64 2e 67 69 73 2e 72 69 74 2e 65 64 75 00 0a 00 08 00 06 00 1d 00 17 00 18 00 0d 00 0e 00 0c 06 03 06 01 05 03 05 01 04 03 04 01 00 0b 00 02 01 00 00 16 00 00 00 17 00 00 00 23 00 00
Breakdown
This problem appears to be not what I think it is. I'm going to close this ticket. Sorry for the noise but thank you for helping me work through this. I'll leave this info here just in case someone else stumbles across a similar question in the future.
dhalbert
commented
5 months ago No problem - I am looking at a number of things about the SSL implementation that I had not tried to understand in detail previously, in order to figure out why we're having other problems, such as memory issues. For instance, I did not know about the TLSv1.2 compilation option.
wz2b
commented
5 months ago If this helps, you can do this:
underlying_sock = pool.socket(pool.AF_INET, pool.SOCK_STREAM)
sock = LoggingSocket(underlying)all it does is spits out the writes. For what I was doing (trying to see the TLS handshake, which is the very first thing it does) it was helpful. Mainly because I discovered that if you paste the hex to ChatGPT and tell it that it's a TLS negotiation, it will decode the whole thing for you.
class LoggingSocket:
def __init__(self, sock):
self._sock = sock
def send(self, data):
# Log the data being sent as hexadecimal
hex_data = ' '.join(f'{b:02x}' for b in data)
print(f'Sending data: {hex_data}')
return self._sock.send(data)
def recv(self, bufsize):
return self._sock.recv(bufsize)
def connect(self, addr):
return self._sock.connect(addr)
def close(self):
return self._sock.close()
def setblocking(self, flag):
return self._sock.setblocking(flag)
def __getattr__(self, name):
# Delegate attribute access to the underlying socket
return getattr(self._sock, name)
I'm using circuitpython 9.0.4 on an m5stack dial and it works great, but I can't connect to my MQTT broker:
I would like to be able to:
but there is no .options
Is there any way to force TLS v1.2? If not, this is a feature request.
Adafruit CircuitPython 9.0.4 on 2024-04-16; M5Stack Dial with ESP32S3