SSL issues

[errorfixrepeat]

I am trying to move over my project onto this library, but am having some issues with SSL. I see that you didn't have much luck, but these settings work with MQTT.robust (albeit slowly). I've tried searching for ssl_handshake_status: -4 to no avail.

The code:

from mqtt_as import MQTTClient
from mqtt_as import config
import uasyncio as asyncio

CERT_FILE = "/flash/8266-01.cert.der"
KEY_FILE = "/flash/8266-01.key.der"
#if you change the ClientId make sure update AWS policy
MQTT_CLIENT_ID = "basicPubSub"
MQTT_PORT = 8883
#if you change the topic make sure update AWS policy
MQTT_TOPIC = "sdk/test/Python"
MQTT_HOST = "amazon endpoint"

try:
    with open(KEY_FILE, "r") as f: 
        key = f.read()
#         print("Got Key")

    with open(CERT_FILE, "r") as f: 
        cert = f.read()
#         print("Got Cert") 
except:
    print('oh fuck')

def callback(topic, msg):
    print((topic, msg))

async def conn_han(client):
    await client.subscribe('foo_topic', 1)

async def main(client):
    await client.connect()
    n = 0
    while True:
        await asyncio.sleep(5)
        print('publish', n)
        # If WiFi is down the following will pause for the duration.
        await client.publish('result', '{}'.format(n), qos = 1)
        n += 1

config['subs_cb'] = callback
config['connect_coro'] = conn_han
config['server'] = MQTT_HOST
config['port'] = MQTT_PORT
config['ssl'] = True
config['ssl_params'] = {"cert":cert, "key":key, "server_side":False}

MQTTClient.DEBUG = True  # Optional: print diagnostic messages
client = MQTTClient(config)
loop = asyncio.get_event_loop()
try:
    loop.run_until_complete(main(client))
finally:
    client.close()  # Prevent LmacRxBlk:1 errors

Here's the error:

Connecting to broker.
ssl_handshake_status: -4
Traceback (most recent call last):
  File "<stdin>", line 57, in <module>
  File "<stdin>", line 55, in <module>
  File "uasyncio/core.py", line 187, in run_until_complete
  File "uasyncio/core.py", line 161, in run_forever
  File "uasyncio/core.py", line 116, in run_forever
  File "uasyncio/core.py", line 184, in _run_and_stop
  File "<stdin>", line 35, in main
  File "mqtt_as.py", line 514, in connect
  File "mqtt_as.py", line 246, in _connect

[peterhinch]

It may be some while before I can look at this: I'm recovering from an operation.

I think SSL support on nonblocking sockets is under development: see this issue. You might like to follow that line of inquiry. Please report back if you discover anything.

[ernstnaezer]

@errorfixrepeat it seems like the issue referenced by @peterhinch is closed and merged. Did you ever get managed to get your sample working over SSL?

I'm trying myself and get a hard crash / reboot. Suggestion on how to debug this are much appreciated. the crash was fixed with #27.

>>> import test
reading private key
reading thing certificate
connecting to mqtt server
Connecting to broker.
ssl_handshake_status: -4

[peterhinch]

I pushed an update yesterday. The current state is that SSL/TLS works on Pyboard D (see tls.py) but similar code fails with handshake status errors on ESP8266 and ESP32 (tls8266.py and tls32.py).

The fact that it works on Pyboard D implies that the problem with TLS on nonblocking sockets has been fixed in current firmware.

I'm out of my depth on this one and any help would be much appreciated.

[ernstnaezer]

Thanks for looking into this. Unfortunately not my area of expertise as well.

What I managed to dig up is that -4 is defined as SSL_EAGAIN in this commit: https://github.com/pfalcon/axtls/commit/dac9176cac58cc5e49669a9a4d404a6f6dd7cc10

"Initial handshake and writes still don't support non-blocking mode and must be done in the blocking way." as per: https://github.com/micropython/micropython/pull/3395

Also; "And in fact CPython requires that the underlying socket be in blocking mode when you wrap it (as long as that socket is already connected), otherwise the SSL handshake won't work. Since uPy only supports wrapping a socket that's already connected, the socket must be in blocking mode when it's passed to wrap_socket." https://github.com/micropython/micropython/pull/3226#discussion_r128948172

[peterhinch]

Thank you for looking into this. The references you cite date back a couple of years.

The fact that it now works on the Pyboard D suggests that the fundamental problem has been fixed. Perhaps this work hasn't yet been ported to ESPx platforms?

[ernstnaezer]

Hi Peter, yes that's true.

There is also a recent commit that adds non-blocking mode support to modussl_axtls.c that I hadn't seen yet. It specifically mentions the SSL_EAGAIN code: https://github.com/micropython/micropython/commit/c76445315f836eb75835814ebe97e14107adbf0f#diff-22ec0e424cb91ec8ae70c108e08f9a48R139-R150

Perhaps the ESP8266 is a bit slower and this is a timing issue?

I don't have by devices at hand today, but could it be that this line is missing an await statement? https://github.com/peterhinch/micropython-mqtt/blob/master/mqtt_as/mqtt_as.py#L232

[peterhinch]

The commit is dated 1 March. My testing on ESP8266 is with firmware built from very recent source.

Your suggestion of a timing issue is possible.

The socket.connect method is synchronous see docs so an await would not work.

[ernstnaezer]

Not much has changed since that commit in these modules. I've also tested with a build made from master this week. Perhaps indeed wait a bit longer, or do a retry as the error code and comment suggests. I'll give it another try. Thanks

btw - yes good one on socket.connect - I got thrown off by Github's reference to line 525 in that same file, by bad.

[peterhinch]

I have had feedback from Damien on this issue:

stm32 and esp32 use the same mbedtls wrapper, but different underlying mbedtls. I would have thought they would both support non-blocking mode in the same way but it sounds like esp32 is different.... not sure why, would have to dig deeper there. For esp8266, axtls just doesn't support non-blocking sockets. I don't know if Paul ever got this working (he did do some work on it) but I don't really have plans to get it working myself. I'd be happy to pull in Paul's changes to axtls if he did manage to make non-blocking work. Otherwise we could make available esp8266 builds with mbedtls, then it'd work exactly the same as stm32.

[ernstnaezer]

Otherwise we could make available esp8266 builds with mbedtls, then it'd work exactly the same as stm32.

That would be fantastic.

[eRJx]

Hi

I am thinking of temporarily changing non-blocking sockets for blocking sockets while wrapping socket for ssl in such a way:

_mqttas.py:

async def _connect(self, clean):
    self._sock = socket.socket()
    self._sock.setblocking(True)
    try:
        self._sock.connect(self._addr)
    except OSError as e:
        if e.args[0] not in BUSY_ERRORS:
            raise
    await asyncio.sleep_ms(_DEFAULT_MS)
    self.dprint('Connecting to broker.')
    if self._ssl:
        import ussl
        self._sock = ussl.wrap_socket(self._sock, **self._ssl_params)
    premsg = bytearray(b"\x10\0\0\0\0\0")
    msg = bytearray(b"\x04MQTT\x04\0\0\0")  # Protocol 3.1.1
    self._sock.setblocking(False) 
    sz = 10 + 2 + len(self._client_id)

What consequences can be expected

[peterhinch]

I have no idea whether this will work; I have my doubts. Damien has written

For esp8266, axtls just doesn't support non-blocking sockets. I don't know if Paul ever got this working (he did do some work on it) but I don't really have plans to get it working myself. I'd be happy to pull in Paul's changes to axtls if he did manage to make non-blocking work. Otherwise we could make available esp8266 builds with mbedtls, then it'd work exactly the same as stm32.

[peterhinch]

SSL/TLS is now fixed.