using PicoMQTT with websockets

[mhaberler]

Hi,

I appreciate your project - works much better for me than other Arduino MQTT broker projects

I would like to use your project with MQTT-over-websockets clients

I came up with a simple websockets-to-tcp proxy on a separe esp32: https://github.com/mhaberler/arduino-websockets-tcp-proxy

this works fine with several MQTT-over-websockets clients, including paho-mqtt Python, paho-mqtt JS, and MQTT.js

however, as soon as I fold the websockets-to-tcp proxy into the same code as PicoMQTT, things get stuck during the MQTT connection handshake phase with the websockets clients- it seems the broker disconnects: https://github.com/mhaberler/PicoMQTT-websockets

I’m asking for a look over my shoulder as I think this might be of wider interest and use - I suspice there is something wrong with scheduling, yield or connect handling and I cant figure out why - I seem to violate some assumption which I am not aware of

thanks in advance!

Michael

[mhaberler]

I observed that a paho mqtt/Python client works fine, whereas paho mqtt/js and MQTT.js get stuck during connect

this is a paho mqtt/Python trace: Python paho-mqtt seems to send the CONNECT frame in one go

this is a paho mqtt/ws subscriber trace:

whereas the paho mqtt/ws (and MQTT.ws) clients send several fragments of the CONNECT message, each with TCP PUSH bit set

looks like a timeout during reassembly of the CONNECT websockets fragments, when the client disconnects after a timeout (packet 86)

I'm using this dissector: https://github.com/nberthet/wireshark-mqtt-ws-plugin (seems to be struggling with fragmented websockets frames)

the PCAP files are at https://static.mah.priv.at/cors/paho-python.pcapng and https://static.mah.priv.at/cors/paho-js.pcapng

thanks in advance,

Michael

ps: reading up, found this blocking loop:

https://github.com/mlesniew/PicoMQTT/blob/master/src/PicoMQTT/connection.cpp#L62-L78

that would prevent scheduling of other code and explain the above

would you suggest to move the proxy server to a separate task? Or do you see a way to handle this through yield() or some other trick?

[mlesniew]

Hey Michael, thanks for trying to extend the library -- many people were asking about websocket support.

You're right that the library locks up at this point: https://github.com/mlesniew/PicoMQTT/blob/master/src/PicoMQTT/connection.cpp#L62-L78.

Your solution creates a web server socket to a simple tcp client socket proxy. Everything that is received by the server is shoveled to the client socket. Next you've configured the proxy to connect to PicoMQTT's broker port.

This is an interesting hack, but as you've noticed it will not work correctly in some cases. In the loop where the lockup happens, the PicoMQTT broker expects data to be received on its socket. It will spin within that loop until a it gets all the required data or hits a timeout.

Everything that already accumulated in the PicoMQTT's socket buffer will be read by the library fine. However, if more data is expected, it will never be received. This is because the ProxyWebSocketsServer might not have yet sent the data. To make it forward data received on its websocket, its loop() would have to be called.

To fix this we would have to find a way to call ProxyWebSocketsServer::loop() from within this loop.

Unfortunately this is not the only place where the library could lock up because of this. So to really make this work in all cases, the Connection class (https://github.com/mlesniew/PicoMQTT/blob/640a3326fe6ca12ee673b38ea23e3a8c370d2790/src/PicoMQTT/connection.h#L25) would have to be extended.

I also see another problem. Taking the approach you proposed will have significantly degraded performance and high resource use. The solution uses three TCP sockets for each client instead of just one. Each TCP connection is costly, especially on microcontroller boards, because of the need to maintain buffers and tracking what has been sent and what hasn't.

It would also be best to forward data received on the web socket directly into PicoMQTT instead of sending it through yey another socket.

It would be ideal to introduce a class that inherits from Arduino's Client (https://github.com/esp8266/Arduino/blob/eda4e0855fa5ebcf1d7f621f35f25f6bab503335/cores/esp8266/Client.h#L26), just like WiFiClient, but internally encapsulates web socket communication. PicoMQTT could then be extended to use that new class instead of the current one (perhaps leveraging c++ templates).

This library seems like a good starting point: https://github.com/u0078867/Arduino-Websocket-Fast for doing that, but it's still a ton of work...

[mhaberler]

Hello Michael,

thanks for your thorough answer, appreciated

on your second issue:

the proxy server idea certainly is suboptimal as far as resource usage and speed goes - it was not intended as a permanent solution but to get mqtt-over-websockets going quickly while I learn the API’s, and look for a better solution while that box is ticked, so no PR coming for that even if it works - extending PicoMQTT to support websockets directly certainly is the way to go

that said:

until that integration is done, I need something working so others can play with mqtt-over-websockets clients, so I’ll see if I can get the proxy server to work either with the callout you mentioned, or by placing yield() appropriately in PicoMQTT code and moving the proxy server to a bona-fide task which then gets a chance to run

I’d be challenged to venture for the direct integration of the library you mentioned, at least I’d need patient help at it

However, I am prepared to support work for direct websockets integration to a reasonable extent - let’s talk terms, see private email

Michael

[mhaberler]

I have a working interim solution

it was in fact quite simple:

• in the Connection::wait_for_reply loop, insert a yield()
• move the websockets proxy to a separate RTOS thread
• an extra yield which I'm not sure if it's actually required

while not very fast, this works fine with Paho MQTT/Python and JS with websockets transports, as well as MQTT.js

good enough for now until we have something more polished.

[mlesniew]

The library now supports websocket servers and clients as described in the readme.

I have now released the latest changes as version 1.0.0. It should soon become installable via PlatformIO and the Arduino IDE.