Feature Request: PortentaH7 MQTT Server

[hpssjellis]

Feature Request: PortentaH7 MQTT Server

https://github.com/arduino-libraries/ArduinoMqttClient works fine on the PortentaH7 to connect to an MQTT server, but things would make so much more sense if the Portenta could actually be a light weight MQTT server for testing purposes.

Sorry @facchinm, hopefully this is one of my last requests before I start updating my Robotics Curriculum using the PortentaH7 and the LoRa Vision Shield with EdgeImpulse.com?

There are lots of Cloud MQTT servers, and the Raspberry Pi can easily connect the mosquitto MQTT server, but The Portenta should be able to handle MQTT. I have found this cross compile here.

If the Portenta can't do it could the RP2040 Connect be able to run an MQTT server? I work best with simple example code.

...

Digging around I have found an ESP32 MQTT server. Wondering if anyone has the skills to convert this library to the Portenta.

https://github.com/hsaturn/TinyMqtt

I put a topic in the Arduino Portenta forum. Has a bit more information there. https://forum.arduino.cc/t/portenta-as-an-mqtt-broker/871412

[hpssjellis]

@facchinm

Not as crazy as it sounds: This would give the M4 core something to do for anyone using their local network either with WiFi or Ethernet. Would be a big boost for people only wanting to use the Portenta instead of dealing with other systems.

Feature Request: Small MQTT Broker (server) for Portenta

A small, perhaps max 10 topics and max 30 subscribers local MQTT broker, would allow users to keep data within a local network without having to purchase and operate a Raspberry Pi running the mosquito broker.

I can probably design the code, I just need a little help understanding the MQTT handshake, but am presently researching it. Any suggestions?

[hpssjellis]

So I got tinyMQTT broker a standard Arduino library for the ESP32 working on a Node MCU-32S, so it can be done. Probably the best bet is to try to convert this library to the Portenta. Any Arduino people with experience converting from ESP32 to MBEd feel free to give me advice. I assume it is very hard to do.

[facchinm]

:wave: I tried to compile the simple-broker example after applying this patch

diff --git a/src/TinyMqtt.h b/src/TinyMqtt.h
index b1a1cd3..6919c48 100644
--- a/src/TinyMqtt.h
+++ b/src/TinyMqtt.h
@@ -14,6 +14,10 @@
        #ifdef TCP_ASYNC
          #include <AsyncTCP.h> // https://github.com/me-no-dev/AsyncTCP
   #endif
+#elif defined(ARDUINO_PORTENTA_H7_M7)
+  #include <WiFi.h>
+  #include <WiFiClient.h>
+  #include <WiFiServer.h>
 #endif
 #ifdef EPOXY_DUINO
   #define dbg_ptr uint64_t

removing WiFi.mode(WIFI_STA); and it compiles just fine :slightly_smiling_face: Didn't test it though.

[hpssjellis]

Thanks @facchinm so much for starting converting tinyMQTT to mbed. I will PR your code if it works.

The error I consistently get is

Exception in thread "Thread-258" java.util.ConcurrentModificationException
 lots more errors ...

Using their code I get the hard fault red flashing LED almost immediately. Using my code I don't get the red flashing, and can connect to WiFi but still get the same error.

I feel that I am doing something wrong with WiFiServer and/or WiFiClient.

#include "Arduino.h"
#include "WiFi.h"

#include "TinyMqtt.h"   // https://github.com/hsaturn/TinyMqtt

/** Basic Mqtt Broker
  *
  *  +-----------------------------+
  *  | ESP                         |
  *  |       +--------+            | 
  *  |       | broker |            | 1883 <--- External client/s
  *  |       +--------+            |
  *  |                             |
  *  +-----------------------------+
  */

#define PORT 1883
MqttBroker broker(PORT);

// Choose either the following arduino_secrets.h tab and bracket out the next 2 lines after it
// That route is more secure.
// Or just change the lines below for your Network and Password. Easier but not as secure
// Since if you share this file it will have your info on it

//#include "arduino_secrets.h"   // more safe
#define SECRET_SSID "YOUR SSID HERE"
#define SECRET_PASS "YOUR PASSWORD HERE"

char ssid[] = SECRET_SSID;    // Changing the secret strings to a character array
char pass[] = SECRET_PASS;    
int keyIndex = 0;              

int status = WL_IDLE_STATUS;
WiFiServer server(80);

void setup() {
  Serial.begin(115200);      // initialize serial communication
  delay(5000);
  Serial.println("Wait a bit to connect serial monitor");
  delay(5000);
  Serial.println("Wait a bit");
  delay(5000);
  Serial.println("Wait a bit");

  pinMode(LED_BUILTIN, OUTPUT);      // set the LED pin mode
  pinMode(LEDB, OUTPUT);      // set the LED pin mode
  digitalWrite(LEDB, LOW);  

  // attempt to connect to Wifi network:
  while (status != WL_CONNECTED) {
    Serial.print("Attempting to connect to Network named: ");
    Serial.println(ssid);                   // print the network name (SSID);

    // Connect to WPA/WPA2 network. Change this line if using open or WEP network:
    status = WiFi.begin(ssid, pass);
    // wait 5 seconds for connection:
    delay(5000);
  }
  server.begin();                           // start the web server on port 80
  printWifiStatus();                        // you're connected now, so print out the status
  digitalWrite(LEDB, HIGH);  
  digitalWrite(LED_BUILTIN, LOW); 

  broker.begin();

}

void loop() {

    broker.loop();

}

void printWifiStatus() {
  // print the SSID of the network you're attached to:
  Serial.print("SSID: ");
  Serial.println(WiFi.SSID());

  // print your board's IP address:
  IPAddress ip = WiFi.localIP();
  Serial.print("IP Address: ");
  Serial.println(ip);

  // print the received signal strength:
  long rssi = WiFi.RSSI();
  Serial.print("signal strength (RSSI):");
  Serial.print(rssi);
  Serial.println(" dBm");
  // print where to go in a browser:
  Serial.print("To see this page in action, open a browser to http://");
  Serial.println(ip);
}

I have also tried using Ethernet and that probably causes extra issues. The error is different

Exception in thread "AWT-EventQueue-0" java.util.ConcurrentModificationException
    at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
    at java.util.LinkedList$ListItr.next(LinkedList.java:888)

I feel that Ethernet would be a more stable MQTT server anyway. Here is my code.

#include <Portenta_Ethernet.h>
#include <Ethernet.h>
#include "TinyMqtt.h"   // https://github.com/hsaturn/TinyMqtt

//#define WifiClient EthernetClient
//#define WifiServer EthernetServer

//#define TcpClient = EthernetClient
//#define TcpServer = EthernetServer

//using TcpClient = EthernetClient;
//using TcpServer = EthernetServer;

#define PORT 1883

#define PORT 1883
MqttBroker broker(PORT);

/** Basic Mqtt Broker
  *
  *  +-----------------------------+
  *  | ESP                         |
  *  |       +--------+            | 
  *  |       | broker |            | 1883 <--- External client/s
  *  |       +--------+            |
  *  |                             |
  *  +-----------------------------+
  */

// Enter a MAC address and IP address for your controller below.
// The IP address will be dependent on your local network:
byte mac[] = {
  0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED
};
IPAddress ip(192, 168, 1, 177);  // what are these for??

// Initialize the Ethernet server library
// with the IP address and port you want to use
// (port 80 is default for HTTP):
EthernetServer server(80);

/*
  ArduinoMqttClient - WiFi Simple Sender

  This example connects to a MQTT broker and publishes a message to
  a topic once a second.

  The circuit:
  - Arduino MKR 1000, MKR 1010 or Uno WiFi Rev.2 board

  This example code is in the public domain.
*/

EthernetClient wifiClient;
//MqttClient mqttClient(wifiClient);

void setup() {
   // Open serial communications and wait for port to open:
  Serial.begin(115200);
  pinMode(LED_BUILTIN, OUTPUT);      // set the LED Green pin mode  
  pinMode(LEDB, OUTPUT);             
  digitalWrite(LEDB, HIGH);  
  digitalWrite(LED_BUILTIN, LOW);  

  delay(5000);  // time to get serial monitor working if needed
  Serial.println("Wait a few seconds for a non-blocking chance to activate the serial monitor");
  delay(5000);  
  Serial.println("Wait 5 more seconds for a non-blocking chance to activate the serial monitor");
  delay(5000);   
  Serial.println("Ethernet WebServer Example");

  // start the Ethernet connection and the server:
  Ethernet.begin(mac);

  // Check for Ethernet hardware present
  if (Ethernet.hardwareStatus() == EthernetNoHardware) {
    Serial.println("Ethernet shield was not found.  Sorry, can't run without hardware. :(");
    while (true) {
      delay(1); // do nothing, no point running without Ethernet hardware
    }
  }
  if (Ethernet.linkStatus() == LinkOFF) {
    Serial.println("Ethernet cable is not connected.");
  }

  Serial.print("server is at ");
  Serial.println(Ethernet.localIP());
  Serial.println();

 // Serial << "Connected to " << ssid << "IP address: " << WiFi.localIP() << endl;  

  broker.begin();
  Serial << "Broker ready : " << Ethernet.localIP() << " on port " << PORT << endl;

}

void loop() {
  broker.loop();
}

[hpssjellis]

Aaargh. I should be good with the WiFi part, still working on the Ethernet.

I did some stuff, got the WiFi MQTT broker working and can't completely remember what I did. LOL

I made a github fork that can be installed alongside tinyMQTT as tinyMQTTPortenta but it is not working again.

https://github.com/hpssjellis/TinyMqtt

Some points I was working with from the original Repo.

Debug can be set here

// #define TINY_MQTT_DEBUG

These lines seem to be a problem https://github.com/hsaturn/TinyMqtt/blob/90dea36ab07090c7b6aa137c1a9c80e1628127ba/src/TinyMqtt.h#L40-L43

#else 
   using TcpClient = WiFiClient;  
   using TcpServer = WiFiServer;
#endif

I changed the above to defines which seemed to help.

[hpssjellis]

update: The examples: z-portenta-wifi-mqtt-broker.ino code example in my library fork at https://github.com/hpssjellis/TinyMqtt

works randomly for one MQTT topic, crashing for the two errors already mentioned above that seem interconnected.

Exception in thread "AWT-EventQueue-0" java.util.ConcurrentModificationException
    at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
    at java.util.LinkedList$ListItr.next(LinkedList.java:888)

and

Exception in thread "Thread-258" java.util.ConcurrentModificationException
 lots more errors ...

I will research those next, while trying to figure out how to get the Ethernet connection working.

@facchinm

Anyone finding the "java" errors weird for the Arduino IDE? Is this something that might not occur if I use a different compiler?

[hpssjellis]

This is probably more a stack overflow issue

[hpssjellis]

@pennam any opinion about tinyMQTT working on the Portenta

Massimo seems to think it works, I cant get any results using v2.3.1 mbed core.

[manchoz]

Hi @hpssjellis, it should work out-of-the-box with WiFi making the changes suggested by @facchinm, but you need to install the ArduinoCore-mbed upstream from this repo.

In case you would like to experiment also with Ethernet, here is my branch of TinyMQTT with all the needed fixes in place. Look for the simple-broker-eth.ino example.

[hpssjellis]

@manchoz things are looking much better, everything compiles and my first tests show good results. Still testing.

1. Can your fork be made to work for either WiFi or Ethernet?
2. Any chance the original repo would take a PR with your changes or just have it as a separate library?

[manchoz]

Hi @hpssjellis, My plan is to PR to the upstream repo, but I think that there will be a need for a bit of discussion with the maintainer to make a cleaner design of the API to support both WiFi and Ethernet (actually, the family of the Arduino's Client class).

[hpssjellis]

@manchoz Last I talked with the maintainer he was concentrating on some other things here I have setup a test Portenta Library here which you are welcome to PR to while things are testing. Somehow we need a #define to say if you are doing wifi or Ethernet. I used the wifi library define but my code is not working here

#elif defined(ARDUINO_PORTENTA_H7_M7)
  #include <Arduino.h>
  using namespace arduino;
  #if defined(WiFi_h)
     #include <WiFiClient.h>
     #include <WiFiServer.h>
  #else
     #include <Ethernet.h>
     #include <PortentaEthernet.h>
  #endif
   // #include <WiFi.h>

#endif
#ifdef EPOXY_DUINO
  #define dbg_ptr uint64_t
#else
  #define dbg_ptr uint32_t
#endif
#include <vector>
#include <set>
#include <string>
#include "StringIndexer.h"
#include <MqttStreaming.h>

// #define TINY_MQTT_DEBUG

#ifdef TINY_MQTT_DEBUG
  #define debug(what) { Serial << __LINE__ << ' ' << what << endl; delay(100); }
#else
  #define debug(what) {}
#endif

#ifdef TCP_ASYNC
  using TcpClient = AsyncClient;
  using TcpServer = AsyncServer;
#else
    #if defined(WiFi_h)
      using TcpClient = WifiClient;
      using TcpServer = WifiServer;
    #else
      using TcpClient = EthernetClient;
      using TcpServer = EthernetServer;
   #ifdef
#endif

The multiple #ifdef are messing me up. Any suggestions for a better way? I am thinking of just making a

#define portentaWiFi

[manchoz]

@hpssjellis I think that the ultimate solution would be using C++ templates where needed or, otherwise, split the library in separate classes for Ethernet and WiFi.

[hpssjellis]

Anyone got any idea if this is live with mbed core 2.4.1 release? @pennam

[pennam]

Hi @hpssjellis, i've made a quick test using @manchoz branch and @facchinm suggestions. simple-broker sketch runs fine on portenta and i can connect to the broker using a desktop mqtt-client

[hpssjellis]

@pennam Your right, the wifi one works fine, haven't tested the Ethernet yet, but things are looking good.

Guess I can close this issue. Well done folks at Arduino.

[hpssjellis]

Having stability issues on multiple sends where the board shuts down and flashes a red LED, and a weird error that I will try to track down. see image

[hpssjellis]

@manchoz Can we retest both the WiFi MQTT server and the Ethernet MQTT server when the next version of the Arduino MBED comes out (the version after 2.4.1 ). A new version of tinyMQTT was released and things are not running as smooth for me.

[facchinm]

Hi Jeremy, I just tested the current version of the core (will become 2.5.x) , TinyMqtt 0.8.0 with the following patch applied and simple-broker example.

#if defined(ARDUINO_PORTENTA_H7_M7)
  #include <WiFi.h>
  #include <WiFiClient.h>
  #include <WiFiServer.h>
#endif

Everything seem to work fine; can you test on your own too and, in case it fails, report the exact steps to reproduce?

https://user-images.githubusercontent.com/8843437/134476746-bda417a3-8511-42e6-a7d9-00028db19974.mp4

[hpssjellis]

Looks awesome @facchinm I will run some tests after I play around with the split memory with a Machine Learning model. Great that 2.5.2 is now live.

[hpssjellis]

Sorry @facchinm I did test this using Wifi and your correct it does work. I just have not yet got my library working for both WiFi and Ethernet yet.

Thanks so much for working on this. It really is a huge deal.