Reconnect instead of messageReceived() callback with The Things Network (TTN)

[matthias-bs]

Hi,

I encounter a very strange behaviour in a sketch based on ESP32DevelopmentBoard.ino.

Actually, I want to receive a topic from the TTN MQTT broker. See https://www.thethingsindustries.com/docs/integrations/mqtt/ for details; it uses MQTT Standard Version 3.1.1 (only QoS 0). I can see with two other MQTT clients (MQTT Explorer on Linux and IoT MQTT Panel Pro on Android) when the messages are sent.

Instead of receiving the messages in the messageReceived() callback, connect() is invoked at the same time the message is expected (according to the other clients).

When I change the MQTT broker (to my local Mosquitto) and the according subscription, the messageReceived() callback is invoked as expected. (the switching is done with the defines TTN and FLORA from credentials.h in the code below)

So the code fails with the first MQTT broker/topic setup and works as intended with the second one.

If have tried the following without observing any difference:

• commented/uncomment the cyclic message publishing (which is actually not needed; the client is only needed for incoming topics)
• changed the timing with setOptions() from default to the same settings I am using with the Android clients
• copied the topic from the MQTT explorer and double-checked it
• changed subscribe topic from v3/flora-lora@ttn/devices/eui-9876b6000011c87b/up to v3/+/devices/eui-9876b6000011c87b/up (in case there is an issue with the length or the @-character)
• changed Arduino IDE preferences to emit all compiler warnings - there are none
• compiled with two different partition schemes
• changed subscribe topic to #
• checked the code on ESP8266 (Wemos/Lolin D1 Mini) - same result

Any ideas how to track this down?

My setup is

• Arduino 1.8.19
• ESP32 Board Support Package 2.0.0
• arduino-mqtt v2.5.0
• ESP32-WROOM-32 (WaveShare e-Paper ESP32 Driver Board)

Please find my sketch below:

#include "credentials.h"
#include <WiFi.h>
#include <MQTT.h>

#define MQTT_PAYLOAD_SIZE 512
#ifdef FLORA
#define MQTT_SUB_IN "ESPWeather-267B81/data/WeatherSensor"
#endif
#ifdef TTN
#define MQTT_SUB_IN "v3/flora-lora@ttn/devices/eui-9876b6000011c87b/up"
#endif

WiFiClient  net;
MQTTClient  client(MQTT_PAYLOAD_SIZE);
char        MqttBuf[MQTT_PAYLOAD_SIZE+1];
bool        mqttMessageReceived = false;

unsigned long lastMillis = 0;

void connect() {
  Serial.print("checking wifi...");
  while (WiFi.status() != WL_CONNECTED) {
    Serial.print(".");
    delay(1000);
  }

  Serial.printf("\nconnecting to %s:%d...", MQTT_HOST, MQTT_PORT);
  while (!client.connect(Hostname, MQTT_USER, MQTT_PASS)) {
    Serial.print(".");
    delay(1000);
  }

  Serial.println("\nconnected!");

  if (client.subscribe(MQTT_SUB_IN)) {
    Serial.println("\nsubscription o.k.!");
  } else {
    Serial.println("\nsubscription failed!");    
  }
  // client.unsubscribe("/hello");
}

void messageReceived(String &topic, String &payload) {
  Serial.println("incoming: " + topic + " - " + payload);

  // Note: Do not use the client in the callback to publish, subscribe or
  // unsubscribe as it may cause deadlocks when other things arrive while
  // sending and receiving acknowledgments. Instead, change a global variable,
  // or push to a queue and handle it in the loop after calling `client.loop()`.

  mqttMessageReceived = true;
}

void setup() {
  Serial.begin(115200);
  WiFi.begin(ssid, password);

  // Note: Local domain names (e.g. "Computer.local" on OSX) are not supported
  // by Arduino. You need to set the IP address directly.
  client.begin(MQTT_HOST, MQTT_PORT, net);
  client.setOptions(60 /* keepAlive [s] */, true /* cleanSession */, 30000 /* timeout [ms] */);
  client.onMessage(messageReceived);

  connect();
}

void loop() {
  client.loop();
  delay(10);  // <- fixes some issues with WiFi stability

  if (!client.connected()) {
    connect();
  }
#if 0
  // publish a message roughly every second.
  if (millis() - lastMillis > 1000) {
    lastMillis = millis();
    client.publish("/hello", "world");
  }
#endif
}

credentials.h:

const char* ssid          = "SSID"; 
const char* password = "PASSWD";

//#define TTN
#define FLORA

#ifdef FLORA
const int MQTT_PORT = 1883;
const char *MQTT_HOST = "192.168.0.12";
const char *MQTT_USER = "mosquitto";
const char *MQTT_PASS = "XXX";
#endif

#ifdef TTN
const int MQTT_PORT = 1883;
const char *MQTT_HOST = "eu1.cloud.thethings.network";
const char *MQTT_USER = "flora-lora@ttn";
const char *MQTT_PASS = "XXX";
#endif

const char *Hostname = "Wetterstation";

[matthias-bs]

One thing which is 'special' or at least different in the TTN case: The username has 15 characters and the password has 98 characters. However, connecting seems to work and the buffer size of 512 bytes should be sufficient, right?

[matthias-bs]

Problem solved! After increasing the buffer size (MQTT_PAYLOAD_SIZE) to 4096, it worked!

I found the hint here: https://www.thethingsnetwork.org/forum/t/ttn-v3-esp32-mqtt-message-dont-arrive/47509/10

The reason is, that not only the payload, but a whole bunch of metadata is sent in the uplink message...

So the problem was not really related to arduino-mqtt. However, I wonder if there were a way to indicate a buffer overrun!?