[BUG] disconnected from MQTT: 7

[hmueller01]

esp8266, TLS, Platformio, latest Arduino 8266 git version (3.2.0-dev), SDK 3.0.5

MQTT broker: mosquitto 2.0.15

Debugging activated by -D DEBUG_ESP_PORT=Serial -D DEBUG_ESP_MQTT_CLIENT

PubSubClient works with the same config (broker, user, pass, fingerprint) ...

mqttSetup: MQTT server mqtt:8883
mqttSetup: MQTT client id: RHTS-0015A582
mqttSetup: HomA system id: 15a582-rhts
Alloc (l:66)
state 0 --> 1
mqttConnect: connected using espMqttClient
state 1 --> 2
state 2 --> 3
tx 66/66 (10)
state 3 --> 7
state 7 --> 8
clearing queue (clear session: 0)
state 8 --> 0
onMqttDisconnect: disconnected from MQTT: 7
Alloc (l:66)
state 0 --> 1
mqttConnect: connected using espMqttClient
state 1 --> 2
state 2 --> 3
tx 66/66 (10)
state 3 --> 7
state 7 --> 8
clearing queue (clear session: 0)
state 8 --> 0
onMqttDisconnect: disconnected from MQTT: 7

It never comes to onMqttConnect and cycles endless ...

Code

#include <espMqttClient.h>  // https://github.com/bertmelis/espMqttClient
espMqttClientSecure m_mqtt_client;

void mqttSetup() {
  INFOF_P("%s: MQTT server %s:%s" LF, __func__, mqtt_host, mqtt_port);
  INFOF_P("%s: MQTT client id: %s" LF, __func__, host_name);

  SAFE_FREE(m_homa_id);
  m_homa_id = (char *)malloc(8 + strlen_P(PSTR(_HOMA_SYSTEM_ID)));
  sprintf_P(m_homa_id, PSTR(_HOMA_SYSTEM_ID), ESP.getChipId());
  INFOF_P("%s: HomA system id: %s" LF, __func__, m_homa_id);

  char topic[100];
  sprintf_P(topic, PSTR("/devices/%s/controls/State"), m_homa_id);
  m_mqtt_client.setClientId(host_name)
      .setServer(mqtt_host, atoi(mqtt_port))
      .setCredentials(mqtt_user, mqtt_pass)
      .setFingerprint(mqtt_fingerprint)
      .setWill(topic, QOS1, true, "offline")
      .setCleanSession(true)
      .onConnect(onMqttConnect)
      .onDisconnect(onMqttDisconnect)
      .onMessage(onMqttMessage);
  m_mqtt_reconnect = true;
  m_mqtt_reconnect_time = 0;
}

void mqttConnect() {
  if (m_mqtt_reconnect) {
    // client not connected, try to connect
    long now = millis();
    if (now - m_mqtt_reconnect_time > 5000) {
      m_mqtt_reconnect_time = now;
      // attempt to connect
      if (m_mqtt_client.connect()) {
        INFOF_P("%s: connected using espMqttClient" LF, __func__);
        m_mqtt_reconnect = false;
      } else {
        ERRORF_P("%s: Error: Connecting using espMqttClient failed" LF, __func__);
      }
    }
  }
}

void onMqttConnect(bool sessionPresent) {
  char topic[100];

  INFOF_P("%s: connected using espMqttClient" LF, __func__);

  sprintf_P(topic, PSTR("/sys/%s/#"), m_homa_id);
  m_mqtt_client.subscribe(topic, QOS1);
  sprintf_P(topic, PSTR("/devices/%s/meta/#"), m_homa_id);
  m_mqtt_client.subscribe(topic, QOS1);
  sprintf_P(topic, MQTT_HKP_TOPIC, m_homa_id);
  m_mqtt_client.subscribe(topic, QOS1);
}

void onMqttDisconnect(espMqttClientTypes::DisconnectReason reason) {
  INFOF_P("%s: disconnected from MQTT: %u" LF, __func__, static_cast<uint8_t>(reason));

  m_mqtt_reconnect = true;
  m_mqtt_reconnect_time = millis();
}

void loop() {
  m_wifi_manager.run();

  if (!m_wifi_manager.isConfigMode()) {
    mqttConnect();
    m_mqtt_client.loop();
  }
}

Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif8266/esp12e.html
PLATFORM: Espressif 8266 (4.2.1) > Espressif ESP8266 ESP-12E
HARDWARE: ESP8266 80MHz, 80KB RAM, 4MB Flash
PACKAGES: 
 - framework-arduinoespressif8266 @ 3.2.0-dev+sha.de1029f 
 - tool-esptool @ 1.413.0 (4.13) 
 - tool-esptoolpy @ 1.30000.201119 (3.0.0) 
 - toolchain-xtensa @ 2.100300.220621 (10.3.0)
LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ deep+, Compatibility ~ soft
Found 51 compatible libraries
Scanning dependencies...
Dependency Graph
|-- arduino-sht @ 1.2.3
|-- ArduinoJson @ 6.21.3
|-- StreamUtils @ 1.7.3
|-- espMqttClient @ 1.5.0+sha.b270cc6
|-- mr-common @ 1.0.0+sha.b2d7ae3
|-- ESP_WiFiManager_Lite @ 1.11.0-wip+sha.36d3c02
|-- ArduinoOTA @ 1.0
|-- ESP8266WiFi @ 1.0
|-- ESP8266httpUpdate @ 1.3
|-- ESP8266HTTPClient @ 1.2
|-- LittleFS @ 0.1.0
|-- Wire @ 1.0
|-- EEPROM @ 1.0
|-- Ticker @ 1.0
|-- ESP8266WebServer @ 1.0
|-- DNSServer @ 1.1.1
|-- ESP_MultiResetDetector @ 1.3.2
|-- common

What can I do?

[bertmelis]

Do you have access to the broker logs?

[hmueller01]

I removed the

//.setWill(topic, QOS1, true, "offline")

setWill, which makes a difference:

mqttSetup: MQTT server mqtt:8883
mqttSetup: MQTT client id: RHTS-0015A582
mqttSetup: HomA system id: 15a582-rhts
Alloc (l:46)
state 0 --> 1
mqttConnect: connected using espMqttClient
state 1 --> 2
state 2 --> 3
tx 46/46 (10)
rx len 4
Packet type: 0x20
Remaining length: 2
Packet complete
state 3 --> 4
clearing queue (clear session: 1)
onMqttConnect: connected using espMqttClient
Packet buffer not allocated: low memory
Could not create SUBSCRIBE packet
Packet buffer not allocated: low memory
Could not create SUBSCRIBE packet
Packet buffer not allocated: low memory
Could not create SUBSCRIBE packet
Packet buffer not allocated: low memory
Could not create PUBLISH packet
Packet buffer not allocated: low memory
Could not create PUBLISH packet
Packet buffer not allocated: low memory
Could not create PUBLISH packet
Packet buffer not allocated: low memory
Could not create PUBLISH packet
Packet buffer not allocated: low memory
Could not create PUBLISH packet
Packet buffer not allocated: low memory
...

In the broker log I see

2024-02-03T22:28:07: New connection from 192.168.2.64:62176 on port 8883.
2024-02-03T22:28:08: New client connected from 192.168.2.64:62176 as RHTS-0015A582 (p2, c1, k15, u'xxx').
2024-02-03T22:28:32: Client RHTS-0015A582 has exceeded timeout, disconnecting.

[bertmelis]

espMqttClient is a tad more memory hungry than pubsubclient. As you can see you are running out of memory so the publish packets cannot be generated. Mind that TLS on esp8266 is already a stretch. With all the features you have in your program (based on the dependencies), it might be too much.

You can try to lower the free memory requirement in the config file or set it via a define in platformio.ini.

[hmueller01]

Ok, I set -D EMC_MIN_FREE_MEMORY=1024 and now it works with .setWill(topic, QOS1, true, "offline") as well.

But memory fragmentation is really high:

getFreeHeap: 10608, getHeapFragmentation: 28, getMaxFreeBlockSize: 6944

compared to the PubSubClient variant

getFreeHeap: 11920, getHeapFragmentation: 8, getMaxFreeBlockSize: 10992
getFreeHeap: 12376, getHeapFragmentation: 10, getMaxFreeBlockSize: 11136
getFreeHeap: 12376, getHeapFragmentation: 9, getMaxFreeBlockSize: 11264

@bertmelis Thanks for your quick responses and the help you gave me!

[bertmelis]

Memory usage and fragmentation are indeed higher. Added features don't come for free...

I'm thinking about a crude memory pool but if you're already running low on memory it might just not work at all anymore. It's not on the top of my priority list.

[hmueller01]

My major issue about PubSubClient wasn't the features (ok, yes. QOS support needs to be improved). It's just abandoned by @knolleary. No needed PRs are processed any more. I was hoping the guys from thingsboard would take over, but it doesn't look like that. And I was hoping to find a way more light TLS client in your lib. But you are also just using the monster BearSSL ... so I doubt to get happy with the memory consumption using this lib. But thanks again for answering my problems here so fast!

[proddy]

and the new kid https://github.com/theelims/PsychicMqttClient

[hmueller01]

and the new kid https://github.com/theelims/PsychicMqttClient

Tx, but only ESP32

architectures=esp32

I need something for ESP8266 ... But maybe I'll give https://github.com/256dpi/arduino-mqtt a trail ... (@bertmelis sorry for waisting your issue with these kind of information, I'll stop here)

[bertmelis]

It's not a problem at all. Memory usage is a topic for this lib. It is for all libs that try to be compliant.

Pre-allocating memory helps for fragmentation but it uses a chunk of memory that isn't available for other processes anymore. Not a problem if you have enough memory left but in that case, fragmentation also isn't a real issue.

[bertmelis]

and the new kid https://github.com/theelims/PsychicMqttClient

Interesting but not really 'new'. It's a wrapper around the MQTT client from ESP-IDF. The ESP-IDF MQTT client has the same memory issues as my lib: packets are stored in heap memory --> https://github.com/espressif/esp-mqtt/blob/master/lib/mqtt_outbox.c#L44