Unable to receive payload greater than 65536 bytes

[shadowfighterCA]

I am trying to receive around 2 MB of payload via MQTT on esp32. But maximum packet size that I can set is 65536 via client.setBufferSize() function. The argument of this function is a uint16_t hence i am unable to set receiving packet size more than 65536. Such large payloads can be easily received by ESP-IDF. Can you help me with this ? I tried setting it up higher than 65536 but it simply does not work.

Code for reference -

#include <WiFi.h>
#include <PubSubClient.h>

// WiFi
const char *ssid = "hello"; // Enter your WiFi name
const char *password = "world";  // Enter WiFi password

// MQTT Broker
const char *mqtt_broker = "broker.emqx.io";// broker address
const char *topic = "esp32/test"; // define topic 
const char *mqtt_username = "emqx"; // username for authentication
const char *mqtt_password = "public";// password for authentication
const int mqtt_port = 1883;// port of MQTT over TCP

WiFiClient espClient;
PubSubClient client(espClient);

void setup() {
 // Set software serial baud to 115200;
 Serial.begin(115200);
 // connecting to a WiFi network
 WiFi.begin(ssid, password);
 while (WiFi.status() != WL_CONNECTED) {
     delay(500);
     Serial.println("Connecting to WiFi..");
 }
 Serial.println("Connected to the WiFi network");
 //connecting to a mqtt broker
 client.setServer(mqtt_broker, mqtt_port);
 client.setCallback(callback);
client.setBufferSize(65536);
 while (!client.connected()) {
     String client_id = "esp32-client-";
     client_id += String(WiFi.macAddress());
     Serial.printf("The client %s connects to the public mqtt broker\n", client_id.c_str());
     if (client.connect(client_id.c_str(), mqtt_username, mqtt_password)) {
         Serial.println("Public emqx mqtt broker connected");
     } else {
         Serial.print("failed with state ");
         Serial.print(client.state());
         delay(2000);
     }
 }
 // publish and subscribe
 client.publish(topic, "Hi EMQX I'm ESP32 ^^");
 client.subscribe(topic);
}

void callback(char *topic, byte *payload, unsigned int length) {
 Serial.print("Message arrived in topic: ");
 Serial.println(topic);
 Serial.print("Message:");
 for (int i = 0; i < length; i++) {
     Serial.print((char) payload[i]);
 }
 Serial.println();
 Serial.println("-----------------------");
}

void loop() {
 client.loop();
}

[superbool]

You can split the packets to small ones then send one by one . In you esp32 when received try to merge the packets.

https://mqtt.org "It is designed as an extremely lightweight publish/subscribe messaging transport that is ideal for connecting remote devices with a small code footprint and minimal network bandwidth", it's not suit to transport large data.

[shadowfighterCA]

hey, thanks for response. As per mqtt specs maximum payload can be 256 MB. And I have received more than 10 MB payload on the same hardware (ESP32) using native framework (ESP IDF). So as far as hardware and protocol goes, it is possible to receive more than 10 MB payload. I was asking if someone has modified this library for ESP32. Arduino framework uses esp idf in backend so doing this is possible.

[HamzaHajeir]

Hi

Please feel free to checkout my maintained fork of H4AsyncMQTT. Just keep in mind the availability of RAM to handle continuous TCP fragments.

[danbachar]

I'm struggling with a similar problem, but for me it's sending. I can't send anything higher than the above specified payload limit, even after increasing the payload size appropriately. @HamzaHajeir is your library also relevant for this use case?

[HamzaHajeir]

I'm struggling with a similar problem, but for me it's sending. I can't send anything higher than the above specified payload limit, even after increasing the payload size appropriately. @HamzaHajeir is your library also relevant for this use case?

Hi @danbachar, yes, as there's no limit to the incoming payload size, starting from reassembling TCP fragments from LwIP upwards. The library accepts any reasonable payload size (~Max free block / 2).

[danbachar]

I mean to send a ~130 kb payload from the esp32, not receive

[HamzaHajeir]

Likewise, there need to be a free block of a bit more than two times the payload size. Therefore something like 260kb of a block size (the mqtt packet needs to be saved in a contiguous block of memory).

What free heap/max block you have in your system?

On Sat, Nov 30, 2024, 23:22 Dan Bachar @.***> wrote:

I mean to send a ~130 kb payload from the esp32, not receive

— Reply to this email directly, view it on GitHub https://github.com/knolleary/pubsubclient/issues/1017#issuecomment-2509202831, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH3O7J4S4DKGRBMYTDAURO32DINCDAVCNFSM6AAAAABSYXEHBOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMBZGIYDEOBTGE . You are receiving this because you were mentioned.Message ID: @.***>