search

espressif / esp-mqtt

ESP32 mqtt component
Apache License 2.0
603 stars 255 forks source link

Sending large data over MQTT (IDFGH-1601) #123

Closed davctv closed 5 years ago

davctv commented 5 years ago

Hi everybody, I post this issue related to this topic in esp32.com forum

https://esp32.com/viewtopic.php?f=2&t=11718

I have to send multiple files (about 1000) content (for a total of about 1 MB or more) via MQTT to a remote broker.

Since ESP32-WROOM-32 has limited RAM I can't read all files and put in a buffer before publish it.

Is there any way to perform a "chuncked/segmented" publishing with esp-mqtt library? It seems that

esp_mqtt_client_publish

need to know the length of the message and the pointer to content, but in my case I can't get the total length and I would like to open the files one by one and send the data during the publish connection. I'm testing on esp-mqtt dc37d3a commit

Thanks

david-cermak commented 5 years ago

Hi @davctv Yes it's supported by default, if the message is longer then configured out-buffer then the library will publish sequentially, but the total length must be known in before transmitting.

You can easily test this by supplying a flash ptr and a long (~1MB) length to esp_mqtt_client_publish. The commit you're referencing supports this mode.

davctv commented 5 years ago

Hello @david-cermak,

Thanks for reply. However this is not a good a news for our project, because we have saved on flash many files (up to 5000) and each of them could be 1Kbyte. Therefore, even if I retrieve the total length of the files, I'm not able to store in RAM all the contents (total of about 5MB)

In your knowledge, isn't there a method to publish with "unknown length"? i.e. client connect to remote broker and start sending data without specifying the legth of the message?

Or we will have to manage in an "application level", i.e. creating a "protocol" to inform the remote server that a single publish is a part of a bigger one.

Thanks and regards.

david-cermak commented 5 years ago

No, it is not possible to publish "unknown length", MQTT packet header contains message length information.

If the files are stored in flash, you can just pass the pointer to flash, no need to copy to RAM; but if the data are variable (i.e. retrieved from some other communication protocol or so), there's no easy way to stream it as one MQTT message.

davctv commented 5 years ago

Thanks @david-cermak,

unfortunately the latter case is mine..I have different files (in different partition on flash), so we are going to study a method to pass data in different publish.

I close the issue. Thanks for support.