Support for bigger ESP-NOW messages (IDFGH-7935)

[AmirHmZz]

I think it would be better to implement something like multipart message in esp-now. 250 bytes might be small for some purposes. Each message can be split into multiple ESPNow messages + one part number. These messages can be sent and received as byte streams.

[MaxwellAlan]

Hi @AmirHmZz

Thanks for your feature request.

Could you provide more detail for this feature that you may used in what kind of scenarios ? And how about the Throughput/Latency/Power/Security etc that required?

[AmirHmZz]

@MaxwellAlan Thanks for your quick replay! To be honest, I don't know where that 250 bytes message length limitation comes from exactly, but :

• If it's just a high level software layer limitation: I think it would be better to make ESP-NOW able to transfer bigger messages at a specific cost ( lower throughput / higher latency / more power usage / etc. ).
• If it's hardware layer or low level protocol limitation which prevents the maximum length to be increased: Maximum message length can still remain 250 bytes but it would better to have something like multipart messages in esp-idf level.

In my own company I had to implement these multipart messages in many projects, I think If this feature gets added to esp-idf developers can take advantage of it in many scenarios :

1. Sending OTA updates over ESP-NOW
2. Transferring configuration-like messages ( which is more than 250 bytes in my projects )
3. Transferring bigger messages in lower rate

ESP-NOW is really a fantastic masterpiece but it's simplicity also make it difficult to use it in some scenarios. I should also mention that I know what policy is leading ESP-NOW to this simplicity, but I think it's not bad to also develop some features on top of ESP-NOW to facilitate non professional IOT developers work.

[MaxwellAlan]

Hi @AmirHmZz

Have recheck the code and 802.11 spec, the limitation is that we use Vendor Specific element field of Action frame to transmit data, which length field is only 1 byte(0xff = 255).

[AmirHmZz]

@MaxwellAlan Thanks a lot for your quick reply. Just for my curiosity, I've got four questions:

1. Does ESP-NOW force messages to have a fixed size? I mean if it's possible, can ESP-NOW protocol let messgaes to have desired lengthes?

2. Does message size effect on other parameters like latency, power consumption, etc?

3. How large ESP-NOW messages can get?

4. Can ESP-NOW gets data of big messages as streaming buffers? I mean for a message with length of 128KB ( like reading it from a file or another stream ) we cannot have all data in RAM. Also in the receiver side, message cannot be received in one piece and should be streamed or saved somewhere.

[AmirHmZz]

@MaxwellAlan based on your edit, we cannot increase size of ESP-NOW messages, What's your opinion about implementing multipart messages?

[AmirHmZz]

I mean esp-idf have developed lots of features like MESH networks on top of WiFi but when it comes to esp-now we have limited features...

[MaxwellAlan]

@AmirHmZz

Since there are some reserved field in 802.11 Managment Frame, maybe we can try with these fields, but there maybe some confilcts when the new 802.11 spec update. Or could you try with esp_wifi_80211_tx this API to send and sniffer mode to receive, both could work only base on WiFi stack and without TCP/IP stack.

[ceribus]

ESP-NOW uses vendor-specific action frames. These contain 1 byte Category, 3 bytes Organization Identifier and Vendor Specific Content.

According to the 802.11 standard the Vendor Specific Content could contain multiple Vendor-Specific Fields. 802.11-2020, 9.6.5 Vendor-specific action details: "...The Vendor Specific Content field contains vendor-specific field(s)." But right now ESP-NOW is only using one Vendor-Specific IE, which is limited to 255 bytes.

@MaxwellAlan Am I missing something or would it be possible to split the data into multiple IE?

[AmirHmZz]

@ceribu accroding to 802.11, how many Vendor Specific IE could be present in a vendor specific content?

[ceribus]

@AmirHmZz "The length of the vendor specific content in a Vendor Specific frame is limited by the maximum allowed MMPDU size." Seems to be limited by MMPDU size only, which is 2304 bytes max for b/g/n.

[AmirHmZz]

@ceribu So theoretically, the maximum length of ESP-NOW messages cannot get bigger than 2304 bytes, am I right ?

[ceribus]

@AmirHmZz Excluding headers, it could be about 2200 bytes.

[AmirHmZz]

Any updates on this?

[zhangyanjiaoesp]

@AmirHmZz We will consider increasing the length of ESP-NOW by adding more Vendor-Specific IE. But currently we have higher priority work to do, so this feature is delayed. Have you tried to use the esp_wifi_80211_tx API ?

[AmirHmZz]

What is exactly esp_wifi_80211_tx API?

[zhangyanjiaoesp]

@AmirHmZz You can refer to the doc about esp_wifi_80211_tx: https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/network/esp_wifi.html?highlight=80211_tx#_CPPv417esp_wifi_80211_tx16wifi_interface_tPKvib

[ceribus]

@zhangyanjiaoesp I was just wondering why Vendor Specific IEs are used for ESP-NOW? Since they are not explicitly required by the 802.11 standard, any custom type of content could be used for the Vendor Specific Content field of the Vendor Specific Action Frame.

[derkalaender]

I'm by no means an expert, but this is also how I understood the 802.11 standard. The action frame body requires the action field, which in turn for the vendor-specific action frame is defined as Category, Organization Identifier, and then Vendor Specific Content. "The Vendor Specific Content field contains vendor-specific field(s)". However, these vendor-specific fields are not defined anywhere, so I assume it can be any custom data.

I think the vendor-specific element is used by mistake. The action frame body, after the action field, also specifies that "one or more vendor-specific elements are optionally present", however, only if the category is not vendor-specific [...] which it is in this case!

And I think you are aware of this, because the "Random Values" field you use is not defined as part of the action frame. So I think that already constitutes "vendor specific fields", otherwise ESP-NOW would not be follow the standard.

[zhangyanjiaoesp]

@ceribus @derkalaender You are right, any custom type of content could be used for the Vendor Specific Content field of the Vendor Specific Action Frame. And the ESP-NOW was designed to using Vendor Specific IEs.

When changing code to support for bigger ESP-NOW messages, we should consider the compatibility with the original ESP-NOW product. Adding more Vendor Specific IEs is one way, and using a new type of content is another way, we will try to find a suitable way to ensure compatibility and the needs of customers.

[nicklasb]

Just found this thread, I have several use cases where I occasionally want to send a, say 30-100 kbytes low-res image without the overhead and power consumption of a full wifi session. There are lots of other cases. If the max length could be for example 1.5 Kb, (which seems possible according to the above api), that would probably make for a much more efficient transfer of data than 250 bytes. Also..I mean we are in the to 2.4 GHz band among the gamerz and netflixers, do we really need to be this shy? :-) I would still have to implement multi-part data transfers, but I feel it would be a little bit less problematic get it performant. Ideally, I would want ESP-NOW to handle that behind the scenes.

[zhangyanjiaoesp]

@nicklasb We will consider your scenario when modifying the code.

[nicklasb]

@nicklasb We will consider your scenario when modifying the code.

That's encouraging. :-)

[AmirHmZz]

Any updates or milestones about this feature request?

[zhangyanjiaoesp]

@AmirHmZz We will solve this problem in a high priority.

[zhangyanjiaoesp]

@AmirHmZz which IDF branch do you use? which chip do you use ? I have modify the code, and I can provide a wifi lib for you to test.

[AmirHmZz]

@zhangyanjiaoesp Hello again. I use esp-idf v5.1.2 on ESP32 chip. Thank you.

[zhangyanjiaoesp]

@AmirHmZz Here is the wifi lib: wifi_lib_esp32_v5.1.2.zip wifi firmware version: 6979f8f Currently, the max length is 1600, and the new ESP-NOW version which can support longer length is 2.0, the old ESP-NOW version is 1.0, when you are using ESP-NOW to send/receive, the devices should use the same version.

[AmirHmZz]

Thank you. Do I need to change ESP_NOW_MAX_DATA_LEN? https://github.com/espressif/esp-idf/blob/003f3bb5dc7c8af8b71926b7a0118cfc503cab11/components/esp_wifi/include/esp_now.h#L52

[zhangyanjiaoesp]

Thank you. Do I need to change ESP_NOW_MAX_DATA_LEN?

https://github.com/espressif/esp-idf/blob/003f3bb5dc7c8af8b71926b7a0118cfc503cab11/components/esp_wifi/include/esp_now.h#L52

Yes, change it to 1600.