search

PerMalmberg / Smooth

C++ framework for embedded programming on top of Espressif's ESP-IDF.
Apache License 2.0
325 stars 31 forks source link

MQTT maximum throughput #161

Open Elfelsoufim opened 3 years ago

Elfelsoufim commented 3 years ago

Hello,

For my application, I need to send messages from an ESP32 to my main computer (which hosts a mosquitto broker) at a minimum rate of 1000 messages per second. However, when I tested Smooth MQTT throughput using the example found in Smooth/test/mqtt, I get the following result:

Transmission Rate:

5 messages per second for QoS = Exactly once. 10 to 12 messages per second for QoS = At least once. Same for QoS = At most once.

That, however, is not the expected behaviour, since:

  1. ESP32 allegedly can send about 2000 messages per second. https://www.reddit.com/r/esp8266/comments/5trsvl/esp32_wifi_real_life_throughput/ddokmz6/?utm_source=share&utm_medium=web2x&context=3.
  2. Mosquitto broker was tested to be able to handle 7000 messages per second. http://rexpie.github.io/2015/08/23/stress-testing-mosquitto.html.

For troubleshooting, I have tried several solutions. Notably:

To detect the source of the problem, I tried timing several methods such as publish(), publish_next() and send_packet(). All of these take at most a few milliseconds to execute. I also tried timing the loop inside exec(). Here I found that it sometimes takes up to 150 ms to execute the loop. Knowing that, I tried circumventing exec() by not calling start() (start() calls exec()). I did this by calling init() and then send_message() repeatedly. Please see the code provided below.

After trying this fix, the throughput did not improve at all, which leads me to think that send_message() (which really is just publish()) is guilty of this issue. But considering that publish() takes only a few milliseconds to put the data in the transmit buffer, and after having read this https://github.com/PerMalmberg/Smooth/issues/142, I conclude that there must be a problem with how messages are being sent.

What I really want to ask here is:

  1. Am I using "MqttClient.cpp" wrong?
  2. What is the expected throughput of MQTT client? Is there a benchmark?
  3. How could I increase the maximum transmission rate of "MqttClient.cpp"?

For reference, here is the environment I used:

Here is the code I used to test MQTT: GIST

PerMalmberg commented 3 years ago

Hello,

  1. I don't think you are.
  2. No, it was fast enough for my purposes, but I remember being able to send quite a lot of messages per second. Have you tried the test/example?
  3. I wish I could tell you, but I don't have an answer. It could be as simple as moving the ESP32 to another location, I remember doing that and getting much better Wifi speeds.
squonk11 commented 3 years ago

I had quite interesting speed improvement when I reduce the timeout for select from 10ms to 1ms here. Also my feeling is that the there is somewhere a delay when processing messages between tasks. If I interpret this code right then the tick() function is not called when a message arrived; tick() will then be called during next Task cycle. Here it is important which is the Task cycle time for your MQTT task. Maybe it could make sense to put a call to tick() also in the else path of if (!queue_ptr.owner_before(empty_ptr) && !empty_ptr.owner_before(queue_ptr))in Task::exec()?

PerMalmberg commented 3 years ago

It does wait for data to become available, @squonk11 https://github.com/PerMalmberg/Smooth/blob/c874fed1de71f9b8dac65e0e8c238b2020e658a3/lib/smooth/core/Task.cpp#L164

squonk11 commented 3 years ago

yes, I agree, but in case there is a message, then only the message is being processed and tick() is not being called (if I interpret the code correctly).

PerMalmberg commented 3 years ago

Correct - but tick() isn't meant to handle events (i.e. incoming messages). Tick is meant to do things that are not triggered by an event.

The Task impl. attempts to ensure that the tick is called on the desired interval. https://github.com/PerMalmberg/Smooth/blob/c874fed1de71f9b8dac65e0e8c238b2020e658a3/lib/smooth/core/Task.cpp#L148

squonk11 commented 3 years ago

ah, o.k. I understand.

squonk11 commented 3 years ago

@Elfelsoufim : The link in your post seems to point to a wrong location? But two comments from my side:

  1. The test done there is made using a simple ping/pong protocol in polling mode via UDP. That probably isn't comparable with a protocol like MQTT via TCP runing on a cyclic task.
  2. The network infrastructure you are using can also make a difference. A direct connection between PC and ESP32 will give better results as if the telegrams pass e.g. a WiFi router, a WiFi Repeater, an ethernet switch etc.
Elfelsoufim commented 3 years ago

Hello, thank you both for your timely support.

@squonk11 I have tried all the solutions you have suggested both in this issue and in https://github.com/PerMalmberg/Smooth/issues/142, without success, unfortunately. My transmission rate seems to be fixed at around 12 messages per second.

I ran a few tests to benchmark Smooth MQTT. I put my results and my conclusion in this short document: Benchmark MQTT.docx Benchmark MQTT.pdf

The tests and documentation are rather informal, but I believe they illustrate my main point: Smooth's MQTT can be optimized further 😄.

PerMalmberg commented 3 years ago

Serious investigation :)

Yes, I'm sure there are ways to optimize it. I have a faint memory of the parsing of incoming messages being a bottleneck, iirc it was read a byte at a time to determine the message size etc. I don't remember if I optimized that part or not. so that may be a starting point.

Elfelsoufim commented 3 years ago

Hello,

I have tested ESP-IDF's MQTT library using very similar code & conditions to the TCP/IP benchmark I have done previously to give us a useful comparison with Smooth MQTT. Here is the documentation: Benchmark ESP_IDF_MQTT.pdf

The result is an average of 640 messages sent per second. I am only a student so it is not something I could do myself, but this would be a nice result to achieve with Smooth MQTT.