Mosquitto Broker working is slow

[Meghadoot]

I'm currently using Mosquitto Broker which is installed on raspberry pi.

Environment

Mosquitto Broker: 1.3.4. Hardware: Raspberry pi 3B No TLS

Configuration file "mosquitto.conf"

pid_file /var/run/mosquitto.pid

persistence true
persistence_location /var/lib/mosquitto/
log_dest file /var/log/mosquitto/mosquitto.log

# log_type error
# log_type warning
# log_type notice
# log_type information
log_type debug

log_timestamp true
include_dir /etc/mosquitto/conf.d

Implementation Details

I have a distributed system with multiple controllers, with overall 50 sensors and 10 relays (actuators). Every controller is sending sensor reading periodically(every 1 sec) to raspberry-pi on CAN bus. As soon as sensor reading (float value) is received by Raspberry-pi, Client application in C (written using paho-mqtt-c libraries) is responsible for publishing a MQTT message with the topic related to particular sensor (with QoS "1" to gain the advantage of persistent session and faster message delivery than QoS "2"). Topic length is around 70 characters and payload is of 5 bytes size.

Telegraf (MQTT Metric collector) is installed on raspberry-pi itself, which acts a subscriber. It subscribes to all MQTT topics mentioned above and finally fetches the data into InfluxDB (which is located on remote server).

MQTT Publisher (on Raspberry-PI) <--> Mosquitto-Broker (on Raspberry-PI) <--> Telegraf/subscriber(on Raspberry-PI)

Scenario 1 In above scenario, Published messages are received late (~3 to 4 minutes of delay) by telegraf and hence, seen late in database. It is also tested with simple subscriber (CLI- mosquitto_sub) to ensure delay doesn't possess with Telegraf plugin processing.

Scenario 2 If publishing period of the sensors is increased from 1 sec to 15-20 seconds. Entire operation is real-time. MQTT message published by publisher can be immediately (~0.4 seconds) seen on database.

Hence there is no any issue (neither slow publisher nor delayed reception) observed in MQTT publisher and on subscriber side, as tested with scenario 2. Hence It is observed that Mosquitto Broker processing is slow when traffic intensity is higher. Also Testing with QoS level "0" have minimal impact, as reception is still delayed with ~3 minutes and loss of frames is predominantly more.

From application point of view, it is not intended to increase sensor data publication period. So, questions are: Is there any recommended solution (or factors to be seen) to improve performance of Mosquitto Broker for fast processing of messages? Is there any performance evaluation benchmark related to latency or delay?

[ralight]

1.3.4 is the current Raspian/Debian Jessie provided version, but it is 5 years old now. 60 messages per second should be nothing even on a Pi. Are you able to upgrade to a more recent version, there have been many many performance improvements in the past five years. There is a Debian/Raspian repository provided by the project at https://repo.mosquitto.org/

[Meghadoot]

Thanks for a quick response.

I have already tried with broker version V1.5.3 installed on local system. No luck! Reception was still laggy, approximately 2-2.5 minutes delay.

But I will surely give it a shot by upgrading to latest broker on pi (>V1.6) .

Can you please provide any benchmark regarding minimum number of messages mosquitto broker can process per second for QoS 1 or 2? Also apart from version upgrade, Can you please suggest any specific factors that might be slowing down the broker?

[ralight]

I don't have any hard numbers to hand, but way beyond what you are trying to do. As an example the test.mosquitto.org server is a low spec atom processor and currently has ~3200 clients connected, is sending out 1000 messages per second (probably a minority are qos 1 though) and is not struggling at all.

What does the load look like on the pi?

[Meghadoot]

Apart from broker and 2 MQTT publisher clients, there are Nodered and telegraf applications running on pi. (Average) CPU usage is around 70%. Sometimes may hit around 100% for one or two cores.

[ralight]

Do you have any other config in /etc/mosquitto/conf.d?

[Meghadoot]

No. there is not.

[ralight]

You've checked the connection on the subscribing side using an alternative client, have you also been able to check that the publishing client is doing what it is supposed to?

A tcpdump of port 1883 on the broker machine would be very useful, if you could manage that. I can take it by email or other means and keep it private if necessary.

[Meghadoot]

At this moment, broker and publishers are on the same pi. I will have to check for permission to share tcpdump.

By that time, I will install and test with latest version of broker.

Edit: Moreover, publishers are synchronous (developed with synchronous paho lib). In other words, they remain in blocked state until receive any response. As Broker processing is slower, publishers are also working slow.

[Meghadoot]

I have also installed Mosquitto V1.6.2 on local machine to test, there isn't any improvement in situation.

I can provide you tcpdump.

I would like to know, what is typical/observed connection time required for publisher to establish connection with broker? My publisher application has similar workflow as mentioned in Paho MQTT C Client.

"Create a client -> Set up connection with Broker -> Publish a message -> Disconnect the client -> Free memory used by the client."

[ralight]

Using gettimeofday(). with a connection to a local machine (my main desktop) and instrumented mosquitto_pub -t test -m test -q qos, I get:

QoS 0: 678us, 447us, 605us QoS 1: 662us, 628us, 609us QoS 2: 663us, 633us. 988us

[Meghadoot]

Working with asynchronous MQTT APIs and eliminating disconnection phase allows faster forwarding of messages by broker. Thanks for your time!!