eclipse-mosquitto / mosquitto

Eclipse Mosquitto - An open source MQTT broker
https://mosquitto.org
Other
9.05k stars 2.39k forks source link

Mosquitto memory consumption reaching 100% #2455

Open bhavinjethra opened 2 years ago

bhavinjethra commented 2 years ago

Hi guys, I have a situation of mosquitto service's memory consumption reaching 100% and crashing while using mosquitto 2.0.14 on Ubuntu 18.04.

Setup: A single Paho Mqtt publisher 1.5.0 that publishes 300 messages per second to the local mosquitto broker with the QoS level 2. The local broker then forwards the data to an external broker. Persistent is set to true and cleansession is set to false.

Scenario: There is 6 hours of forced network outage between the local mosquitto broker and the external mosquitto broker. So the local broker dumps the data in mosquitto.db every 120 seconds (manually set in mosquitto.conf). the file size of mosquitto.db grows as expected. However, while monitoring the memory consumption after disconnecting the external broker, it was observed that the memory usage grows from 0 to 100% and crashes eventually. The service attempts a restart however it goes into a forever crash cycle.

Local Mosquitto Broker config file: image

Subscriber stats: (Look at the growing heap size. The memory consumption is at 35% currently, and will soon get to 100%.) image

Let me know if I can help with any more details.

Chunosov commented 2 years ago

while monitoring the memory consumption after disconnecting the external broker, it was observed that the memory usage grows from 0 to 100% and crashes eventually

Probably you have to add memory limits as

# max 1GB queued messages
max_queued_bytes 1073741824

# or if you have fixed-size messages
max_queued_messages 1000

For example, having such settings I see this memory consumption in my local broker, when the external one gets disconnected: 20220405_mem_consumption_edge_with_bytes_limit

The service attempts a restart however it goes into a forever crash cycle

I think it's because mosquitto.db is getting so large that it can't not load it properly