Mqtt server memory leak version 4.2.0.706

[Mahersrasra]

Describe the bug

I have an mqtt server built using MQTTnet library 4.0.2.221, i'm sending 1 million message per second to my mqtt server and another client is subscribing to those messages. I noticed an excessive memory leak in my mqtt server, even when debugging i noticed that the objects that causing the memory leaks are objects used in mqtt net library as attached below. Another thing , even after the subscribed mqtt client finished consuming all the messages the server didn't dispose the used objects only after a long time. is there a way to reduce the server memory consumption or optimize the performance of objects disposing?

Which component is your bug related to?

• Server

Expected behavior

A reduced memory consumption , but even if it happens at least objects disposing should be quicker than the present situation.

Screenshots

This is a screenshot of some of the objects that caused memory leak in my scenario.

[chkr1011]

Please try the latest version of this library. We fixed several performance and memory relevant bugs. If the error persists please reopen this ticket.

[Mahersrasra]

I upgraded the mqttnet version to the latest one (4.2.0.706) but the same issue persists excessive memory consumption and a very long time taken by the server to dispose its objects and return to his first state. there are any actions that can be taken to improve memory consumption performance in general ?

[logicaloud]

MQTTnet may not be able to process 1 million messages per second. I suspect that messages end up in a queue somewhere and processing of the queue lags behind. As the queue grows, it may look like a memory leak.

You could try processing a smaller number of messages per second (i.e. start with 20000 and increase from there) to see if or at what point there seems to be a memory leak. If it is systematic, then there should be a memory leak at a lower message throughput and it should just take longer to be noticeable.

You may be able to speed up garbage collection by calling GC.Collect on occasions.

For further investigation, it would be helpful, if you could provide a minimal client / server example that produces the problem.

[Mahersrasra]

that's what i did yes, i reduced the rate of messages sent my the clients it was the same situation In my case i have to send total of one million message , i played with the rate/s of sent messages , but eventually it didn't help much . Could you please investigate the reason, why it takes so much for the server to return to his first state of memory even if there a are no more messages sent and the subscribed client consumed all the messages , i don't know is it related to objects disposing or what , this is the most disturbing thing , i get memory can increase in the process of pub/sub but what i don't get why it takes so much time after the process ends for the server to return to his first state. Thank you in advance

[chkr1011]

I am wondering why you assume that the leak is caused by this library. The screenshot shows that there are lots of Hashtables which is not a class defined or used in this library. I also see a component named "NetMQ". Is it possible to see where all this Hashtables belong to in the object graph?

[Mahersrasra]

• so to avoid any misinterpretation i created a sample mqtt server using your sample you provide in your source code

below is the code .

• you will notice that i raised the max pending messages to 1 million because i faced data loss when sending 1 million message after some researches it turned to be that max pending messages was the reason for the data loss issue and feel free to correct me if i'm wrong , i received the totality of messages as it is shown below after that .

• but in the mean time i faced the same memory leak issue running this sample as it shown in the figure below

• memory consumption reached nearly 1 GB, and when sending 1 million messages multiple times memory usage reaches 2 even 3 GB that's coming i think from the slow objects disposing time .

• Also noticed also high cpu usage sometimes it reached 50% of the totaly of my cpu , this could crash a windows session at anytime.

• anyways the snapshot details of memory usage is below :

• Maybe you can advise me on how to configure the server or you have an explanation for this memory consumption

thank you for being so responsive really appreciate it !

[logicaloud]

Thank you for the info. The screenshots don't seem to indicate a memory leak. The chart shows an increase 910 MB and then memory usage drops back to zero. Whether that is reasonable also depends on your typical application message length (I currently don't know what the memory overhead of pending messages is).

The Managed Memory snapshot shows a total memory usage of less than 1 MB (unit is bytes).

Garbage collection is best left to dotnet, and dotnet may be lazy collecting memory if there is no memory pressure, i.e. if you have lots of memory on your machine. Hanging on to disposed objects in itself is not a memory leak. You could try and use GC.Collect in your code after receiving 1 million messages to see what effect it has.

I'd be happy to look into this further if you could post a minimal example in text form, including client and server code, that demonstrates the issue. That would help clarifying whether this is just lazy garbage collection by dotnet or if there is indeed a memory leak triggered by your code.

[Mahersrasra]

Okay got you , the server code is what attached above and the sending clients you can use Hive mq swarm , follow this link for more info https://docs.hivemq.com/swarm/4.14/swarm/scenarios.html#scenario

• Concerning Garbage collection ,I tried GC.Collect after every received 1Million message and it solved the problem of memory hanging at high level after consuming the messages also objects disposing gets longer each time I send 1 Million message . That's a work around until we know for sure what caused slow objects disposing.

• Another thing i want to point out if you could clarify its effects , at first when I run the server with the default value of max pending messages i didn't face memory increase at all it was stable but as I mentioned I had data loss , so I had to increase the max pending messages to 1 Million and then i had this memory increase , could you explain its effect maybe its the reason for this excessive memory increase.

• Worth mentioning I tried the same scenario (same sending clients , same messages same everything) with other brokers and I didn't face any data loss or memory increase.

thank you again

[chkr1011]

The pending messages is the count of messages which are being caches in memory for each session. This means a session can get new messages if the client is not connected. It will receive the messages later on. This is a feature of MQTT. But for the time the clients are offline or even not able to consume the messages due to low network performance the memory footprint of the server will grow and grow. That's why you need to increase the limit. You can disable this via setting persistent sessions in the server options to false. This also might be the case for other servers you used.

[Mahersrasra]

• even with disabling persistent sessions I encountered the same behavior (excessive memory and CPU increase) as you can notice below .