Unknown memory leak

[PontusHolmberg]

Describe the bug

Memory usage runaway.

Which component is your bug related to?

Server

To Reproduce

Steps to reproduce the behavior:

1. Using version v4.3.0.858
2. Set up a simple server application with the following events ValidatingConnectionAsync - Checks clients against database InterceptingPublishAsync - Just to increase a counter to keep stats on number of msg handled LoadingRetainedMessageAsync - Uses code from samples RetainedMessageChangedAsync - Uses code from samples
3. The server has the following options:

   var options = new MqttServerOptionsBuilder().
               WithPersistentSessions(true).
               WithConnectionBacklog(2000).
               WithDefaultEndpointPort(1883).
               WithDefaultEndpoint().
               WithKeepAlive().
               WithMaxPendingMessagesPerClient(20000);

4. See error.

Expected behavior

The memory should vary with the number of connected clients and number of persistent messages in memory.

Screenshots

Additional context / logging

The server also has a timer, that once a sec updates the GUI with info about: connected clients, numb of processed msg and msg/sec. Before I also displayed all connected clients in a ListView (WPF). Has worked without issue, but that was before everything was made asynchronous.

I made a client-app that spams the server with both connections and messages that should be persistent to try to find the leak. I have not used the memory profiler before, but if I understand it correctly it has something to do with getting the connected client count.

Let me know if I'm missing something.

[PontusHolmberg]

Anyone has any idea what is going on here? Am I reading the memory profiling correctly?

[chkr1011]

How many clients to you connect? Please also share the code how you publish the messages.

[PontusHolmberg]

hm, during a normal day there about 120-150 stable connections, meaning clients that says connected for longer periods of time. I've also got a PHP backend that connects to it using the MQTT client from karpy47

The environment where I use your broker, is where I have a web based portal and an android app. Changes made on the portal are pushed out to the app, so the app dont need to be polling the server. The app can affect each other, but they need to notify the server of the change and it will then push it out the whom ever needs that info.

So the backend only connects, publishes and then disconnects, and the code is very simple:

$client = new MQTTClient($server, $port);
function connectToMQTT() {
    return $client->sendConnect(uniqid());
}
function sendMsg($topic, $dataArr) {
    $client->sendPublish($topic, json_encode($dataArr));
}

function closeConnection() {
    $client->sendDisconnect(); 
    $client->close();
}

Then it's used like:

include 'pushMsgHandler.php';
if (connectToMQTT()) {
    foreach($newCandy as $candy) {
        */ Some logic and stuff */
        $dataOut = [];
        $dataOut['NAME']=$candy['NAME'];
        $dataOut['TYPE']=$candy['TYPE'];
        sendMsg('*some ID*/eventName', $dataOut);
    }
}
closeConnection();

It's sent with QoS 1 (default).

The server also has a timer that publishes to all clients 3 times per day, telling them to sync up with the server, in case something did get missed. The is published with this:

 var message = new MqttApplicationMessageBuilder()
    .WithTopic(topic)
    .WithPayload(msg)
    .WithQualityOfServiceLevel(MqttQualityOfServiceLevel.ExactlyOnce)
    .Build();

 mqttServer.InjectApplicationMessage(
    new InjectedMqttApplicationMessage(message)
    {
        SenderClientId = "broker"
    });

[PontusHolmberg]

Any ideas or additional info I can provide to help sort this out?

[PontusHolmberg]

Could persistent messages be the issue here? I do need that feature, so it there anyway to implement a feature where the persistent messages are saved to a file instead of to memory? I feel like retaining does this, but I also get a feeling that that is different from persistent. Correct?

[PontusHolmberg]

I've run some more tests. I was worried that the issue was actually with the unacknowledged packages. I've check the count of _unacknowledgedPublishPackets and this is what I found. On friday when I shut down my computer, there was around 160 sessions and 1550 unacknowledged packages in total. Memory usages was around 230Mb

Today (monday morning) I check in on it again and now there was around 210 sessions and 1500 unacknowledged packages. Memory usages was around 365Mb.

It's not resonable that 50 sessions consumes 130+Mb of memory.. Taking a snapshot of the memory usages points me to a HashMap and/or Dictionary in MQTTSession and MqttClientSubscriptionsManager