Client application is missing messages at higher loads on 32bit .Net 4.8 applications.

vinaybr commented 1 year ago

Client application is missing DataReceived events

Which component is your bug related to?

Client

To Reproduce

i have data being received on a musquitto mqtt broker. the publisher module is publishing data at an interval of 500ms. I have 2 clients applications(same source, different target) one built using .Net6 and x64bit while the other app is built to target .Net4.8 and x86. while the .Net6 client receives all the messages, the .Net4.8 is missing some of the data packets. This is observed at high loads where the publisher is publishing 100msgs / 500ms (each of the 100msgs is for a different topic).

Expected behavior

All clients should receive all the messages

Screenshots

-

Additional context / logging

vinaybr commented 1 year ago

an update to the issue here, we are seeing messages missed and in some cases received in wrong order. its more frequent in 32bit .net4.8 but happens even on 64bit .Net6.

coffeeaddict19 commented 1 year ago

In mqttnet, the default QoS subscription for a topic is 0. That may explain why you are missing messages or seeing them in an unexpected order. Not having seen a sample of your connection code this is only a guess.

Example ...

var subscriptionOptions = m_mqttFactory.CreateSubscribeOptionsBuilder(); foreach(var topicname in connectionData.TopicsToSubscribeTo) { subscriptionOptions.WithTopicFilter(topicname, MqttQualityOfServiceLevel.ExactlyOnce); } await m_client.SubscribeAsync(subscriptionOptions.Build(), m_cancel);

tkouba commented 8 months ago

I have about 13500 retained messages/topics, but I can receive only 1179. The same number after many runs. I tried change QoS of subscription, nothing. Broker mosquitto.

MGasztold commented 2 months ago

Hello. I also have this issue. I use .NET Framework 4.6.1 and MQTTnet 4.3.7.1207.

My app is subscribed to a topic on which devices in my system report Bluetooth Low Energy scan reports.

For a test I filter the received messages to print only the ones related to single device to the console.

I have MQTT explorer desktop app on Windows (the same PC on which I develop the .NET app) and mosquitto_sub on my other laptop with Linux as reference connected to the same MQTT broker. They both receive the same number of messages for the device i am interested in, e.g. 32 messages every couple of minutes.

At the same time my .NET app receives only 9 after approximately 40 seconds delay, the references mentioned above received the same number of messages (32) and all happens in the same time on both.

The fact is that the rate of messages being published on that MQTT topic is quite high, there are around 12k messages published per second. My app is only subscribed to listen. it does no other work.

I am not doing any work with the messages in .NET. at this point. I use very standard MQTTnet client implementation, I tried already subscribing with QOS 2. Nothing helped.

I commented out adding messages to the queue that I use for processing them just to be sure there is no bottleneck anywhere and I just want to log each message that contains my mac address. Simple as that:

mqttClient.ApplicationMessageReceivedAsync += arg =>
            {
                var segment = arg.ApplicationMessage.PayloadSegment;
                string message = Encoding.UTF8.GetString(segment.Array, segment.Offset, segment.Count).Trim();
                if(message.Contains("fd29f03a8931"))
                {
                    Console.WriteLine(DateTime.Now.ToString("hh:mmm:ss") + " fd29f03a8931 scanned! " + message);
                }
                return Task.CompletedTask;
                //_messageQueue.Add(arg.ApplicationMessage);
            };

Any idea what may be the reason of my problem and how to workaround ? Thank you.

MGasztold commented 2 months ago

I am starting to think that basically C# is not the right language for the job I need to do due to too high rate of messages on MQTT.

I am constraint to having my project in .NET but maybe I must make it launch another process (e.g. written in go) in the background to do the work in MQTT...

dotnet / MQTTnet