Supressing redundant data on MQTT

[supermachinista]

It took a hard few days to get usable data from my new Wican adapter, and the main reason for that was the high frequency traffic on MQTT. Even when idle there are two telegrams per second with always the same content. I had to look for the important data like I was looking for a needle in a haystack.

I'm probably not alone in this experience and I have read that there is a filter architecture implemented in the firmware that is intended to reduce data traffic and also brings some other advantages such as calculation. So far I haven't been able to really filter out the data in focus, but I think I'm still making a mistake somewhere.

The reason I'm writing now is to ask whether a different approach could solve the problem: Is it possible to implement a mechanism that only sends an MQTT message if the message has changed and all redundant messages otherwise suppressed? Is this possibly possible by simply taking the cross sums of the messages and comparing them?

Best regards Thmas

[meatpiHQ]

@supermachinista I can help you to set up the filter, it should be perfect for what you're trying to do.

Please post here an example of a CAN frame that you want to process and which bytes you want to use in the calculation.

[supermachinista]

 

Hello, thank you very much for Your reply. I think I understood the filter mechanism and the calculation option. It is kind of a "whitelist" which only lets pass the desired id´s. The problem is that despite the filters set, "idle frames" appear in MQTT traffic, twice per second, regardless of whether there is an RX with data or a publish or nothing. These frames come constantly and they always have 32 in the first byte and 16 in the second. The last byte is sometimes 0 and sometimes 128. The ID is always 401604624 and "extd" is true. I noticed this on an e-Golf and an Audi. It may be that this problem only occurs with VAG vehicles, i.e. Volkswagen, Volkswagen Commercial Vehicles, ŠKODA, SEAT, CUPRA, Audi, Lamborghini, Bentley, Porsche and Ducati. But maybe it's the same with other vehicles. In any case, it would be very helpful if there was a way to hide frames of a certain type, i.e. a blacklist. Would something like that be possible?

 

The "idle frames" all look like this:

{"bus":"0","type":"rx","ts":37993,"frame":[{"id":401604624,"dlc":8,"rtr":false,"extd":true,"data":[32,16,0,0,0,0,0,0]}]}

{"bus":"0","type":"rx","ts":4801,"frame":[{"id":401604624,"dlc":8,"rtr":false,"extd":true,"data":[32,16,0,0,0,0,0,128]}]}

Best regards

Thomas

   

Gesendet: Sonntag, 24. März 2024 um 11:07 Uhr Von: "MeatPi Electronics" @.> An: "meatpiHQ/wican-fw" @.> Cc: "supermachinista" @.>, "Mention" @.> Betreff: Re: [meatpiHQ/wican-fw] Supressing redundant data on MQTT (Issue #109)

 

@supermachinista I can help you to set up the filter, it should be perfect for what you're trying to do.

Please post here an example of a CAN frame that you want to process and which bytes you want to use in the calculation.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

[meatpiHQ]

@supermachinista this is weird because only the frames set in the filter should pass and nothing else. Can you please post a screenshot of your filter settings? I'll try to replicate on my side.

[supermachinista]

@meatpiHQ Yes, you are right, please excuse the confusion. I routed the MQTT traffic via a public MQTT test server (broker.hivemq.com), that's where the problems occurred. After switching to my local mosquitto server, like magic the filter worked as it should. WICAN now delivers the data perfectly. Sorry again and thank you for your time.

Regards Thomas

[meatpiHQ]

@supermachinista please don't connect your car to a public mqtt broker.