MQTT Message Delivery Improvement

[Chapoly1305]

As mentioned by @System21 in https://github.com/seemoo-lab/openhaystack/discussions/213#discussioncomment-7870449, seems there have been a few times that message failed to deliver to the receivers, however, there was no further information provided.

I have looked at the code of this project, multiple places used QoS:0 which might allow the message to be discarded, which makes me wonder maybe it's worth to give a try to adjust the location message QoS from 0 to 1 in code could improve the delivery.

https://github.com/mrmay-dev/owntags/blob/30e02091c5bfc52cbf180fbb7f358e07603c3e75/application/OwnTags_plugin.py#L184

Ref: https://www.hivemq.com/blog/mqtt-essentials-part-6-mqtt-quality-of-service-levels/#:~:text=Published%3A%20February%2016%2C%202015,guarantee%20for%20a%20specific%20message.

[Systm21]

I dont think, this is a problem with this Code, it has something todo with owntracks itself.

I have 12 Trackers... Its Random, but 4-5 are sometimes not getting an Update (in Hass and in the App). So when updates are coming on 12.00h, 13.00h and 14.00h, i have it, that one or two values are missing. Dont know why, but in mqtt, the messages are getting piped. So it would be better to have a native mqtt support (plus Mqtt discovery) in Findmy, without any detours over other plugins.

I dont know but maybe its conceivable to have some kind of "bridge" with Mqtt, HTTP Endoint and API or something like that. If i have the programming skills, i have done that, but i have not.

[Chapoly1305]

That certainly worth to deep dive for a bit more. From what I experience in the past, it's sometime caused by the networking issue, this could happen between this project as publisher to MQTT Broker, or Broker to subscriber. Could be tricky to locate the cause if have no traffic capture or program logs to locate the issue.

Could you help on elaborating on your environment? Like,

• [ ] have you used public free MQTT server/broker, or you have setup your own broker?
• [ ] has this happened frequently, what's the rate of occurrence? Every day, a few times a week?
• [ ] when it happened, were the tags at the same place and in the range of other Apple Devices?
• [ ] when mentioned "Dont know why, but in mqtt, the messages are getting piped.", is the "getting piped"means the messages have been sent from this project to the broker, or ... ?

The OpenHeyStack was programmed with Swift and I don't really have experience with this language, the OpenHeyStack heavily depends on MacOS 13 and it's no longer supported by MacOS 14 without a significant refactor. To support the concept for Android users and for the future, it's invertible to use all sorts of combination of plugins to emulate an apple device to interact with Apple server and to retrieve the reports, after that, return the report as a response of API, or forward the message to MQTT broker. It's quite hard to see this could be 99.999% reliable as a commercial product.

However, if you wish to have a much stable tracking, the Apple MFi AirTag is absolutely worth to have a look. On Aliexpress, which is much much cheaper than the stock price AirTag, about 10USD per piece, and can use the native Apple FindMy App (if you have an iPhone or other apple product this is more convenience).

[Systm21]

I think, you think to big. The "bridge" could maybe integrated in your web_service.py

[Chapoly1305]

@Systm21 I updated some of the code in my personal fork https://github.com/Chapoly1305/FindMy , you may pull the latest and set up your own environment to make an experiment if the stability is improved. I have set the report to be retrieved and published at every minutes with MQTT QoS 1. I have tested with my raspberry pi as the tag, it seems working to me.

If you don't mind the password and location reports being saved in database for forwarding, you could also use the website on my server https://hello-world-proud-fire-51f6.hallo-world.workers.dev/docs as the bridge to retrieve the reports of your devices. Just use the /KeyToMonitor API for adding credentials and delete them later with /Tag_Removal API if you want. No need to call the /Publish_MQTT, I have set the system to call it automatically, but if you want to set up your own server, you will need a crontab task for automation.

* * * * * /usr/bin/curl -X 'POST' 'https://SERVER_ADDRESS/Publish_MQTT/' -H 'accept: application/json' -d ''

🫡 🫡 🫡

[Systm21]

You really should make docs, that describe the functions. Extracting everything from the Code is really laborious.

What can i do with your site, its blank?! I dont want to pipe my data over 3rd Party Servers, so the script should do it locally.

Is it based on Owntracks or Homeassistent MQTT Autodiscovery, or both?!

[Chapoly1305]

It's tested with owntracks. The link itself is an API document, I have no issue access it from both my phone and laptop.

[mrmay-dev]

If messages are missing on the OwnTracks client the problem could happening in three places.

1. When OwnTags script requests reports from Apple.
2. When location reports are sent from OwnTags to the MQTT Broker.
3. Between the MQTT Broker and the OwnTracks client.

The first two would trigger errors from the OwnTags script. Output from STDOUT and STDERR would diagnose this.

For the third problem, I think @Chapoly1305 has an interesting idea. As I understand it, QoS is changed to adjust traffic on the network. Lower QoS generate fewer verification messages but delivery is less reliable, higher QoS ensures delivery at the cost of increased traffic.

I asked ChatGPT to explain more about QoS. Check out the response here: MQTT QoS Brief.

I'll add a QoS option to the script.

[Systm21]

@Chapoly1305 Problem persists, i have a newer location in the headless-haystack app (its plausible), than in MQTT (but time of last change is changing). Dont know the problem but its really annoying.