search

homieiot / convention

🏡 The Homie Convention: a lightweight MQTT convention for the IoT
https://homieiot.github.io/
Other
710 stars 59 forks source link

Extension proposal: last update time for property #191

Closed maxberger closed 4 years ago

maxberger commented 4 years ago

Hi,

I would like to propose an extension which adds a time for a last updated of a property. But there are several items I would like to discuss.

Basic idea: Any property can have an "last updated" value which represents the time at which the backend for this value has actually been updated.

Why is this needed: Some backends, specifically gateways to other systems, may publish "old" values, which may or may not have changed since. This needs to be on a property basis, so adding this to the overall device is not sufficient.

Open Questions: 1 What is the best format for such a timestamp? I see several options: 1.1 seconds since the actual update, integer format 1.2 unix timestamp of the last update, integer format 1.3 time of the last update, some common time format (such as ISO time)

2 What name would fit best? $last-update?

3 What is the best level for this extension? I would need it for property, but is there a reason to also have this on node?

Some Pro's and Con's I see for 1: 1.1: Pro: Very easy to create, does not depend on time synchronisation 1.1: Con: Clients need to calculate timestamp based on their own clock when they receive the message. If the message is RETAINed, that does not work. 1.2: Pro: This is the datatype i get from the backend and which I want to see in the frontend. So in my case, no calculation / conversion is necessary 1.1, 1.2. Pro: No additional datatype needed 1.2, 1.3 Con: Time may not be synchronous between devices, a device may not even have an absolute clock. 1.3: Pro: Human readable 1.3: Con: ISO time is not an current data format, so that would also needed to be added 1.3 Con: The usual time zone issues

What I plan to do:

[Background: I am currently writing a zway -> Homie gateway, hopefully to be published publically at some point].

jamesmyatt commented 4 years ago

In my opinion, if you have information like this, then it needs to be atomic (i.e. you get both value and timestamp or neither), especially since you can't guarantee the order that the messages are received or the time between the messages being received, or even that the requested QoS is granted by the broker and that the message arrives at all. For MQTT, this means publishing a single structured message that includes the time and value(s), e.g. JSON-encoded, and relying on the consumers to unpack it.

UNIX timestamps in UTC would be by far the easiest to implement and process. ISO-8601-format times are good for humans but are an unnecessary overhead on resource-limited devices, i.e. the sort of device that you'd want to use MQTT with.

Now the Homie firmware doesn't include NTP (or similar) by default and ESP8266 projects don't usually have RTC chips unless they're clocks, so there's no guarantee that an absolute time is available, otherwise it's up to each device to get its own time, which is not trivial (but also not hard). There's also typically a delay from start-up to getting a correct absolute time, so you'd need to decide what to do in this period.

davidgraeff commented 4 years ago

An extension with absolute timestamps is doable. MQTT is unfortunately not the right protocol to request atomic updates for multiple paths. $last-changed can only be interpreted as an additional, optional, maybe-inaccurate help for a controller to have an idea of the last update time.

maxberger commented 4 years ago

Hi, so to sum it up:

For Json: bundling the last update with the actual data would make this incompatible with everything else, so I will not go into that idea.

Thanks! I'll try to write something up based on that feedback.

kollokollo commented 4 years ago

In case we would not want to use JSON for the payload, how would one add a timestamp value to the (lets say number)? Just separated by a blank, following the number? Maybe the timstamp value can identify itself by a single Byte, like

"12.5 T1580396855.826"

Here the first part is the actual properties value and the part starting with a "T" is a timestamp. This extension would be compatible if clients which do not evaluate timestamps, just ignore any part after the first blank.

mjcumming commented 4 years ago

We had a discussion about date/time here https://github.com/homieiot/convention/issues/160

I wonder if you could change the payload to encapsulate both the time and the value. Otherwise if you can't match the time to the value I think this extension would not live up to its purpose.

ThomDietrich commented 4 years ago

Adding timestamp to the value is nonsensical. This breaks everything, including the aimed for simplicity of Homie. The additional topic is not synchronized but fulfills all other requirements. Please also remember that an extension ought not to modify Homie core behavior, e.g. the value of a property.

If a sync is seriously needed a combination of value and time stamp could be implemented in the extension (value&timestamp checksum?). Please name a scenario in which this would be necessary.

I suggest to continue discussion for this topic in the existing Pull Request.

https://github.com/homieiot/convention/pull/192