Proposal: Read initial state of properties from retained value of MQTT broker

[euphi]

I think it would be great if Homie could read the initial value of a property from its retained MQTT message.

Example use case: If a value can be set from a homie device (but is not a sensor value), the homie-device should restore it after restart. (E.g. a set-temperature for a thermostat.)

So:

1. homie-esp8266 publishes a property as retained message. E.g. homie/Labor/Thermostat/SetTemp 22.10.

2. homie-esp8266 is restarted (for whatever reason). The MQTT broker and the automation system (e.g. openhab) continue to run. (Note: Thanks to retained messages, the case for restart of the automation system is already covered.)

3. After restart, homie-esp8266 subscribes for its property messages (at this step, it does not yet subscribe for the ../set message!)

4. So, if a retained message is available, it is received by homie-esp8266 and is used to set the initial value of the property.

What do you think about the proposal?

Open questions for an implementation:

• Should homie-esp8266 automatically subscribe or should there be a getRetainedValue() method?
• Should homie-esp8266 use the handleInput(...) method to handle the received retained value or should there be an extra callback, e.g. handleRetainedValue(...)?
• When is the correct time to unsubscribe again? Inside the handleRetainedValue(...) method?

Other approaches to find a good solution for the use case "persist values in homie":

• Store locally in SPIFFS or EEPROM --> drawback: Not useful for data that is often changed (Use lazy write?)

• Make the home automation system send the persisted values in a set-message after a reconnect from a devicve. --> drawback: Quite complicated to configure.

• Variant: Let the home automation system always send a "set" message if a value has been changed by the home-device itself. --> drawback: homie is supposed to "confirm" a set-value by sending the set value as mqtt message. So this would result in a loop.

[bertmelis]

It would be very welcome for me too. I'm thinking of my gasmeter pulse counter: the total count (= representing the current meter total) is reset when restarting the ESP. Hence I have to manually set the total value again. And as my logging is working with diff values, there's always a spike when doing so, which I have to "delete" too. This could be avoided by resuming using the retained value.

My preferred method would be something analog as the .settable implementation. I think it's up to the integrator to decide whether is should be implemented or not.

Homienode.advertise("property").settable(SetProperty).setRetainedValue(RestoreProperty);

bool RestoreProperty(const HomieRange& range, const String& value) {
  //handle restore
}

[marvinroger]

Even though Homie for ESP8266 is an high level abstraction, I'm not sure everything should be abstracted. Indeed, in your case, the data is specific to the device, so it makes no sense to store it in the broker, knowing this is very simple to store data on the ESP8266 (SPIFFS, EEPROM, see https://github.com/marvinroger/homie-esp8266/issues/154)

[euphi]

As my thermostat is now running fine, I still think the most elegant solution to restore the last value of the set-temperature is to read the retained value from MQTT.

I think, this should by added to the homie convention as recommended way to restore data that is set to the device.

Reasoning:

• data that is sent from the automation system to the Homie device belongs to the automation system.
• the MQTT broker is the proxy of the automation system.
• if a Homie device restarts, it needs to know the last value of settable properties.
• the "pessimistic feedback" approach of Homie has the effect that the valid value is always sent from Homie to the broker.
• if this value has been sent retained, it can easily be read from MQTT during startup.

[luebbe]

An additional plus is that it wouldn't require a reboot to change "trivial" config values. Currently I have a display node that fetches values from a MQTT topic and displays them. Right now the topic is stored in the config in order for the display to start up and show the values automatically. Whenever I want to change the topic, I have to send a new config and reboot the device. If the topic can be read from a retained "set" property, it would allow the device to start in a defined state plus the topic can be changed easily without a reboot.

[timpur]

In theory, homie should receive the retained message from any /set property? if this is not the case it might be a bug and ill look into fixing this?

[jamesmyatt]

I think that the MQTT broker can only send a retained message again (or any other sucessfully completed message) if the device re-connects with the "clean start/session" flag set, i.e. if the broker thinks the message has never been sent before. Alternatively, you could unsubscribe then subscribe again to that topic.

[luebbe]

@Nzbuu That would explain why mqttfx doesn't display some messages when I just subscribe to a particular subtopic after having caught "all" (retained) messages with a homie//# subscription before. It always worked with mosquitto_sub, but since mosquitto_sub re-connects every time, it's now clear to me, why it worked. Thanks, learned something again. This was driving me mad.

[timpur]

I think this is a bug now, just noticing that it doesn't pickup on retained messages in the nodes, i recall that it use to. Will fix.

[timpur]

Actually i think i misunderstood whats going on here. @euphi what did you want from homie?

[euphi]

In my opinion it would be the best solution that:

• Homie shall have a flag that a property is reloaded from retained value.
• If this flag is set and Homie reconnects to the broker (or only on first connect after reboot?), Homie shall read the property value from its retained value on MQTT.

Best way to set the flag would be to add this to the PropertyInterface.

The feature is most useful on settable properties, but it can also be useful for properties that are not settable over MQTT, but settable in other means, e.g. user-interaction.

[jamesmyatt]

It turns out that it is possible to obtain the retain message by re-subscribing without unsubscribing first (according to the v3.1.1 spec)

If a Server receives a SUBSCRIBE Packet containing a Topic Filter that is identical to an existing Subscription’s Topic Filter then it MUST completely replace that existing Subscription with a new Subscription. The Topic Filter in the new Subscription will be identical to that in the previous Subscription, although its maximum QoS value could be different. Any existing retained messages matching the Topic Filter MUST be re-sent, but the flow of publications MUST NOT be interrupted [MQTT-3.8.4-3].

I assume that Homie re-subscribes to everything every time it connects, since it won't know what it's currently subscribed to. Hence, you should get a retained message at that point.

However, then there's always the issue that the broker may discard the retained message at some point, which it is permitted to do by the specification when it was published at QoS 0:

[The Server] SHOULD store the new QoS 0 message as the new retained message for that topic, but MAY choose to discard it at any time - if this happens there will be no retained message for that topic [MQTT-3.3.1-7]

[euphi]

Ok, as the homie-device shall reread its own values, its up to the device to set retained flag and the QoS level.

[timpur]

So just add a flag to show when the message is the loaded retained messages for the set handler ?

[euphi]

No. An additional "one-time" subscription for the property-topic without set.

I use my thermostat for an example:

homie/wz_thermo/Thermostat/$properties SetTemp:settable,Mode:settable

Both Properties of the "ThermostatNode" has been set by my controller (openHAB) - and then confirmed by Homie with a sustained message:

homie/wz_thermo/Thermostat/SetTemp 20.90
homie/wz_thermo/Thermostat/Mode Auto_OFF

When there is a restart, the device needs to know the current SetTemp and mode to display it correctly.

The simplest way would be to reload the value from these messages.

Why not homie/wz_thermo/Thermostat/SetTemp/set?

• There is no need to send this message sustained, because it is just a command that is invalid after confirmation.
• The SetTemp can also be changed by the Thermostat itself, then no set topic has been sent.

[timpur]

I see now. Let me look into this and I'll get back to you about the best solution (in my opinion).

[jamesmyatt]

I think that seems OK for this relatively simple situation, but it's worth considering that there might be other ways to change the properties. For example, you might also have buttons on your thermostat to change the temperature directly. How would it work in this situation?

I see the "set" messages as commands. They should be executed once and then discarded.

[nerdfirefighter]

@Nzbuu I tend to agree with you. the examples provided, the gas and thermostat both in my mind would be best done as Custom Settings (stored on the device) for the following reasons:

1. they don't change that often. (really how often do you change the themo set temp of the heater/cooler)
2. They both (should) have a requirement to work even when the MQTT is not available e.g. what happens if the gas monitor is rebooted while the network is down, you miss pulses because when it reconnects and get the last posted values, which is now a number of pules behind. or you have to write more code to handle this gracefully. for Thermostat, your example of if there is local control as well as remote control, the source of truth has to be the device its self, to the message it posts.

MQTT persistence is really more for the consumer so they can get an initial value between the time they start consuming the MQTT channel and when the device sends a new message.

Any way, my two cents on the matter and sorry for playing devils advocate.

[ingoogni]

Agree with @Nzbuu the state of things may have changed during the down period and or reset and you'd be working with stale data. Also a "drop out" should be seen as a "drop out" and not be smoothed away.

In analogy with @bertmelis gas meter, what I do with a water flow. Every second I push the flow and measurement period to the database. Then I use Postgresql's listen/notify pub/sub system to publish the total volume back tot the device that measures and that displays it. Both messages contain an uuid, when they don't match something went wrong in a measurement series. (currently not via MQTT and Homie-ESP but direct from ESP to DB and back)

[jamesmyatt]

These are all really good examples of cases where you might want to retrieve a property value that isn't updated via a "set" command at all. It would be great to be able to restore the last value at power-up without storing every increment in SPIFFS. I see this as storing a "state" variable (or dictionary) rather than a "command" variable.

I think that it's just tough-luck if you miss some pulses while the device is powered-off. If that's a problem, then you need some kind of battery back up.

[euphi]

The feature to reread values from sustained message is meant for a restart of the Homie device - not a restart of the MQTT broker.

In detail:

@Nzbuu Thats exactly the reason why I think the property message from homie to controller shall be re-read and not the set message.

@nerdfirefighter The feature to reread sustained values is meant for failure and restart of the Homie device. If the MQTT broker is unavailable there is no need to read values after the reconnection to the broker, because the values are still stored in the homie device.

@ingoogni Most data can't change during the down time of the Homie device. If it is a settable property the homie device must confirm the value to make it valid. This is not possible as long as the homie device is unavailable. If it is a value that is "created" at the Homie device, it can't be updated when the Homie device is down. Then, in your gas/water meter examples you loose "counts", however solving this with redundancy etc. is beyond the scope of Homie.

The only "tricky" situation I see is if both MQTT broker and Homie device are restarted at the same time (e.g. due to power fail) AND the MQTT broker takes signifcant longer to restart than the Homie device, so the Homie device can "count" several ticks before it can read it old value from MQTT. However, also in this case, rereading the sustained value helps solving the problem: After MQTT reconnction, the homie device can add the counted ticks during to the retained value and update the value acconrdingly.

[nerdfirefighter]

On 11 Jan. 2018 23:57, "James Myatt" notifications@github.com wrote:

These are all really good examples of cases where you might want to retrieve a property value that isn't updated via a "set" command at all. It would be great to be able to restore the last value at power-up without storing every increment in SPIFFS. I see this as storing a "state" variable (or dictionary) rather than a "command" variable.

Yet to see any examples of why you would do this. It seems to me like its bloating a really nice lite weight code base.

I think that it's just tough-luck if you miss some pulses while the device is powered-off. If that's a problem, then you need some kind of battery back up.

I think you have missed the point here. Point of the example was, most of the time the device where you need to start from a last know good value. Is likely a device that needs to operate with out relying on mqtt to configure its start up values.

E.g. i have a relay with its own physical button. I have a requirement that it returns to its last state on power up. I keep the source of truth for state in a custom config item. That way if there are any mqtt issues, or the device boots faster on recovery then my server, wifi, etc it will return the relay to its saved last state. It also ensures that if the physical button is used while mqtt is down, the last good state is still tracked.

[ingoogni]

The water does not stop flowing when power fails. On restart the count continues but I have 100l more in the tank than the display tells because it continued with old data. Not Homie or MQTT problem to solve IMO.

[nerdfirefighter]

On 12 Jan. 2018 00:04, "Ian Hubbertz" notifications@github.com wrote:

The feature to reread values from sustained message is meant for a restart of the Homie device - not a restart of the MQTT broker.

In detail:

@Nzbuu https://github.com/nzbuu Thats exactly the reason why I think the property message from homie to controller shall be re-read and not the set message.

@nerdfirefighter https://github.com/nerdfirefighter The feature to reread sustained values is meant for failure and restart of the Homie device. If the MQTT broker is unavailable there is no need to read values after the reconnection to the broker, because the values are still stored in the homie device.

@ingoogni https://github.com/ingoogni Most data can't change during the down time of the Homie device. If it is a settable property the homie device must confirm the value to make it valid. This is not possible as long as the homie device is unavailable. If it is a value that is "created" at the Homie device, it can't be updated when the Homie device is down. Then, in your gas/water meter examples you loose "counts", however solving this with redundancy etc. is beyond the scope of Homie.

The only "tricky" situation I see is if both MQTT broker and Homie device are restarted at the same time (e.g. due to power fail) AND the MQTT broker takes signifcant longer to restart than the Homie device, so the Homie device can "count" several ticks before it can read it old value from MQTT.

Thats my point.

However, also in this case, rereading the sustained value helps solving the problem: After MQTT reconnction, the homie device can add the counted ticks during to the retained value and update the value acconrdingly.

This adds complexity because you have to write code for that logic and related data validations. And also work out when to do that logic. By using a custom setting to store the source of truth you remove the need for this over conplication of logic.

[nerdfirefighter]

On 12 Jan. 2018 00:16, "ingoogni" notifications@github.com wrote:

The water does not stop flowing when power fails. On restart the count continues but I have 100l more in the tank than the display tells because it continued with old data.

I get your point but that is a bad example as that is the measure of the capacity of the tank which you cant reliably get from flow rate in to the tank due to this situation indicated. You would measure the flow to calculate how long it would take to fill/empty the tank and measure the capacity of the tank to know how much you have in the tank.

Regardless of if the value comes from the mqtt last set or on board its still going to be wrong value in your example. You would need to send a new set command to the device to advance/correct it to resolve the issue. Which is different to reading back the last state.

[ingoogni]

Yes. The only way to know a state is to measure/determine it and not depend on data from the past, regardless where they come from DB, Broker, EEPROM...

[timpur]

I might be misunderstanding but the current state of the property should be the same as the /set.

The property should be in sync with the /set. The property is there to show feed back of a changed state from /set (for settable properties). If you have a property that is settable, then you should always use the \set to set the a new state internal or external. This way you always follow the same flow to change states and you can get the latest stat as the retained /set will be received and flow through the same slow, as in the set by the set handler.

Now currently there are issues with trying to follow this flow, but i believe we should be following this flow and fix the issues with homie to allow this, instead of creating a new flow. Follow the same flow for internal as you normally do from external property setting.

An issues is, if you lose connection to mqtt, pushing to /set is impossible.... Fix could be that if we are disconnected and we use /set then it will bypass to the set handler?? This way we can always use this flow if were online or offline to set new states of the device..

obvious issue is now do we send the /set state, when back online because the /set was not sent as we were offline....? I think that it will be hard to make homie 100% full proof.... We could implement a queue .... but not things are getting complex

lest find a happy medium ??

[jamesmyatt]

The problem is that there are lots of situations where it's not as straightforward as just re-executing the set command. For example, there may be no set command like in a pulse counting device.

Also, the value sent to the set command may not be the value you want to restore. For example, you may want to implement your device to handle "volume/set" <- up or down. Then there's no point in restoring "up" or "down".

[jamesmyatt]

To clarify, I think it would be very useful to be able to restore settings and last known state variables from an MQTT retained message, but I think it's going to be hard to find something that satisfies most use-cases and I don't think that just re-executing the last set commands is going to work: there are too many exceptions.

[marvinroger]

Remember that Homie is state-based. You don’t tell your device to “turn on the device”, but to put its “on” state to “true”.

Same thing for the volume.

Cordialement, Marvin ROGER

Le 12 janv. 2018 à 17:32 +0100, James Myatt notifications@github.com, a écrit :

The problem is that there are lots of situations where it's not as straightforward as just re-executing the set command. For example, there may be no set command like in a pulse counting device. Also, the value sent to the set command may not be the value you want to restore. For example, you may want to implement your device to handle "volume/set" <- up or down. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[timpur]

I dont think homie should go there, you should be solving this through pattern not new feature.

I think ill add a flag for the set handler, so you can tell if its the restored retained message and you can decide if you want to restore this message (act on it).

You should have a volume node with properties of control (up/down) which influences the volume state property (percent), just like a dimmer has value and switch... restore the last set volume percent but not the last control (up/down) in that example, something like that.

We dont want the true current state to be store in any number of locations, on the device or on MQTT, it get complex quick....

[euphi]

@timpur I can't follow your last comment. You say that there should be no new feature, but also that the current state should not be stored in different locations.

Your control example is a good example why retaining the set-message is not useful in many cases.

I still think an optional additional subscription to the property topic that is unsubscribed after first message is most useful for the majority of use cases.

BTW: Using the set topic also for changing the property for internal reasons, is not only problematic if there is no connection, but also if connection is slow/disturbed. With packet loss the set command may be received so late that another event in the users program happend, e.g. second button press. Handling this would complicate things a lot in the user's program.

[nerdfirefighter]

@timpur just to provide feed back on your earlier comment in my understanding of the framework. The short answer is that the advertised settable property's (e.g. 'on') state does not need to be the same as the /set of that property. The /set is the command topic not the state topic and is used only to request the property to change its state to the state requested as the message of the /set topic. There is also no guarantee that the property state will change, e.g. if the property is looking for 'True/False' and you send it 'TURN_ON'. The state change is only confirmed when the property publishes it state.

Long answer: I will use the example code for IteadSonoffButton.ino which is in this repo.

i have a device with a node called 'switch' which contains a settable property of 'on'.

In the example we can turn the switch on and off ether via the 'on/set' or via the physical on board switch.

When done via MQTT the message flow is: devices/sonoffbtn/switch/on/set <- false devices/sonoffbtn/switch/on -> false we can see the 'on/set' has been published from another device on the MQTT asking the node 'switch' to set its state to false and then the node 'switch' advertising that its current state is now false.

if we then press the physical button on the device, the only message we see on the MQTT is: devices/sonoffbtn/switch/on -> true This being the node advertising that is current state is now true.

My philosophy from my reading of the framework and my implementations are that the device should be the source of true of its current state. I can not find an example where the need to pull some form of default value, or last know good state would be better coming from the MQTT rather than from the device its self. (If anyone has some that would be great) In each situation i can think of or that i have seen the device is likely to have other local influences that may cause its state of a settable property to be changed between the the time the last state was published and when the device needs to read it from MQTT.

I think it would be of more benefit to have a flag that allows switchNode.setProperty("on").send(value) to also save that value (state) to SPIFFS (e.g. the config or maybe a state.json file). a flag should also be in the switchNode.advertise("on").settable(switchOnHandler); function to tell Homie if it should set the state to the value in the file when the setup loop is ran for this specific property. This can also be used if you always want properties to be a specific state on boot, this can be done by telling Homie to use the saved state during setup, but have the flag for the setProperty.send not to save its value/state to the file. thereby using the saved state as the default state on each boot.

[timpur]

@nerdfirefighter

Great points. You guys might be right, maybe the device should be the source of truth. Let me ponder on this, just thinking the best way around this, spiffs can be problematic if we write to it to often ....

[euphi]

Using SPIFFS is dangerous. The write/erase cycle count of the typically used winbond flash chips is just around 100'000. If you write a value every minute, this is just 70 days.

Or - if you want to your device to last at least 5 years, you shall not write to SPIFFS more often than every 26 minutes.

Update/Edit: Oh.. it seems that SPIFFS supports wear leveling and does not erase a block if other blocks are free, so the limit is much higher, especially if the SPIFFS is not that full. However, on most of my devices, the SPIFFS "partition" is only 64kb, so 1 block, 512 pages. With the minimum of 2 pages per files and assuming most other pages are unused (--> no UI bundle installed!) , the "5 years" write cycle time reduces to approx 6 seconds.

[timpur]

@euphi was that a typo ? 6 seconds ??

We can probs use a states.json file which will be small so spiffs should be able the cycle this file around to conserve spiffs life span. (ps. reducing the ui size is in work for v2.1)

Also this reminds me to check were using spiffs correctly for updating files (delete and write new file).

[euphi]

No, it wasn't.

If you want your flash to last at least 5 years and you have a 64 k SPIFFS with only config.json and states.json, you can write to SPIFFS every 6 seconds.

[timpur]

okay so from 26 min to 6 sec. Awesome :)

[timpur]

@euphi Work with me on implementing this, thinking about adding it to v2.1

[euphi]

https://stackoverflow.com/questions/49636231/mqtt-how-to-know-when-all-retained-messages-have-been-received

[euphi]

I'm wondering how to know when all sustained messages have been received, so I opened a question on stackoverflow.com

[timpur]

You have an answer :( doesn't seem will know so will have to assume ...

[euphi]

At least there is a proposal to send an un-retained message that should be received after the last retained message.

[BasvanH]

Has this ever got implemented? Can't find any reference of it in the docs for v3.0.x

[not7cd]

@BasvanH it looks like it was dropped from the roadmap

[Mediokerman]

Just to help anyone who comes across this thread trying to get their Homie device to read the last value of a settable property after restarting the device (eg: whether the light should be on or off, what temperature the thermostat is supposed to be set at, etc.):

This problem seems to be solved by using "retained" MQTT messages. This may have always been the case, I'm not familiar with how Homie or OpenHAB functioned in 2017. Your setup may be different, but for me using OpenHAB I just had to change the Channel config on the Thing and turn on the "Retained" option. Seems obvious now, but I spent hours trying to solve this "newbie" problem, so hopefully this will save someone else some trouble. =)

A retained message is a normal MQTT message with the retained flag set to true. The broker stores the last retained message and the corresponding QoS for that topic. Each client that subscribes to a topic pattern that matches the topic of the retained message receives the retained message immediately after they subscribe. (from https://www.hivemq.com/blog/mqtt-essentials-part-8-retained-messages/)

[camiloavil]

@Mediokerman Could you explain how you changed the channel config on OpenHAB to set the retained option?

[Mediokerman]

@camiloavil I don't know if this will be the case for everyone because it seems like there are several different ways to configure MQTT in OpenHAB, but the way I am using it, I have a "Thing" that is of the type "Generic MQTT Thing". When adding/editing a channel on this Thing in Paper UI you need to click the "Show More" button and then it will give you the option to make it Retained. Hopefully that helps!

[valpackett]

It's unfortunate that the spec just says

A Homie controller publishes to the set command topic with non-retained messages only.

A really important use case for retained set is being able to enqueue a "to-do" task for a device that spends most of its time in deep sleep, waking up e.g. every 10 minutes to check if the "to-do" flag has been set.

I suppose the current way to do it would be to have the device hang around for some time after waking up, waiting for the controller to resend the flag in response to the device becoming online?