Update form outside detected. Original value restored!

[sognen]

I have an item defined as

       **"fanPower": {
          "$name": "Power",
          "$datatype": "string",
          "$settable": true,
          "$retained": true
        }**

that when flows are restarted (or at startup) gives the following error message

"[homie.state:fan] fan/fanPower=True update form outside detected. Original value restored! (False)"

This causes the value to rapidly shift from True to False and back to True in an endless loop. It only occurs at node-red startup and when flows are restarted.

I, for the life of me, don't understand what could cause this error. MQTT nodes are configured as follows:

mqtt-in Topic: homie/livingroom-hvac/fan/fanPower QOS: 1

mqtt-out Topic: homie/livingroom-hvac/fan/fanPower/set QOS: 1 Retained: false

The Homie node is configured with auto confirm and pass messages through both enabled.

Any insight into what might be causing this is much appreciated.

[Christian-Me]

Hi.

this should be a feature not bug 😀

If someone writes to anything except homie/?/?/set this message is thrown. If the value of the parameter changes someone is writing to it. if homematic confirms the message than it looks to me that this could cause the loop.

• check if you really write to /set
• Switch off auto confirm if Homematic confirms the input
• Disable the homie node and check the behavior of your flow with mqtt explorer. Perhaps disable the homie config too
• Use the homie node instead of mqtt nodes
• stop yor flow, erase the homie node on your broker using mqtt explorer and then start

Last but not least perhaps it is a bug

[sognen]

Hi, Thank you, with the help of your response I believe I now got it sorted. The flip flop behavior described above appeared to be caused by a misconfigured MQTT broker as well as the method I used for clearing the broker did not fully work causing residual values to reappear. I also had to disable and then re-enable passthrough to make it work.

Thanks again for your help, and the node itself.

Cato

[Christian-Me]

@sognen Good to hear that it is working. There is a big rework of the homie node(s) in the pipeline. That's why there are no updates over the last year (?)

[sognen]

Hi, That is good news. Does that mean that you are accepting feature requests ;)

Unfortunately, the flip-flop issue is back with vengeance. Its been challenging to debug due to its erratic nature; it would sometimes occur when reloading flows, and not always on the same topics. This led me to believe that it may have something to do with the rapid updates that sometimes occur when flows are started. For example, it may cycle through a flow assuming that you are "away", then it triggers again as it receives information that you are "home", and then cyle again as it figures out the appropriate "scene", presence in the room, and all other conditions found the flows.

I verified this by creating a new homie node and two inject nodes. When rapidly submitting two different values to topic/set it only takes a few seconds for the "update form outside detected" to be triggered. I did not test if the same occurs updates are submitted through your node.

On the bright side, it provides some encouragement towards cleaner and more economical flows. For now, I have inserted a rate limiter before each mqtt-out node to restrict the rate of updates sent to topic/set. Still, I imagine that there may be situations where this could occur organically, for example, the system could active "away", and at the same time, I request google assistant to active "home" scene. Publishing to topic (without set) also works as it does not trigger the "update form outside detected" message.

I am conscious that using mqtt-nodes which may be well outside how you intended your node to be used, but I wanted to share the above in case it is relevant for your work on the new version.

When ready, happy to help out with beta testing the new version if useful.

Cato

[Christian-Me]

Hi,

Strange behavior. In the new version I removed the "supervisor" feature. I never used homie with the mqtt node. I always use the homie device node even for pure Node-RED nodes. These are totally rewritten now providing a separate client connection for each "virtual" device so LWT for all devices work if Node-RED shuts down unexpectedly. My problem currently is that the new homie nodes will probably be a breaking change. So either do a version 2 or live with that. Im still unsure. Feature request - always. But I think I covered everything. Even hacks of the convention and custom extensions are possible. Auto detection of homie root topics and own root topics are also implemented. Give me a few days and I will push the new version on github. There is still a nasty bug I have to fix first. If you like you can send me your flow to take a look.

[sognen]

Hi, I am excited about the new version.

I have attached the flow that demonstrates the "update form outside detected" warning when rapidly submitting values. It contains your homie node, two inject nodes and a single MQTT node.

Steps to reproduce "update form outside detected" warning

1. After deployment, restart your flows
2. Click rapidly on the inject button named "hot - click here fast"

After a few seconds, you should see the warning.

Flow: [{"id":"6ae4d276.60d81c","type":"homie-convention-node","z":"a7fefaea.2efa78","broker":"e6a23911.f60378","name":"hotcold","nodeId":"testNode","nodeName":"A test Node","nodeType":"Node-RED homie-node","passMsg":true,"autoConfirm":true,"infoHomie":false,"infoError":false,"exposeHomieNodes":false,"homieNodeConfig":"{\n \"homie\": {\n \"$name\": \"testNode\",\n \"$type\": \"A test Node\",\n \"$properties\": {\n \"hotcold\":{\n \"$name\": \"hotcold\",\n \"$datatype\": \"string\",\n \"$settable\": true,\n \"$retained\": true\n } \n }\n }\n}","x":570,"y":180,"wires":[["f32f9143.07083"]]},{"id":"a1880f35.06dd8","type":"mqtt out","z":"a7fefaea.2efa78","name":"","topic":"homie/Node-RED/testNode/hotcold/set","qos":"1","retain":"false","broker":"887f143b.dd5858","x":930,"y":240,"wires":[]},{"id":"1377c137.a070ef","type":"inject","z":"a7fefaea.2efa78","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"1","crontab":"","once":true,"onceDelay":0.1,"topic":"","payload":"cold","payloadType":"str","x":580,"y":240,"wires":[["a1880f35.06dd8"]]},{"id":"ab897e56.ee8c2","type":"inject","z":"a7fefaea.2efa78","name":"hot - click here fast","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"hot","payloadType":"str","x":600,"y":300,"wires":[["a1880f35.06dd8"]]},{"id":"f32f9143.07083","type":"debug","z":"a7fefaea.2efa78","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":800,"y":180,"wires":[]},{"id":"e6a23911.f60378","type":"homie-convention-broker-config","mqtt-host":"192.168.1.104","mqtt-port":"1883","tls":"224962c.03df19e","usetls":false,"name":"localhost@hp-spectre-360","homieName":"Node-RED","homieFriendlyName":"Node-RED","homieRoot":"homie","storeGlobal":false},{"id":"887f143b.dd5858","type":"mqtt-broker","name":"MQTT","broker":"192.168.1.104","port":"1883","clientid":"nodered","usetls":false,"compatmode":false,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"1","birthRetain":"false","birthPayload":"","closeTopic":"","closeQos":"1","closeRetain":"false","closePayload":"","willTopic":"","willQos":"1","willRetain":"false","willPayload":""},{"id":"224962c.03df19e","type":"tls-config","name":"Localhost ca.crt","cert":"","key":"","ca":"","certname":"","keyname":"","caname":"ca.crt","servername":"ah-berlin-0148","verifyservercert":false}]

[Christian-Me]

It would be interesting for which kind of sensors / actors you are using homie. For iot sensors the update frequency normally is not above 1hz. Ok perhaps a slider setting a dimmer. But for this i have a solution. The new homie node will provide a dynamic limiter. No new message is issued until the last is confirmed. All in between messages are dropped. The most recent is kept and sent as soon as the acknowledgment arrives. So you can automatically get the highest possible bandwidth.

[Christian-Me]

Ok, had a test ... and yes I could reproduce that problem. It is caused by the way the detection is done. If a messages on the /set topic is received it sets a "predicted" flag. It is expected that the confirmation confirm the "original" payload. Until then everything else is "detected". If the message is confirmed on the property topic the prediced falg is removed.

As it is possible that messages arrives "not in order" especially in heavy load (I tried sending every 10ms) this could fail. And Node-RED itself could cause delays in the stream.

you could comment this part in homie-node.js line 333:

                // reject all property changes form outside except the first time
                if (homieProperty && homieProperty.hasOwnProperty("value") && homieProperty.value!==null) {
                  if (homieProperty.hasOwnProperty("valuePredicted") && 
                      node.broker.formatValueToString(homieProperty.$datatype,homieProperty.$format,homieProperty.valuePredicted)!=originalPayload) {
                    node.status({fill: 'yellow', shape: 'dot', text: propertyId+' changed externally! rejected'});
                    node.addToLog("warn",nodeId+'/'+propertyId+'='+originalPayload+" update form outside detected. Original value restored! ("+homieProperty.value.toString()+")");
                    node.mqttPublish(node.broker.baseTopic + '/' + nodeId + '/' + propertyId,
                      node.broker.formatValueToString(homieProperty.$datatype,homieProperty.$format,homieProperty.value));                  
                  } else {
                    updateProperty();
                  }
                } else {
                  updateProperty();
                }

and replace it by a singe updateProeprty();

                // reject all property changes form outside except the first time
                /* comment start
                if (homieProperty && homieProperty.hasOwnProperty("value") && homieProperty.value!==null) {
                  if (homieProperty.hasOwnProperty("valuePredicted") && 
                      node.broker.formatValueToString(homieProperty.$datatype,homieProperty.$format,homieProperty.valuePredicted)!=originalPayload) {
                    node.status({fill: 'yellow', shape: 'dot', text: propertyId+' changed externally! rejected'});
                    node.addToLog("warn",nodeId+'/'+propertyId+'='+originalPayload+" update form outside detected. Original value restored! ("+homieProperty.value.toString()+")");
                    node.mqttPublish(node.broker.baseTopic + '/' + nodeId + '/' + propertyId,
                      node.broker.formatValueToString(homieProperty.$datatype,homieProperty.$format,homieProperty.value));                  
                  } else {
                    updateProperty();
                  }
                } else {
                  updateProperty();
                }
                comment end */
                updateProperty(); // this is new!

MQTT is not made for high frequencies and the queue can fill up and messages can be dropped. My queue is still sending messages as I write this ;) It is done totally different in the new version.

[sognen]

Thanks for the code update. In order to circumvent the issue I have

• removed retain where possible,
• revised my flow to be more economical, less updates and
• delayed execution scenes of various scenes when starting up/reloading flows.

After making these changes I no longer see the warning. I believe there were just too many updates coming in too quickly. If I see it resurface I will try the code change.

When it comes to inputs that have frequent updates, in my case audio volume changes, I have inserted a debouncer node before the MQTT node.

With the above, I am now a very happy user of the homie nodes and will update my flows and nodes when the new version comes out.

I want to say thank you for your help and assistance in solving this issue, much appreciated.

Cato

[Christian-Me]

Hi Cato, Still I have some questions and ideas

• "removed retain where possible": Retain is an essential part of the homie mechanism and should be on per default and off if necessary (not the other way around). Only values valid when received LIVE should be non retained. Your heating example should always retain as you want new subscribers informed of the actual state and even the control device itself should reconstruct the desired state if it restarts (i.e. my switch plugs always recover their state on/off after I moved them to a new location simply by re-subscribing on reboot and getting the last /set message. Or think of a sleeping device reacting on a /set change after wake up. This way sleeping actors can be done if you don`t need instant reactions.
• less updates are always good. No need to send state updates as the last change should be retained no updates are necessary, only on state change or value change by a significant amount (threshold)
• Delays should not be necessary as the state is stored on the broker. On Node-Red restart all values should be restored. Currently I have 2.336 topics on my broker with no issues. If you calculate 10 topics per property you come to around 250 values.

still my question ... why so high frequency state changes? Especial for a heater. My pressure sensors on my pup do fast updates triggered by a threshold value up do every 0.1s but all the rest are updates around every minute/hour or better only on significant value change.

Will keep you in the loop