Change Lock entity to Unavailable when MQTT is disconnected

[leranp]

Hi, Love this component, but there is something that i think that needs to be changed, when the MQTT is disconnected for some reason, the lock entity still shows status that can be worng. In all of my MQTT base devices, when the MQTT is not connected, the device goes to Unavailable State

[technyon]

It wouldn't make sense to change the lock state if MQTT is disconnected. The lock doesn't necessarily change state just because MQTT is disconnected.

What you want to monitor is the "maintenance/mqttConnectionState" node, which reflects wether or not the ESP is connected ... which reminds me I'll have to add this to the readme.

[mundschenk-at]

I disagree, in HA that absolutely would make sense. When the ESP is disconnected from MQTT, the lock state is unavailable. Adding it to auto-discovery should be pretty straightforward, I can try to come up with something.

(You are not changing the lock state topic, just the way HA parses that into an entity.)

[technyon]

ok, understood. However @leranp didn't mention this is HA related.

[mundschenk-at]

Oh, you are right, I just assumed because of the terminology. @leranp, you are asking about Home Assistant, right?

[leranp]

Yes, i am talking about HA

[technyon]

@mundschenk-at Let's update the auto discovery then. We can release this together with the fix for the auto discovery warnings.

[mundschenk-at]

Sorry for the delay, I'll do the write-up this weekend.

[technyon]

@mundschenk-at ok

[mundschenk-at]

So in YAML you'd need to add this to auto-discovery:

      availability:
        - topic: <prefix>/maintenance/mqttConnectionState

No need to set the payloads, as they default to online and offline anyway. So for the JSON payload, it would be:

"avty": [{"t": "~/maintenance/mqttConnectionState"}]

I am using the availability syntax here because it would allow us to add several availability topics, e.g. not only for the MQTT connection but also for the BLE connection - the state of the lock is only accurate if there was recent-ish BLE connection and MQTT is connected. You could set such a topic to offline when a BLE restart is necessary. That would make such interruptions visible in the HA logbook.

[mundschenk-at]

Anyway, it was just a thought, probably too niche a use-case. The MQTT availability topic should probably be added to all auto-discovery entities except the MQTT connected state.

[technyon]

I've added availability to the json as suggested. Please check and confirm that it works. So far only for the lock, I'd added it to the other entities if it's confirmed it works for the lock.

nuki_hub-8.27-pre-2.zip

[mundschenk-at]

I tried it, works fine for the lock, but It breaks the opener because the connection state is not available under the opener prefix. Not sure if you can separate them with inheritance?

I do have a suggestion how to handle this (with my long-promised write-up of a better auto-discovery device split), but that probably needs some discussion.

[technyon]

It should work the same way for the opener, shouldn't it? The easiest fix would be to duplicate the connection state topic under the opener node.

[mundschenk-at]

This would be the quick fix, but it is kind of dirty. Or you could simply reference the mqttConnectionState under the lock prefix.

IMHO nicer would be to move all the Nuki Hub/ESP-specific topics to their own HA "hub" device (and maybe MQTT topic, but that would be a breaking change of course) and have the Lock and Opener be connected devices to that "hub". (I'd have to look into it whether we would then still need the separate availability topic in the Lock and Opener device definitions, maybe we do.)

[technyon]

All paths are relative right? Should be easy enough to point to the lock maintenance topic.

[technyon]

Please check if it works like this.

nuki_hub-8.27-pre-3.zip

[leranp]

Please check if it works like this.

nuki_hub-8.27-pre-3.zip

Looks fine for me

[technyon]

@mundschenk-at Does it fix the issue with the opener?

[mundschenk-at]

@mundschenk-at Does it fix the issue with the opener?

Unfortunately, no. I missed your question about paths being relative, to which the answer would be no (or rather: MQTT topics are not paths that can be navigated). So

"availability": {
  "topic": "nukiopener/../nuki/maintenance/mqttConnectionState"
}

is incorrect, it would need to be

"availability": {
  "topic": "nuki/maintenance/mqttConnectionState"
}

[technyon]

@mundschenk-at Give this a try

nuki_hub-8.27-pre-4.zip

[mundschenk-at]

Sorry I wasn't more explicit yesterday, the new version works fine.

[technyon]

Check release 8.27

[bcutter]

OK I updated to 8.27 some days ago (same day it became available). Now that I restarted HA for the first time since then, all my locks are shown as unavailable, even they are reachable and at the same time the MQTT connected entities show on.

Not sure what screwed things up - I've NEVER ever seen the lock entities in unavailable state before the 8.27 update. Will try to downgrade one Nuki hub to 8.26 for now and see if that changes things.

Edit:

• Yes, immediately after downgrading 8.27 to 8.26 the lock entity became available again.
• So something's not right with the 8.27 - or my Home Assistant. Downgraded the other one too now (same behaviour: 8.26 in place - lock entity immediately available) as I need both functional of course. So I'm out for now for further testing.
• Summary: Restarting HA without any lock action (not sure if local button action would have triggered something to make the lock entities state switch from unavailable to locked or unlocked) leads to unavailable lock entities although MQTT is connected.

[technyon]

What path did you configure for the lock? And could you share the content of the HA discovery topic for the lock?

[bcutter]

Could you please guide me precisely on how to get that information?

Afaik I never changed/customized things so everything should be default.

[technyon]

As a start, maybe post the content of the system information page.

Which MQTT broker are you using?

[mundschenk-at]

Could you please guide me precisely on how to get that information?

In HA: Settings > Devices & Services > MQTT > Your lock device > 3 dots > Download diagnostics

[bcutter]

System information page (masked)

NUKI Hub version: 8.26
run: true
deviceId: XXXXXXXXXXX
deviceIdOp: XXXXXXXXXXX
mqttbroker: xxx.xxx.xxx.xxx
mqttport: 1883
mqttuser: ***
mqttpass: ***
mqttlog: false
lockena: true
mqttpath: nuki_doorone
openerena: false
mqttoppath: 
maxkpad: 5
opmaxkpad: 
mqttca: 
mqttcrt: 
mqttkey: 
hassdiscovery: homeassistant
dhcpena: true
ipaddr: xxx.xxx.xxx.xxx
ipsub: 255.255.255.0
ipgtw: xxx.xxx.xxx.xxx
dnssrv: xxx.xxx.xxx.xxx
nwhw: 1
rssipb: -1
hostname: ESP-Nuki-Hub-DoorOne
nettmout: -1
restdisc: true
rstbcn: 60
lockStInterval: 3600
configInterval: 3600
batInterval: 7200
kpInterval: 1800
kpEnabled: true
accLvl: 
regAsApp: false
nrRetry: 5
rtryDelay: 100
crdusr: ***
crdpass: ***
pubauth: true
pubdbg: false
prdtimeout: 60
hasmac: false
macb0: 
macb1: 
macb2: 
MQTT connected: Yes
Lock firmware version: 3.7.7
Lock hardware version: 5.11
Lock paired: Yes
Lock PIN set: Yes
Lock has door sensor: No
Lock has keypad: Yes
Network device: Built-in Wifi
Uptime: 1088 minutes
Heap: 85424
Stack watermarks: nw: 6104, nuki: 360, pd: 248
Restart reason FW: RestartOnDisconnectWatchdog
Restart reason ESP: ESP_RST_SW: Software reset via esp_restart.

As the MQTT -> lock -> burger menu -> diagnostics output is very very chatty, here's what I think you might have asked for:

        {
          "entity_id": "lock.schloss_doorone",
          "subscriptions": [
            {
              "topic": "nuki_doorone/lock/binaryState",
              "messages": [
               ...................................
                {
                  "payload": "locked",
                  "qos": 0,
                  "retain": 0,
                  "time": "2023-10-24T06:18:06.074445+00:00",
                  "topic": "nuki_doorone/lock/binaryState"
                },
                {
                  "payload": "unlocked",
                  "qos": 0,
                  "retain": 0,
                  "time": "2023-10-24T06:29:16.264524+00:00",
                  "topic": "nuki_doorone/lock/binaryState"
                }
              ]
            }
          ],

Here's the /homeassistant/lock/XXXXXXXXX/smartlock mqtt auto discovery topic:

{"dev":{"ids":["nuki_XXXXXXXX"],"mf":"Nuki","mdl":"SmartLock"},"~":"nuki_doortwo","name":"Door Two","unique_id":"XXXXXXXX_lock","cmd_t":"~/lock/action","pl_lock":"lock","pl_unlk":"unlock","pl_open":"unlatch","stat_t":"~/lock/binaryState","stat_locked":"locked","stat_unlocked":"unlocked","opt":"false"}

Hope that helps.

[mundschenk-at]

No, the discovery topic should be something like homeassistant/lock/<lockid>/<entityname>/config (under the key discovery_data). However, I've just checked mine and there is some corruption going on. The lock entity shows an incorrect discover topic (~/maintenance/mqttConnectionState). However, this looks like a display issue with the HA MQTT integration. The actual discovery topic still exists on my broker.

[bcutter]

One picture maybe tells more than 1.000 words. Here's once again the current status:

[technyon]

Could you post the whole content of that topic?

[bcutter]

I did so already for one of the two locks, see at the bottom of this post https://github.com/technyon/nuki_hub/issues/227#issuecomment-1777984588

[technyon]

It looks good, but it lacks the availability ("avty") part. Is this from version 8.26? It would be important to know where this points. For you it should direct to "nuki_doortwo/maintenance/mqttConnectionState". If that path is correct, the next thing to check would be if that topic exists and is updated correctly ("online" / "offline").

I wonder if the underscore in the name could cause problems (it should not).

[bcutter]

Yes everything I provided is from Nuki Hub 8.26. I don't see "avty". Where exactly should I look at?

Yes, this is as expected:

Existing and updating regularly. On 8.26 (⚠️).

Not sure what else I could provide...

[technyon]

That's what has been added with 8.27, so you won't see it with 8.26

[bcutter]

OK. So I guess I can't add something helpful, do I?

[technyon]

Well... you could update to 8.27 and copy/paste what's there, then go back to 8.26

[bcutter]

Updated one Nuki Hub again to 8.27. autodiscovery topic (config)... ...now has:

And that one still exists of course:

So I assume the avty is not available or not correct or something else during HA restart. Otherwise I can't imagine why the lock entity renders unavailable (which for the records it did not during the short update 8.26-8.27 as I did not restart HA).

Let's compare auto discovery topic: 8.26:

{"dev":{"ids":["nuki_XXXXXXXX"],"mf":"Nuki","mdl":"SmartLock"},"~":"nuki_doortwo","name":"Door Two","unique_id":"XXXXXXXX_lock","cmd_t":"~/lock/action","pl_lock":"lock","pl_unlk":"unlock","pl_open":"unlatch","stat_t":"~/lock/binaryState","stat_locked":"locked","stat_unlocked":"unlocked","opt":"false"}

8.27:

{"dev":{"ids":["nuki_XXXXXXXX"],"mf":"Nuki","mdl":"SmartLock","name":"Door Two"},"~":"nuki_doortwo","name":null,"unique_id":"XXXXXXXX_lock","cmd_t":"~/lock/action","avty":{"t":"~/maintenance/mqttConnectionState"},"pl_lock":"lock","pl_unlk":"unlock","pl_open":"unlatch","stat_t":"~/lock/binaryState","stat_locked":"locked","stat_unlocked":"unlocked","opt":"false"}

Comparing both I only discovered that a) name exists two times now with 8.27, b) first name has another position but is correct and c) second name is on the same place as with 8.26 but is null (is that suspicious or relevant at all?)

[mundschenk-at]

That's on purpose due to how recent HA versions handle device vs. entity names.

[bcutter]

OK. So could that be a definite issue, running an older HA version?

[mundschenk-at]

Normally <prefix>/maintenance/mqttConnectionState will only change to offline if the hub is disconnected from the MQTT broker (by pulling power for example). A reboot of the ESP will not change the topic to offline (probably because it is too fast for the session timeout).

[mundschenk-at]

OK. So could that be a definite issue, running an older HA version?

Which version are you running? But no, auto-discovery itself has not changed and the name setting only affects newly created entities, so that should probably not be the issue.

[bcutter]

Normally <prefix>/maintenance/mqttConnectionState will only change to offline if the hub is disconnected from the MQTT broker (by pulling power for example). A reboot of the ESP will not change the topic to offline (probably because it is too fast for the session timeout).

Yes, but as initially described here https://github.com/technyon/nuki_hub/issues/227#issuecomment-1773373364 with

all my locks are shown as unavailable, even they are reachable and at the same time the MQTT connected entities show on.

that's not the issue. MQTT connection state completely fine, but lock entities unavailable.

Which version are you running?

Im running HA Core 2023.3.6 at the moment.

So now it's up to you to figure out a) why this happens and b) if/why only for me (what's the update pace, how often has 8.27 been downloaded, how many of those use HA etc.)

I don't see what else I could provide. 8.26 everything's running fine, 8.27 screws up lock entities once HA restarts. No idea what you changed in detail in the 8.27 code.

[technyon]

From the issues received here, I'd say the majority of users use HA. The code that changed in details doesn't matter all that much, what's important is what's in MQTT. "avty" was added so HA knows when the ESP goes offline. Why it doesn't work in your case is somewhat of a mystery, since everything looks fine.

We can try to figure out what's different in your configuration. What MQTT broker are you using, maybe a specific broker causes problems. Also you have an underscore in the path ... that shouldn't matter but then it could. Maybe as a test change the path to not use an underscore.

[bcutter]

I'm just using the default Mosquitto broker shipped as addon for HA (https://github.com/home-assistant/addons/tree/master/mosquitto).

Well, instead of ruining my productive system and making my other household members sharpen their knifes, I hope you experts have the possibility of a test system. So you could just add a (very very VERY likely unproblematic _) to one of your locks and give it a go.

[mundschenk-at]

Well, you might also want to update your HA version to something more recent. That should not be the issue, but as is, there are a lot of variables here. You can't expect people here to reproduce your exact system.

[bcutter]

Well, you might also want to update your HA version to something more recent. That should not be the issue, but as is, there are a lot of variables here.

Yes, I will update HA once there's the needed time. I also don't think this will make a difference as I checked all breaking changes in all monthly release notes between 2023.3 and 2023.10 - nothing in MQTT domain which could have an impact on this way it renders (lock) entities unavailable as false positive.

You can't expect people here to reproduce your exact system.

No of course not. But one can the pros have more test possibilities and rule out low-level assumptions like a certain character in a name breaking things :-)

[bcutter]

Small note: found something in HA 2023.10 release notes which sounds kinda interesting:

https://www.home-assistant.io/blog/2023/10/04/release-202310/#breaking-changes

An MQTT lock with a configured state topic will initialize with state unknown instead of state unlocked unless the lock is set to optimistic mode.

MQTT locks set to optimistic mode will still be initialized with the unlocked state. You should check if your automations are affected.

While I'm not 100 % sure what the detailed impact of this change on lock entities in HA is, it's impossible my issue is caused by this - as I'm still few release cycles behind that 2023.10 update. Maybe you want do take a look at this change anyway in case you would need to configure something.

[technyon]

I'm not really sure how to reproduce this. Maybe updating your system first would make sense so we can eliminate at leasta one variable. Other than that, we could think about a way to disable publishing avty.

[bcutter]

Going through the updates, but it takes some time, probably weeks. 2023.4 made no difference so far (and tbh I doubt it will be fixed with a HA update).

So annoying when updates break (core) things... 8.26 was running so smooth and stable. I get the "want have" factor for having lock entities unavailable depending on the MQTT connection state, but well... not working.

Maybe we can have this avty thing as config option? So being on as default this way everyone could simply decide on his/her own.