MQTT Discovery payload incorrect.

[YogoGit]

Checklist

• [X] I am not using Home Assistant. Or: a developer has told me to come here.
• [X] I have checked the troubleshooting section and my problem is not described there.
• [X] I have read the changelog and my problem is not mentioned there.

Deploy method

Docker

Z-Wave JS UI version

9.9.1

ZwaveJS version

12.4.4

Describe the bug

The generated MQTT discovery (I'm NOT using the built-in device integration as my ZWave devices are over a hundred miles away from my HA server) is incorrect, and was not updated to follow recent HA MQTT discovery changes (> 6 months ago).

https://developers.home-assistant.io/blog/2023/06/28/entity-name-changes?_highlight=changing

This occurs for ALL of my zwave devices, causing errors similar to the following to litter my logs.

2024-03-19 11:19:21.657 WARNING (MainThread) [homeassistant.components.mqtt.mixins] MQTT entity name starts with the device name in your config {'payload_off': '0', 'payload_on': '1', 'value_template': Template<template=({{ value_json.value }}) renders=0>, 'command_topic': 'zwave-seeleyhouse/seeleyhouse/south-deck_dining/configuration/endpoint_0/param026/set', 'enabled_by_default': False, 'entity_category': <EntityCategory.CONFIG: 'config'>, 'state_topic': 'zwave-seeleyhouse/seeleyhouse/south-deck_dining/configuration/endpoint_0/param026', 'availability': [{'payload_available': 'true', 'payload_not_available': 'false', 'topic': 'zwave-seeleyhouse/seeleyhouse/south-deck_dining/status', 'value_template': Template<template=({{'true' if value_json.value else 'false'}}) renders=0>}, {'topic': 'zwave-seeleyhouse/_CLIENTS/ZWAVE_GATEWAY-zwave-js-ui/status', 'value_template': Template<template=({{'online' if value_json.value else 'offline'}}) renders=0>, 'payload_not_available': 'offline', 'payload_available': 'online'}, {'payload_available': 'true', 'payload_not_available': 'false', 'topic': 'zwave-seeleyhouse/driver/status'}], 'availability_mode': 'all', 'device': {'identifiers': ['zwavejs2mqtt_0xc37a038c_node10'], 'manufacturer': 'Zooz', 'model': 'Dimmer Switch (ZEN72 800LR)', 'name': 'seeleyhouse-south-deck_dining', 'sw_version': '3.30.0', 'connections': []}, 'name': 'seeleyhouse-south-deck_dining_config_switch_26', 'unique_id': 'zwavejs2mqtt_0xc37a038c_10-112-0-26', 'payload_available': 'online', 'encoding': 'utf-8', 'optimistic': False, 'qos': 0, 'retain': False, 'payload_not_available': 'offline'}, this is not expected. Please correct your configuration. The device name prefix will be stripped off the entity name and becomes '_config_switch_26'

The above is the error from my Zooz Zen72 800LR, although I have similar errors for my other switches, thermostats (non-Zooz), etc...

Here is the actual payload

{
  "value_template": "{{ value_json.value }}",
  "command_topic": "zwave-seeleyhouse/seeleyhouse/hall_living/configuration/endpoint_0/Custom_Brightness_Level/set",
  "enabled_by_default": false,
  "entity_category": "config",
  "min": 0,
  "max": 99,
  "state_topic": "zwave-seeleyhouse/seeleyhouse/hall_living/configuration/endpoint_0/Custom_Brightness_Level",
  "availability": [
    {
      "payload_available": "true",
      "payload_not_available": "false",
      "topic": "zwave-seeleyhouse/seeleyhouse/hall_living/status",
      "value_template": "{{'true' if value_json.value else 'false'}}"
    },
    {
      "topic": "zwave-seeleyhouse/_CLIENTS/ZWAVE_GATEWAY-zwave-js-ui/status",
      "value_template": "{{'online' if value_json.value else 'offline'}}"
    },
    {
      "payload_available": "true",
      "payload_not_available": "false",
      "topic": "zwave-seeleyhouse/driver/status"
    }
  ],
  "availability_mode": "all",
  "device": {
    "identifiers": [
      "zwavejs2mqtt_0xc37a038c_node8"
    ],
    "manufacturer": "Zooz",
    "model": "Dimmer Switch (ZEN72 800LR)",
    "name": "seeleyhouse-hall_living",
    "sw_version": "3.30.0"
  },
  "name": "seeleyhouse-hall_living_config_number_18",
  "unique_id": "zwavejs2mqtt_0xc37a038c_8-112-0-18"
}

It is my understanding that the "name" field above should be either NOT be set, or set to null (from the error logs). I can manually fix ALL of my devices but modifying the configuration for all my devices (hundreds of messages) and disabling MQTT discovery, but IMO this really should be fixed at the zwave-js-ui (or zwave-js server and/or node-zwave-js).

Otherwise, everytime I add a new device, I have to re-enable MQTT discovery (to recognize the new device) and then manually fix all the messages again, or manually codify new MQTT discovery messages.

To Reproduce

Add a Zwave device to zwave-js-ui after disabling WS-Server and enabling MQTT discovery.

Expected behavior

A valid MQTT discovery payload would be sent for new devices, with the proper value in the 'name' field.

Additional context

As mentioned, my ZWave server is located 100 miles away from my HA server, so using MQTT is a MUCH lower-cost (and lower bandwidth) solution. In addition, the use of MQTT allows me to do additional intergrations that are not possible with the WS-Server coupling directly to HA.

Export of the Zen72 is provided below as described in the bug_report URL.

node_10.json

[robertsLando]

Hi @YogoGit , I'm wondering if setting the name of the entity to null (or not set it at all) can cause breaking changes on other users. Specifically now that MQTT discovery is also used by Domoticz and OpenHAB users.

@ccutrer any clue?

[ccutrer]

I know the change well... it was quite disruptive- both as a maintainer of MQTT integrations that Home Assistant users use, and as a maintainer of openHABs integration with Home Assistant style MQTT discovery. Essentially, @YogoGit is correct- a change needs to be made. openHAB is perfectly happy with the component name being null now. I can't speak for Domoticz.

However, it's not actually necessary to have null component names - you just can't have the device name be a prefix of the component name. This is simply accomplished with zwave-js-ui, since you can just always include some part of the valueId (or HASS objectid) in the entity name template. So there's no need to consider if Domoticz can actually handle null component names. In fact I've already done so with my instance -- and without a code change. I've simply changed my "Entity name template" to %pn instead of the default of `%ln%o`.

I'm not sure how configuration works, @robertsLando. If we change the default value of the entity name template, would existing installs get that change? Or no, because they would have already persisted whatever they currently have? If they get the new default, then fixing this would be a breaking change. But technically it's something that should be done, to stay "in spec" (🙄 given there is no actual spec) with current Home Assistant versions. And even as a breaking change, it's not difficult for users to "roll back" the change in their own instance by setting the entity name template back to the old default. Perhaps there could be a migration for new installs to get the new default, but old installs to persist the old default, so they can migrate in their own time?

[YogoGit]

@ccutrer - Are you saying that I could modify the code in the docker container (which is what I'm using) to change the default for the entity template, restart the container, and I could verify that it would correct the issue?

In /usr/src/app/.server/lib/Gateway.js, near line 2027:

Stock..

 _getEntityName(node, valueId, cfg, entityTemplate, ignoreLoc) {                
     entityTemplate = entityTemplate || '%ln_%o';                               
     // when getting the entity name of a device use node props                 

Modified:

 _getEntityName(node, valueId, cfg, entityTemplate, ignoreLoc) {                
     entityTemplate = entityTemplate || '%pn';                               
     // when getting the entity name of a device use node props                 

If so, I can try making this change on my device this coming weekend (when I travel to the remote site) and see if it breaks anything on my HA system.

Also, in a COMPLETELY unrelated topic I just noticed, but the directory /us/src/app/node_modules is populated in the docker container, and in my past life (software engineer for > 35 years) we never distributed node_modules inside our containers, as the generated file were created from the libraries prior to creating the final artifact (and weren't always used), so the contents of node_modules was just wasted space in the containers.

Also, if you are aware of a way for me to change it to something else (or null), I am comfortable modifying code inside the container or attempting other changes to help debug this.

Thanks for your attention to this issue!

[ccutrer]

I'm saying you don't even need to modify any code! Go to /#/settings in your zwave-js-ui instance, scroll down to Home Assistant, and modify the Entity name template. I think you'll need to re-discover all your devices after that to get the new setting? Example screenshot of my config:

in my past life (software engineer for > 35 years)

Lulz, containers haven't existed for 35 years! I know I know, that's not what you meant, it just sounded funny. I can't speak to including the node_modules in the image, though - I'm not a JS/TS guy.

[robertsLando]

If we change the default value of the entity name template, would existing installs get that change?

Yes so that would be breaking but considering all home assistant users should never use MQTT Discovery (as there is Zwave integration) and this is a very users specific case I think your workaround may be enough in this case for the ones that wants to fix that.

About the node_modules thing, I dunno what you mean @YogoGit, this is how all nodejs projects are usually shipped with docker. Maybe you meant that node_modules could have been dropped and create an entrypoint that installs them on startup if missing? Could be a solution but the space advantages IMO aren't worth the additional startup time.

[YogoGit]

About the node_modules thing, I dunno what you mean @YogoGit, this is how all nodejs projects are usually shipped with docker. Maybe you meant that node_modules could have been dropped and create an entrypoint that installs them on startup if missing? Could be a solution but the space advantages IMO aren't worth the additional startup time.

I'm pretty sure that's not the case, at least in my experience. Generally, node_modules are excluded (.dockerignore) as node_modules is not used at runtime, and node_modules is almost always MUCH (!) larger than the resulting runtime dependencies. Again, I've created dozens of projects that use node as containers, and none of them have node_modules included in them.

See https://www.digitalocean.com/community/tutorials/how-to-build-a-node-js-application-with-docker

W/regards to size, in the docker container, node_modules contains 244/265 MB of files in /usr/src/app, so it's usage is > 90% of the entire image size. Without it, the image would be MUCH smaller.

zwavejs/zwave-js-ui               9.9.1     14221e682b6d   2 weeks ago    275MB

Removing 244MB would make the image much smaller. (I'll do some investigating to see if it works correctly, and if so create a PR once that changes).

[YogoGit]

Yes so that would be breaking but considering all home assistant users should never use MQTT Discovery (as there is Zwave integration) and this is a very users specific case I think your workaround may be enough in this case for the ones that wants to fix that.

Wow, I wouldn't got that far. I would venture to guess that a majority of the HA users don't use MQTT discovery, but I would guess that a measurable percentage (> 10%) of them probably use MQTT discovery instead of the WS server. And, I appreciate that it's supported, and am asking that this supported feature actually be setup to work properly, given that the setup is in the HA section, and that it doesn't work properly with HA.

Given that it's currently broken, I would think that changing the template (as @ccutrer) mentioned wouldn't break any existing HA users as there setup is currently broken. However, at least I have a workaround, which I will be testing this weekend.

[YogoGit]

@ccutrer - Thanks for the great instructions, I will try this out this coming weekend and update this issue with my results!

[robertsLando]

Generally, node_modules are excluded (.dockerignore) as node_modules is not used at runtime

Oh yeah now I understand what you mean, the reason why I removed them from dockerignore is to skip install inside docker build because of a bad bug that made me lost more then a week (was something related to docker alpine).

Anyway It's not really clear to me why the image should be smaller then as anyway them would have been generated by the build process and then shipped within the image so the result is the same.

[YogoGit]

Update. I modified the entity name template, first with %pn, and then later with ' ', and re-ran discovery. All of my entities appear to be working fine, although I won't know for sure until I add a new device, as I believe it cached some of the old entity_ids.

But, so far so good, and my HA logs are MUCH quieter than before.

[robertsLando]

@YogoGit Thanks for your feedback 🙏🏼 Closing this for now, feel free to re-open in case

[Turbox35]

Hello. I don't want to reopen the issue as @YogoGit shows how to solve. I'm a dual domotic user (Domoticz and HA) and use MQTT for centralised data (zwave, zigbee, enphase, nodemcu...) and since my last update of zwavejsui (ok I had some month late) I run the same issue on Home Assistant side and no issue with Domoticz. I modified the Entity name template as suggested and all devices were recovered. But yesterday I applied for the last Home Assistant update 2024.4 and then I lost half of my zwave device on HA again while it works fine on Domoticz. I remodify the Entity name template and they come back again. So this workaround seems not permanent and it's very annoying.

[robertsLando]

I remodify the Entity name template and they come back again.

It's not clear to me when you say it is not permanent, you mean that zwave-js ui updates don't persist the entity name template or what else? Maybe after the home assistant update you only had to restart zui in order to retrigger the discovery?

[Turbox35]

I remodify the Entity name template and they come back again.

It's not clear to me when you say it is not permanent, you mean that zwave-js ui updates don't persist the entity name template or what else? Maybe after the home assistant update you only had to restart zui in order to retrigger the discovery?

Thanks for your message and sorry for the misunderstanding. On the Zwave-js UI side the parameters of the entity name template persist well (not tested with a restart) but I have modified first with %pn, and then later with ' ' which is for me a come back to the origin. In other words I did this on my last update of Zwave-js UI container and again after the HA update. After the HA update, I didn't try to just restart Zwave-js ui, perhaps it did the job. I will do that if the issue come back.

[robertsLando]

After the HA update, I didn't try to just restart Zwave-js ui, perhaps it did the job. I will do that if the issue come back.

Z-UI listens for Home assistant birth topic in order to detect if/when Home assistatn restarts and in that case it does a re-discovery of all devices. Maybe you didn't configured the birth topic well?

From docs:

In your Home Assistant configuration.yaml:

mqtt:
  discovery: true
  discovery_prefix: <your_discovery_prefix>
  broker: [YOUR MQTT BROKER] # Remove if you want to use builtin-in MQTT broker
  birth_message:
    topic: 'homeassistant/status'
    payload: 'online'
  will_message:
    topic: 'homeassistant/status'
    payload: 'offline'

[Turbox35]

After the HA update, I didn't try to just restart Zwave-js ui, perhaps it did the job. I will do that if the issue come back.

Z-UI listens for Home assistant birth topic in order to detect if/when Home assistatn restarts and in that case it does a re-discovery of all devices. Maybe you didn't configured the birth topic well?

From docs:

In your Home Assistant configuration.yaml:

mqtt:
  discovery: true
  discovery_prefix: <your_discovery_prefix>
  broker: [YOUR MQTT BROKER] # Remove if you want to use builtin-in MQTT broker
  birth_message:
    topic: 'homeassistant/status'
    payload: 'online'
  will_message:
    topic: 'homeassistant/status'
    payload: 'offline'

Hello.

I use the MQTT integration but when I reboot HA it publish offline then online under homeassistant/status. This seems correct.

I confirm too that all nodes are rediscovered because I see it under Domoticz log. But after a reboot of HA, some devices come back unavailable then I need to restart Zwave-js UI or just click on save settings (without change nothing) and the devices come back working into HA. To summarise, restarting HA is not sufficient even if it rediscovers all nodes.

[robertsLando]

But after a reboot of HA, some devices come back unavailable then I need to restart Zwave-js UI or just click on save settings (without change nothing) and the devices come back working into HA.

No clue where the error is so, I'm sorry. @kpine better knows home assitant then me so may be able to help here

[Turbox35]

Hello @robertsLando, I did the last update of Zwave-js UI followed with an hardware migration (docker Pi4 buster to docker Pi4 bulleyes) and then HA update but the problem remains. The problem occurs only when I restart HA, it must be followed by a click on Zwave-js UI parameters save.

[robertsLando]

@Turbox35 Check if when you restart home assitant you see this message on Z-Wave JS UI console:

https://github.com/zwave-js/zwave-js-ui/blob/425a817d9875e360a471270daa93741874bd500d/api/lib/Gateway.ts#L2208-L2209

[Turbox35]

When I reboot HA, in the Zwave-js UI log: 2024-04-26 17:03:56.376 INFO MQTT: Message received on homeassistant/status: 'offline' 2024-04-26 17:03:56.379 INFO GATEWAY: Home Assistant is OFFLINE 2024-04-26 17:04:26.772 INFO MQTT: Message received on homeassistant/status: 'online' 2024-04-26 17:04:26.775 INFO GATEWAY: Home Assistant is ONLINE and under domoticz log I see re-discovery of my devices.

[robertsLando]

Ok so this seems an error on Home Assistant as when that happens I re-send the discovery there (like when you click on save or restart the application), I wish I could be more helpful here

[Turbox35]

Hi @robertsLando , is there a possibility that some retained messages are corrupt or cached? Can I remove all retained messages on mqtt broker and then resend discovery without duplicate devices or something like that?

[robertsLando]

Hi @robertsLando , is there a possibility that some retained messages are corrupt or cached? Can I remove all retained messages on mqtt broker and then resend discovery without duplicate devices or something like that?

Retained messages are always cached on broker side, if you removed some entities it could be them are still loaded on your broker