[mqtt.homeassistant][JINJA] Transformation problem with value template key

[apella12]

I did read this is potentially the wrong place (just need some feedback at this point). I'm not fluent in Java but can read a little bit of code. The actual log message is from the Addons repository, but when I looked into that class it appears to call on the transformation services in the core. However, from within the core I could not find the source.

Background: I'm experimenting with zwave-js-ui for OH. On a test system I can discover all my test zwave devices in OH4M3. Test setup with openhabian (on WSL - Debian 11 with MQTT, Json Path & Jinja transform installed) and all the channels/topics appear. It is early in testing so far, but most channels/topics can be linked to items and display the state (ON/OFF or Number).

The one type of channel /topic with a problem so far; {"value_template":"{{ {22: \"Window/door is open\",23: \"Window/door is closed\",5632: \"Window/door is open in regular position\",5633: \"Window/door is open in tilt position\"}[value_json.value] | default(value_json.value) }}","icon":"mdi:alarm-light","state_topic":"zwave/nodeID_10/113/0/Access_Control/Door_state","json_attributes_topic":"zwave/nodeID_10/113/0/Access_Control/Door_state","device":{"identifiers":["zwavejs2mqtt_0xe2bd1ff1_node10"],"manufacturer":"Zooz","model":"Tilt Shock XS Sensor (ZSE43)","name":"nodeID_10","sw_version":"1.20.1"},"name":"nodeID_10_notification_access_control_door_state","unique_id":"zwavejs2mqtt_0xe2bd1ff1_10-113-0-Access_Control-Door_state"}

Expected Behavior

I would hope this should return the string corresponding with the number. I don't know if this is a bug or just not allowed by the Jinja /JSON specification.

Current Behavior

2023-06-27 14:32:56.534 [WARN ] [t.generic.ChannelStateTransformation] - Executing the JINJA-transformation failed: An error occurred while transformation. InterpretException: Error resolving expression [{22: "Window/door is open",23: "Window/door is closed",5632: "Window/door is open in regular position",5633: "Window/door is open in tilt position"}[value_json.value] | default(value_json.value)]: TemplateStateException: Dict key must be a string or identifier, was: 22

and another; 2023-06-27 14:32:56.174 [WARN ] [t.generic.ChannelStateTransformation] - Executing the JINJA-transformation failed: An error occurred while transformation. InterpretException: Error resolving expression [{0: "off", 1: "heat", 2: "cool"}[value_json.value] | default('off')]: TemplateStateException: Dict key must be a string or identifier, was: 0

Possible Solution

Allow a number to be the index to get the string. If it helps, all the numbers will be a member of integers.

Context

zwave-js-ui includes several features not currently available with the OH Zwave binding including S2 security, OTA firmware updates, 700/800 series controllers and has applied for certification with the Zwave Alliance standards body. I do not know if this is going to work. There could be more issues (possibly impossible to resolve) as I add different devices.

[apella12]

Thanks for moving. Sorry about that.

Today I captured some Debug level logs around the expression in question. debug-jinja.txt

[gargamelonly]

Could also be related: https://github.com/Koenkk/zigbee2mqtt/issues/17395

[apella12]

I believe this will need to be addressed upstream first. I opened an issue on HubSpot/jinjava and it appears there is an open PR to address. []https://github.com/HubSpot/jinjava/pull/934(url). After that OH will need to update to the version that contains that PR.

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/modify-jinja-transform-to-handle-a-number-key-in-json/148484/1

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/test-zwave-js-ui-in-place-of-zwave-binding/150007/3

[fcaputo]

Wouldn't it be possible to iterate over the keys and simply call toString() on them before passing the value to jinjava?

[apella12]

@fcaputo I did try unsuccessfully for some time to modify the HA configs to get jinjava to recognize the keys as strings before reverting to using the generic mqtt thing(s) to set up 135 channels. I'm fully on zwave-js-ui at this point. Modifying the HA configs channels is challenging as they are fixed in OH. Since zwave-js-ui is the main Zwave provider for HA, modifying the configs in the application is not an option. I used Mqtt Explorer to publish test config topics (like adding the toString()) and see how they were handled by OH. I also created and updated jinjava .jars with increased logging and other ideas to get this to work (at least at my level of knowledge).

Regardless, my "meta" idea was to promote the zwave-js-ui 3rd party app as a more modern and more widely used "binding" for OH zwave devices, similar to zigbee to Mqtt. That will not be possible for many OH users without a working discovery component. Even if I missed a way to get the keys recognized as strings, I think it would be too much to ask a casual OH user to modify the HA configs. It is actually more straightforward, if not repetitive to just use the generic mqtt thing. The number key needs to be addressed at the source (IMO). As noted above there is an open PR on the git Hubspot/jinjava, so there is some hope. However, I'd also note that other HA configs don't always work perfectly in OH. At best I think there will need to be a mix of discovered topics/channels and manually created ones. Numbered keys will help tilt the balance to discovery, so I left this open for the "meta" objective.

[fcaputo]

Hi Robert,

let's start a discussion in the openhab forum. I totally agree to your vision of making zwave-js-ui the guy, who connects OH to the devices. But we definitely need autodiscovery. IMO all devices listed in https://devices.zwave-js.io should work out of the box. So I guess we need a bridge. zwave-js-ui has a REST and a WebSocket API, which might be useful. As far as I understood it's providing a "Home Assistant Backend API".

I'm willing to implement such a bridge though I'm completely new to OH. But I need it. I bought a 700 stick and want it to work in OH.

[apella12]

The HA discovery does have some OH limitations (see Docs) and they also only cover certain device types hass device component.txt. Probably the most impactful is the climate (thermostats). I haven't been able to test if the "keys as numbers" would resolve that. (If you have some maven build knowledge I did compile a jinjava with the PR fix noted above for OH testing, but could not get it compiled into OH transform). I was going to try again once winter sets in and my outdoor activities are restricted. My opinion is that full discovery will take some time and always require some generic (manual) mqtt channels.

I'm not a zigbee user, but there seems to be a lot of zigbee2mqtt OH users. I have been wondering how many use discovery or the manual method or both. That would be a good forum question.

An alternative approach that will take some work and time is to create new Mqtt discovery service in OH (easy) and create a complete alternative set of "configs" in zwave-js-ui (very hard-at least for me). The HA discovery templates have been around a long time and are complex. Some could be reused, but some would have to be written (to avoid the "number as key issue" and other quirks and integrated into the ZUI structure. There would probably be other issues. Getting the "number as key" to work will likely be easier and quicker.

Lastly, I wrote a 700 controller PR and compiled it into a test "Release" binding on my github website. Which one to use depends on your OH version. That is going to be the quickest (to use your 700 controller) for now if you want to try.

[fcaputo]

On which branch is it based? I tried to run on OH 4.0.4 with the following exception:

2023-11-16 19:02:45.147 [WARN ] [org.apache.felix.fileinstall        ] - Error while starting bundle: file:/usr/share/openhab/addons/org.openhab.binding.zwave-4.1.0-SNAPSHOT.jar
org.osgi.framework.BundleException: Could not resolve module: org.openhab.binding.zwave [240]
  Unresolved requirement: Import-Package: org.openhab.core.io.transport.serial

        at org.eclipse.osgi.container.Module.start(Module.java:463) ~[org.eclipse.osgi-3.18.0.jar:?]
        at org.eclipse.osgi.internal.framework.EquinoxBundle.start(EquinoxBundle.java:445) ~[org.eclipse.osgi-3.18.0.jar:?]
        at org.apache.felix.fileinstall.internal.DirectoryWatcher.startBundle(DirectoryWatcher.java:1260) ~[?:?]
        at org.apache.felix.fileinstall.internal.DirectoryWatcher.startBundles(DirectoryWatcher.java:1233) ~[?:?]
        at org.apache.felix.fileinstall.internal.DirectoryWatcher.startAllBundles(DirectoryWatcher.java:1221) ~[?:?]
        at org.apache.felix.fileinstall.internal.DirectoryWatcher.doProcess(DirectoryWatcher.java:515) ~[?:?]
        at org.apache.felix.fileinstall.internal.DirectoryWatcher.process(DirectoryWatcher.java:365) ~[?:?]
        at org.apache.felix.fileinstall.internal.DirectoryWatcher.run(DirectoryWatcher.java:316) ~[?:?]

[apella12]

Go into the karaf console (sudo openhab-cli console), your sudo password, then habopen. Then type feature:install openhab-transport-serial Note: this is normally installed with the UI installation of the zwave binding to allow communication with the Zstick. It is not a 700 issue.

They have made some changes in the build since 4.0, but I think 4.0.4 will work with the 10-18-23 version. If a problem (some sort of karaf message) try the July 27 version. Since I converted over to ZUI, I'm not able to test with a 700 controller, but it had been working for about 9 months through all the 4.0 milestones

[ccutrer]

This has been fixed in upstream Jinjava; we're just waiting on them to make a release so we can update.

[apella12]

@ccutrer I was aware you fixed this. Thanks for your efforts. However, the jinjava code owners do not seem to be in any hurry as their last release was Feb 15. IMO - It is too bad OH has to rely on this code for the HA configs. I'll close this

[ccutrer]

👍

Yeah, it's a little frustrating. I opened a new issue with them today asking for a release.

And heh, I got you confused since I mostly see the GitHub notifications which shows your full name, but then going through the issues in your username, and I didn't connect them together!