[widget] add metadata support for oh-input

[stefan-hoehn]

This PR allows to show and edit metadata via the oh-input component:

Here are two examples: (1) access the value of the namespace controlheating and (2) a config property

(1) accessing the value

- component: oh-input-card
  config:
    inputmode: text
    item: testString
    metaconfig: controlheating.value
    outline: true
    sendButton: true
    title: controlheating.value
    type: text
    useDisplayState: false

Accessing a particular property:

- component: oh-input-card
    config:
      title: Zeit morgens
      item: F2_Office_Heating_Temp
      type: text
      inputmode: text
      metaconfig: controlheating.default.setpoint2.time
      outline: true
      sendButton: true
      useDisplayState: false

metadata namespace controlheating of item F2_Office_Heating_Temp

value: ON
config:
  default:
    setpoint2:
      degrees: 22
      time: 08:00
    setpoints: 2

[relativeci[bot]]

Job #785: Bundle Size — 16.23MiB (~+0.01%).

402917c(current) vs 31afa68 main#784(baseline)

Metrics (2 changes)

	Current Job #785	Baseline Job #784
Initial JS	1.86MiB(+0.05%)	1.86MiB
Initial CSS	608.52KiB	608.52KiB
Cache Invalidation	90.63%	90.64%
Chunks	218	218
Assets	688	688
Modules	2019	2019
Duplicate Modules	108	108
Duplicate Code	1.74%	1.74%
Packages	137	137
Duplicate Packages	15	15

Total size by type (2 changes)

|            |       Current
[Job #785](https://app.relative-ci.com/projects/ZNG5hy4VeSJQVQcq1Kvu/jobs/785-oLvNoczA0TgvmRfIVjqJ?utm_source=github&utm_medium=pr-report "View job report") |      Baseline
[Job #784](https://app.relative-ci.com/projects/ZNG5hy4VeSJQVQcq1Kvu/jobs/784-nj0OkV4OHSjegbuirXOX?utm_source=github&utm_medium=pr-report "View baseline job report") | |:--|--:|--:| | [CSS](https://app.relative-ci.com/projects/ZNG5hy4VeSJQVQcq1Kvu/jobs/785-oLvNoczA0TgvmRfIVjqJ/assets?ba=%7B%22filters%22%3A%7B%22ft.CSS%22%3Atrue%2C%22ft.JS%22%3Afalse%2C%22ft.IMG%22%3Afalse%2C%22ft.MEDIA%22%3Afalse%2C%22ft.FONT%22%3Afalse%2C%22ft.HTML%22%3Afalse%2C%22ft.OTHER%22%3Afalse%7D%7D "View all CSS assets") | `857.62KiB` | `857.62KiB` | | [Fonts](https://app.relative-ci.com/projects/ZNG5hy4VeSJQVQcq1Kvu/jobs/785-oLvNoczA0TgvmRfIVjqJ/assets?ba=%7B%22filters%22%3A%7B%22ft.CSS%22%3Afalse%2C%22ft.JS%22%3Afalse%2C%22ft.IMG%22%3Afalse%2C%22ft.MEDIA%22%3Afalse%2C%22ft.FONT%22%3Atrue%2C%22ft.HTML%22%3Afalse%2C%22ft.OTHER%22%3Afalse%7D%7D "View all Fonts assets") | `1.08MiB` | `1.08MiB` | | [HTML](https://app.relative-ci.com/projects/ZNG5hy4VeSJQVQcq1Kvu/jobs/785-oLvNoczA0TgvmRfIVjqJ/assets?ba=%7B%22filters%22%3A%7B%22ft.CSS%22%3Afalse%2C%22ft.JS%22%3Afalse%2C%22ft.IMG%22%3Afalse%2C%22ft.MEDIA%22%3Afalse%2C%22ft.FONT%22%3Afalse%2C%22ft.HTML%22%3Atrue%2C%22ft.OTHER%22%3Afalse%7D%7D "View all HTML assets") | `1.23KiB` | `1.23KiB` | | [IMG](https://app.relative-ci.com/projects/ZNG5hy4VeSJQVQcq1Kvu/jobs/785-oLvNoczA0TgvmRfIVjqJ/assets?ba=%7B%22filters%22%3A%7B%22ft.CSS%22%3Afalse%2C%22ft.JS%22%3Afalse%2C%22ft.IMG%22%3Atrue%2C%22ft.MEDIA%22%3Afalse%2C%22ft.FONT%22%3Afalse%2C%22ft.HTML%22%3Afalse%2C%22ft.OTHER%22%3Afalse%7D%7D "View all IMG assets") | `140.74KiB` | `140.74KiB` | | [JS](https://app.relative-ci.com/projects/ZNG5hy4VeSJQVQcq1Kvu/jobs/785-oLvNoczA0TgvmRfIVjqJ/assets?ba=%7B%22filters%22%3A%7B%22ft.CSS%22%3Afalse%2C%22ft.JS%22%3Atrue%2C%22ft.IMG%22%3Afalse%2C%22ft.MEDIA%22%3Afalse%2C%22ft.FONT%22%3Afalse%2C%22ft.HTML%22%3Afalse%2C%22ft.OTHER%22%3Afalse%7D%7D "View all JS assets") | `9.18MiB` (`+0.01%`) | `9.18MiB` | | [Media](https://app.relative-ci.com/projects/ZNG5hy4VeSJQVQcq1Kvu/jobs/785-oLvNoczA0TgvmRfIVjqJ/assets?ba=%7B%22filters%22%3A%7B%22ft.CSS%22%3Afalse%2C%22ft.JS%22%3Afalse%2C%22ft.IMG%22%3Afalse%2C%22ft.MEDIA%22%3Atrue%2C%22ft.FONT%22%3Afalse%2C%22ft.HTML%22%3Afalse%2C%22ft.OTHER%22%3Afalse%7D%7D "View all Media assets") | `295.6KiB` | `295.6KiB` | | [Other](https://app.relative-ci.com/projects/ZNG5hy4VeSJQVQcq1Kvu/jobs/785-oLvNoczA0TgvmRfIVjqJ/assets?ba=%7B%22filters%22%3A%7B%22ft.CSS%22%3Afalse%2C%22ft.JS%22%3Afalse%2C%22ft.IMG%22%3Afalse%2C%22ft.MEDIA%22%3Afalse%2C%22ft.FONT%22%3Afalse%2C%22ft.HTML%22%3Afalse%2C%22ft.OTHER%22%3Atrue%7D%7D "View all Other assets") | `4.7MiB` (`~+0.01%`) | `4.7MiB` |

---

View job #785 report View main branch activity

[ghys]

This violates a fundamental principle that widgets are controls for end users, and editing metadata is an admin concern, so I don't think this is desirable.

And if we make this one exception, then why not extend it to making any HTTP request, which in OH is better done with a rule or an HTTP thing.

[stefan-hoehn]

I really beg you to allow us to do something like that 🙏. I am actually working towards exactly since weeks with all the metdadata implementation I did.

I like to give you an example that I have already implemented very easily at my house:

I am configuring my heating settings within each heating item and I want to avoid multiple items for that, so I am configuring the item like so

value: ON
config:
  default:
    setpoint1:
      degrees: 19
      time: 22:00
    setpoint2:
      degrees: 22
      time: 08:00
    setpoints: 2
and

value: ON
config:
  default:
    setpoint1:
      degrees: 19
      time: 22:00
    setpoint2:
      degrees: 22
      time: 06:00
    setpoint3:
      degrees: 19
      time: 8:00
    setpoint4:
      degrees: 22
      time: 15:00
    setpoints: 4

I can then let my used configure the heating via the UI

I have an even more complex usecase where I have configurations for my rollershutters which I like my users to configure them via the UI.

I tried many other approaches but they would much more effort in either using in my case dozens of items or complex rules and this would be much more elegant. If you only want to allow this to users that are admins, I would not mind to only allow this to admins (I would give my users admin right for that case then).

So please, Yannick, 🙏🙏🙏🙏 let's think about it ...

[hmerk]

@ghys @stefan-hoehn I just stumbled across this and I have to say, that I second this approach, perpahps we need to think about it and find a way to not fully allow editing all metadata trough widgets but some, lets say the custom metadata. Why do I second this ? You know the main_widget project (still don't like the name, but that's a different story). Within this project we have Rollershutter and RadiatorControll widgets with several time definitions (4 each atm) to handle automated opening/closing or heating. As I have 10 Radiator Controls and 7 Rollershutters, this needs 68 additional DateTime items and another 17 Switch items for enabling/disabling the schedules. Storing those information within a custom metadata would make things easier and significantly reduces the amount of extra items.

[florian-h05]

Whilst I agree with @ghys that editing metadata normally is an admin-only task, I also see the huge benefit in allowing limited metadata access to end-users.

I won‘t list any additional use-cases here since @hmerk clearly pointed out the benefit of using metadata to store user-configured stuff.

In the current state, this PR cannot be merged because core does not allow metadata write access for the user role which is IMHO correct as this is security critical.

I would propose the following approach to allow restricted access to Item metadata to store end-user configured stuff:

1. Add a allowUserAccess (or name it different) flag/property in addition to value and configuration to Metadata
2. If the admin wants to allow end-user write access to metadata, he has to set this new property to true
3. Adjust the REST endpoint to check allowUserAccess on existing metadata, if true process the request, else reject with 401

@ghys @kaikreuzer @J-N-K WDYT? Please let me know.

[ghys]

Just to make my stance clear in this case, and many other similar instances when I've been or will be Grouchy: me making objections is not necessarily equivalent to rejection - notice I haven't closed the PR right off the bat, so I accept discussing about it.

However, I wouldn't be doing my job as a maintainer if I didn't make those objections when I thought they were legitimate - and the PR was initially submitted without context so my objections were based on the initial proposal, not influenced by the subsequent added context. I strongly believe that what's expected of maintainers in this project isn't to do light code reviews and accept everything as a godsend because there was work involved and that should be respected, otherwise what's needed is clerks who merely push the big green button when it appears - not maintainers. Also, I believe any self-respecting software engineer/architect, as I am, would seldom accept "look at my use case" as a sole sufficient justification for throwing away design and architectural principles in favor of making a user happy.

That being said: I absolutely see the merit of allowing end users to modify metadata in some cases, but it needs to be properly defined and a project-wide decision. @florian-h05's proposals are interesting and should be discussed further 👍

[J-N-K]

1. Add a allowUserAccess (or name it different) flag/property in addition to value and configuration to Metadata
2. If the admin wants to allow end-user write access to metadata, he has to set this new property to true
3. Adjust the REST endpoint to check allowUserAccess on existing metadata, if true process the request, else reject with 401

The problem is that the metadata already needs to be present to let the admin change the flag.

[hmerk]

The problem is that the metadata already needs to be present to let the admin change the flag.

Why is that ? If I understood @florian-h05 correct, the additional flag just changes the config of the rest endpoint for metadata to be editable by users, not only admin. At this point, there is no need to know anything about metadata details at all. Or did I miss something?

Edit: Yes, I misunderstood, but still seems to be not an issue to me. Metadata should only be created via UI at item config and only editing should be enabled via widget. So the Admin will create the metadata together with the „security“ config.

[florian-h05]

However, I wouldn't be doing my job as a maintainer if I didn't make those objections when I thought they were legitimate - and the PR was initially submitted without context so my objections were based on the initial proposal, not influenced by the subsequent added context.

I know and I understand your point. I guess that I would have written something equivalent to your review if I hadn't already talked and thought about this topic before this PR was created.

The problem is that the metadata already needs to be present to let the admin change the flag.

… which IMO is no problem and intended behaviour. I guess usually you need that metadata for some configuration, e.g. open/close times for rollershutter, and to process that metadata in a rule you already need to know the metadata. So you have to create it as admin, think about the data model you use (you can store objects inside the metadata configuration).

Why is that ? If I understood @florian-h05 correct, the additional flag just changes the config of the rest endpoint for metadata to be editable by users, not only admin.

You didn‘t understand me right. The additional flag should not change the config of the whole REST endpoint and it should not allow user-access to all metadata.

I proposed to add such a flag to each metadata itself to „whitelist“ a single metadata namespace for user access. The user still does NOT get write access to the other metadata.

If it‘s still unclear, please ask.

[hmerk]

If it‘s still unclear, please ask.

See my Edit, I already corrected myself but did not want to delete my post in case it was already read. 😉

[hmerk]

While thinking a bit more about the usecase I described, I think I found a depending issue. If we use the metadata for time configurations, how can we use this information to run a rule for opening or closing rollershutters e.g. ? Actually, I am using triggers depending on DateTime Items, which is working so far, only need to disable and enable the rules after a restart of openHAB. Should have filed an issue for that.

[florian-h05]

Not sure if there is a metadata changed event that can be used for a rule trigger:

• if not: create a rule that runs every minute and checks your rollershutter Items‘ metadata against the current time
• if yes: On metadata change, you can either (re-)schedule a job to the DateTime stored in the metadata or programatically create a rule, which is e.g. possible with JS Scripting.

[stefan-hoehn]

That is exactly what I do. I run the rule and compare it to the expected time. Before I started implementing this I actually checked the events with the developers section. At least I could see one there.

[hmerk]

Thanks @florian-h05 and @stefan-hoehn , this is an interesting approach, which also will reduce the number of actually needed rules, even it will make the rules more complex.

[florian-h05]

Core PR for my proposed solution is now open: https://github.com/openhab/openhab-core/pull/3401.

[spacemanspiff2007]

I really dislike that you are using metadata as a kind of dynamic data structure. It's meant to statically set up some bindings or statically configure some simple item behavior. Just because you can save arbitrary data in the item metadata doesn't mean you should.

On the other hand creating 100+ items via text editor is super quick (<3mins) if you stick to a proper naming scheme. That would then also provide the benefit of item events, persistence, etc.. Maybe this is more a tooling issue / rule creation issue?

I see the need for some dynamic data structures (e.g. list/mapping) to make repetitive tasks and item definitions easier. But the much nicer solution would be to create a new item type (e.g. JsonItem) which then then could

• provide the corresponding item events
• show proper error messages and do the access handling
• maybe even do content validation

That way it wouldn't be necessary to misuse item metadata. It would also be a net gain for openHAB since some bindings provide future data points as a csv, some as a json string and some provide 100+ channels. These could then all be consolidated in a proper item type.

I'm not in favor of this - do it right or don't do it at all.

[hmerk]

But the much nicer solution would be to create a new item type (e.g. JsonItem)

Why, we already have String Items which can store a Json String.

That way it wouldn't be necessary to misuse item metadata.

Who defines what a misuse would be or not ?

do it right or don't do it at all.

What is wrong and what is right, where is the definition ?

[J-N-K]

I'm not sure about this approach. On the one hand I see the use-case, but I agree with @ghys that editing metadata is an administrative task (and I think it should stay like that). The approach from https://github.com/openhab/openhab-core/pull/3401 feels a bit hacky. I also agree with @spacemanspiff2007 that metadata should not be used for storing dynamic data. For me metadata is something that adds additional information to the item (like "expose this in Alexa as a switch" or "when displaying the state, use the following format"). Dynamic data should be stored in items.

I think what you are really aiming for is similar to https://github.com/openhab/openhab-core/issues/1740. IMO metadata is the correct way to create these relations: The Number item used as setpoint could have a timedsetpoint metadata reference to a DateTime item that carries its start time. That way an UI widget knows about the relationship and can display them together. This also removes the need to allow user-editable metadata, because the relationship needs to be defined only once and the changes are then normal item state changes.

[rkoshak]

Not sure if there is a metadata changed event that can be used for a rule trigger:

Unless this has changed recently, there is no event generated with metadata changes.

I tend to fall somewhere in the middle on this discussion. Let me raise the idea that we (myself included soetimes) tend to lump the end users of these APIs into two categories:

1. APIs used by other parts of the OH system (e.g. bindings, UI add-ons, etc.)
2. APIs used by end users (e.g. Items, Things, etc.).

However, I think there is a growing third class of users who fall somewhere between 1 and 2.

3. APIs used by developers of rule templates, custom UI widgets, and Blockly libraries posted to the marketplace (and of course those users who build stuff like this on their own).

This idea and those like it are clearly directed towards this third class of users. I agree with the sentiment that the average end user almost certainly should not use something like this in the average course of setting up OH. But those of us in category 3 are closer to category 1 than we are to category 2. We are not just developing for ourselves, we are developing for our own end users.

Ultimately, the way I use Item metadata in my templates and my understanding of what's being proposed here is a desire to make it possible for those of us in category 3 to make the configuration and use of our offerings easier for our end users. If you are going to say we can't use metadata for this, please provide a reasonable counter approach. We've already determined that separate Items for everything isn't good enough. It's awkward, unwieldy, results in a huge proliferation of Items, and often lacks structure, and most importantly of all, imposes a significant burden on our end users.

So if we can't use Item metadata, let's come up with something we can use. Is it https://github.com/openhab/openhab-core/issues/1740? 🤷 I personally worry that our current fixed set of Items is not sufficient to support all use cases but can be convinced (note shunting a bunch of JSON into a String Item feels every bit as "wrong" as "dnamic metadata" to me.

Though the creation of a bunch of Items through the UI could be a pain initially and to maintain long term, especially with the way that users insist on putting everything in the semantic model. Maybe if widgets had a way to create Items that are missing at instantiation/configuration time? I used to do that in rules a lot and have pondered doing so in some of my rule templates again.

As for the argument that metadata should not be for dynamic data, I'm not sure that the distinction is quite so cut and dry. What makes it dynamic? The fact that a user enters it through some interface other than Items -> Add metadata to Item? The fact that it's not an admin user making the change? The fact that a user may want to change it periodically? If so how often makes it "dynamic"? The fact that it's added/modified in a rule?

[spacemanspiff2007]

The issue is not with the user role(s), but that something that is intended to be used for item configuration is used to transport item data. It's inevitable that this will lead to issues and problems down the road because that is not the intended use case. It's best practice to separate model definition (here item definition) and model data (here times / temperature) so we should stick to that.

We've already determined that separate Items for everything isn't good enough. It's awkward, unwieldy, results in a huge proliferation of Items, and often lacks structure, and most importantly of all, imposes a significant burden on our end users.

The item count should not be an issue for openHAB. I agree that it's not very good, but it's not as bad as you make it sound, too. Watch me create all the required items from @stefan-hoehn for 5 rooms in ~35 seconds.

(note shunting a bunch of JSON into a String Item feels every bit as "wrong" as "dnamic metadata" to me.

Nobody suggested that. That was just a misunderstanding (or a low effort troll post) of @hmerk . What I suggested was a e.g. JsonItem (or whatever name) that can hold these dynamic data structures. Maybe even with a defined (json) schema, so the structure of the dynamic content is still validated. The widget could then have access to the dynamic data structure (which is not a string).

It seems to be that we currently have no predefined way to model relations between items. @J-N-K suggested to use metadata which is one way, the HomeKit Binding uses groups which is another. A first step might be to come up with a suggested way on how to model the relations and document it.

In terms of solution a way could also be to pass a group to the widget which contains groups with the corresponding items. Or a flat group and the metadata on the item is used to display the input dialogue.

If the general consent is that it's a good idea to use item metadata then please make it at least generic so e.g. item tags, label, groups etc. can also be accessed and changed through the widget.

[hmerk]

@spacemanspiff2007

That was just a misunderstanding (or a low effort troll post) of @hmerk .

Mind your wording and who you are talking about/to. My posts are never trolling, and what I suggested is working. Is it the right way to do is another story.

Edit: To elaborate a bit more: Is there a clear rule that says metadata always have to be static ? I have not seen such.

[florian-h05]

I think what you are really aiming for is similar to openhab/openhab-core#1740. IMO metadata is the correct way to create these relations:

I don't think we want to create such relations. Having these creations doesn't help with creating and later managing dozens of "helper" Items later.

I tend to fall somewhere in the middle on this discussion.

I am more and more going into that direction. Both sides have some aspects to think about:

Storing dynamic information in metadata:

• information belonging to the Item is stored close to the Item
• no events on update
• no update mechanisms for clients
• avoids having dozens of "helper" Items to manage

Storing dynamic information in Items itself:

• Item events (so rules can run when the configuration of e.g. your rollershutter opening times changes and reschedule the action). Much more elegant as a trigger than a rule running every minute.
• state updates for clients with SSE (or WebSocket)
• having dozens of "helper" Items which can be difficult to manage

The item count should not be an issue for openHAB. I agree that it's not very good, but it's not as bad as you make it sound, too. Watch me create all the required items from @stefan-hoehn for 5 rooms in ~35 seconds.

Even though the Item count is no problem for openHAB, I agree that adding these Items is easy if you use the .items file syntax, but how do you mass-manage them later in the UI? I agree with @rkoshak that this can be a pain.

If the general consent is that it's a good idea to use item metadata then please make it at least generic so e.g. item tags, label, groups etc. can also be accessed and changed through the widget.

This is something I really disagree with because IMHO is a security problem. My proposed metadata access is configured for each namespace on each Item and therefore allows fine-grained access security.

I'd like to come back to @spacemanspiff2007 's proposal:

What I suggested was a e.g. JsonItem (or whatever name) that can hold these dynamic data structures. Maybe even with a defined (json) schema, so the structure of the dynamic content is still validated. The widget could then have access to the dynamic data structure (which is not a string).

This feels better than putting a JSON string into a String Item and if we had such a JsonItem we could make the UI automatically parse a JS object from it.

JsonItems could also have another very nice use case: instead of having hundreds of Items for weather forecast (e.g. OWM binding), you could alternatively have a single JsonItem that holds the data as JSON. I think JSON in UI widgets is also simpler than dealing with hundreds of Items. Of course these JSON channels in bindings are only an alternative to the traditional Items, so user can choose what fits best for their use case.

@J-N-K What do you think of such a JsonItem?

[rkoshak]

The item count should not be an issue for openHAB. I agree that it's not very good, but it's not as bad as you make it sound, too. Watch me create all the required items from @stefan-hoehn for 5 rooms in ~35 seconds.

Which is exactly irrelevant to how UI users interact with Items right now. And putting on my category 3 hat, the users of my rule templates and UI widgets overwhelmingly either are UI first, UI only, or wouldn't know how to use a editor to do what you demonstrate in a text editor. You can do it in 15 seconds. My users cannot. Heck, even I have to look up to remember how to do what you demonstrate above making it end up taking longer than just doing it the long./frustrating way.

A first step might be to come up with a suggested way on how to model the relations and document it.

That's only half of the problem. The other half of the problem is that the burden placed on the users of creations on the marketplace (rule templates, UI widgets, etc.) can be significant and we, as the developers of those capabilities, want a way to ease that burden.

Is a way to edit metadata from a UI input widget the right way to meet that? Probably not. But if not, than what? 5th level ninja text editor skills isn't the answer either.

if we had such a JsonItem we could make the UI automatically parse a JS object from it.

It would also have additional uses. JSON is often received by MQTT and HTTP and sometimes Exec Channels and a deliberate decision was made to provide a limited support for handling arrays in the JSONPATH transformation. Creating rules to handle complex JSON would be a tiny bit easier if it comes to the rule already parsed.

But it is a pretty radical departure from what Items represent. And I'm not sure it wholly addresses all the concerns.

[J-N-K]

I don't have the slightest idea how arbitrary data should end up parsed in DSL rules. That's why I also don't see a reason why we would need a JsonItem, if we store it as text anyway then a String item is also fine. For UI widgets it would be enough to tag it with something like JSON. The only language where it is really easy is JS and that already supports JSON.

[rkoshak]

I don't have the slightest idea how arbitrary data should end up parsed in DSL rules.

It's not arbitrary, it's JSON. And, as is the case today, the author of the rule would know something of the format and content of that JSON.

In Rules DSL it's the difference between (yes this could be a profile, it' simplified from examples I've seen on the forum for illustrative purposes):

Rule:

rule "Get tomorrow's temp from JSON"
when
    Item MyJsonStringItem changed
then
    val temp = transform('SCRIPT:graaljs', 'extractTomorrowsTemp.script', MyJsonStringItem)
    if(temp != null) TomorrowTemp.postUpdate(temp)
end

Transform

function((function(data) {
  try {
    const parsed = JSON.parse(data);
    // Get the last element from an array
    const record = parsed.temps[parsed.temps.length-1];
    const forecastTemp = record.highTemp;
    return (forecastTemp) ? forecastTemp : null;
  catch(e) {
    return null;
  }
})(input);

And this (I'm assuming the type of the state would inherit from org.json.JSONObject):

rule "Get tomorrow's temp from JSON"
when
    Item MyJsosItem changed
then
    val records = (MyJsonItem.state as JsonType).getArray('temps')
    val record = records.getJSONObject(records.length - 1);
    if(record.getString('forecastTemp') !== null)
        TomorrowTemp.postUpdate(record.getString('forecastTemp'))
end

Note I just typed these in, I likely got something wrong in the syntax.

I chose a JSON with a variable length array on purpose as that's one case the JSONPATH transform cannot handle.

Is even the second one "really easy"? No, but it's better than needing to break out to a transformation.

A new Item type isn't necessarily the only way to achieve this. MyJsonStringItem.getStateAs(JSONObject) could work too.

[J-N-K]

Your example nicely shows what I meant: The data stored inside the JSON is arbitrary and we can't do what we can do in JS (i.e. using data.qualifier1.qualifier2[index5]), so the benefit of knowing that the item contains JSON is really low and merely justifies a new item type.

But back to the original topic: Looking at Wikipedia we find the following definition:

Metadata is "data that provides information about other data",[1] but not the content of the data, such as the text of a message or the image itself. There are many distinct types of metadata, including:

• Descriptive metadata – the descriptive information about a resource.[vague] It is used for discovery and identification. It includes elements such as title, abstract, author, and keywords. Structural metadata – metadata about containers of data and indicates how compound objects are put together, for example, how pages are ordered to form chapters. It describes the types, versions, relationships, and other characteristics of digital materials.[2]
• Administrative metadata[3] – the information to help manage a resource, like resource type, permissions, and when and how it was created.[4]
• Reference metadata – the information about the contents and quality of statistical data.
• Statistical metadata[5] – also called process data, may describe processes that collect, process, or produce statistical data.[6]
• Legal metadata – provides information about the creator, copyright holder, and public licensing, if provided.

Metadata is not strictly bounded to one of these categories, as it can describe a piece of data in many other ways.

"time1", "temperature1" etc. are clearly additional data, not additional information about the data (item). Therefore my opinion is that storing that in metadata is wrong. I also don't see why anything of what is described above should be user-editable.

We should not make the mistake to use missing UI support (in this case "mass-editing of items") as argument to weaken design principles in core but instead add what is missing to UI. UI is behind what core can do in some other places (e.g. transformation configuration), but nobody would say "add a servlet for that in core because UI doesn't support it".

[rkoshak]

The data stored inside the JSON is arbitrary and we can't do what we can do in JS (i.e. using data.qualifier1.qualifier2[index5]), so the benefit of knowing that the item contains JSON is really low and merely justifies a new item type.

So there's no benefit in providing a way to handle JSON inside Rules DSL at all. Sorry folks, the syntax is a little ugly and error checking is awkward, so go use JavaScript or pound sand. 15 lines of code split between two different files written in two different programming languages compared to nine lines of code in the one file and the one language and there's no benefit?

Notice, my example isn't just "knowing" that the String is JSON. It's providing an already parsed Object from the JSON. Rules DSL doesn't support JSON nor does it support installing and using one of the JSON libraries (as far as I know) so if it's JSON, it can't be done in Rules DSL.

I'm not saying this rises to the point that it justifies a new Item type or even modifying the getStateAs() on a String Item, but I just can't see how you can say there's no benefit.

I also don't see why anything of what is described above should be user-editable. Which user?

• The tail end user of the home automation: I completely agree, these users have no business editing metadata.

• The admin user of the home automation: they already have a UI to add/remove/modify Item metadata, so I agree, these users already have the ability to edit metadata through the UI, they don't need an input widget to do this

• Developers of UI Widgets and Rule Templates: if not this, what can we offer to ease the configuration and deployment burden on the admin users when installing and configuring these?

I hate that the instructions for my rule templates are always: go manually create all these Items, add this metadata to each of them, and then add them all to a Group. Only then can you create an instance of the rule using this template. It's worse in UI widgets because the proliferation of Items and relationship tracking is much higher.

So I'd ask, let's forget about metadata as the solution. IMO that's an XY Problem. How can we help our category 3 users generate UI widgets and rule templates that are easier to configure and use? Maybe if the UI widgets could create the missing Items it needs at instantiation/configuration time? Maybe if we can bundle rules and UI widgets together and a bunch of setup can be put into a rule? Maybe the UI Widgets can ship with the Items it needs and we can have a nice way to link them up with the Thing(s) that drive it? I don't do much UI Widget development so I'm sure there are lots of other potential ideas that could apply that I'm not thinking of.

Just remember, I think @stefan-hoehn , @hmerk , and @florian-h05 are talking about UI widget developers providing an easier way for admin users to install and configure the widgets when they instantiated. I am talking about providing a way for rule template authors to provide an easier way for admin users to install and configure rule templates (though they have an easier time of it since they can do some config stuff on a first run or the like). We want to make it easier for others to use our stuff.

[florian-h05]

The data stored inside the JSON is arbitrary and we can't do what we can do in JS (i.e. using data.qualifier1.qualifier2[index5]), so the benefit of knowing that the item contains JSON is really low and merely justifies a new item type.

Let’s talk about this one last time in this PR. For further discussion we should open a new issue (where?) or a new thread in the development category of the forum. I have just checked, JSON inside UI widgets a JSON is as simple as this expression: =JSON.parse(items.WeatherJson.state). (Maybe this should be documented somewhere to make it clear for all, WDYT @rkoshak?) For working with such JSON strings in rules, JavaScript has the functionality. Rules DSL does not, but TBH I‘m not sure if it needs to, I have the feeling that rules DSL is loosing popularity to JS, Blockly and others.

Just remember, I think @stefan-hoehn , @hmerk , and @florian-h05 are talking about UI widget developers providing an easier way for admin users to install and configure the widgets when they instantiated.

Not exactly, we meant „the tail end user of the home automation“ to configure stuff related to his Items inside the Item‘s metadata. But I see the points made in this discussion and I am not sure if this is the way to take.

How can we help our category 3 users generate UI widgets and rule templates that are easier to configure and use? Maybe if the UI widgets could create the missing Items it needs at instantiation/configuration time? Maybe if we can bundle rules and UI widgets together and a bunch of setup can be put into a rule? Maybe the UI Widgets can ship with the Items it needs and we can have a nice way to link them up with the Thing(s) that drive it?

For UI widgets on the community I‘ve often seen that creators post a .items definition which can be either put inside a .items file or used to add Items in the UI. This is no autonomous solution, but a relatively simple copy and paste for the user. Your idea to ship Items with a rule template or a widget sounds like a good idea!

We should not make the mistake to use missing UI support (in this case "mass-editing of items") as argument to weaken design principles in core but instead add what is missing to UI.

„Mass-editing of Items“ is something that the UI is missing, I can possibly contribute something here but before I start developing anything, I‘d need to know what would functionality would be practical here (I am currently only using textual Items, so don‘t have much experience with Item management in UI). If you have any good ideas how to do that, please post them in discussion https://github.com/openhab/openhab-webui/discussions/1737.

[spacemanspiff2007]

You can do it in 15 seconds. My users cannot. Heck, even I have to look up to remember how to do what you demonstrate above making it end up taking longer than just doing it the long./frustrating way.

It's typically called Multi-Line Edit and it's Alt + Shift in Notepad++ as it is in most IDEs/text editors. Just press the keys and then you can select/edit vertically over the lines. But there might be a misunderstanding - your users aren't supposed to do that. I just wanted to show that you as the openHAB admin have a way to create the individual items quickly. Once these items are created they can be accessed from widgets/rules as usual.

Rules DSL does not, but TBH I‘m not sure if it needs to, I have the feeling that rules DSL is loosing popularity to JS, Blockly and others.

HABApp has support for (internal) items with arbitrary data from the start. And the only place where I'm using it is the weather forecast because it's in 10minute increments and creating 240 items was a little bit too much. Other than that it's always an item (internal or openHAB). Some bindings return json as a string, some bindings use a huge amount of items. Imho it's important to align the behavior but while missing complex datatypes are sometimes annoying and inelegant it's (imho) not that critical. So maybe we can verbalize a best practice for both binding developers and users on how to deal with returning large amount of values. It's something we can do now and adding it to the docs would already be a win.

For UI widgets on the community I‘ve often seen that creators post a .items definition which can be either put inside a .items file or used to add Items in the UI. This is no autonomous solution, but a relatively simple copy and paste for the user.

I think that's the best and easiest option. You probably still have to link the items to channels, some users have a custom naming scheme they want respected, additional groups and tags set, etc.. Auto generation will be very complex both to use and to implement. I'm not sure if it's worth the effort over a simple copy paste. Note that I am not arguing against a better method of mass editing items, I'm just pointing out that for this use case copy paste might be the best and most simple solution.

Imho this issue should be split. We're discussing metadata, modeling item relations, json item and now mass editing and auto generating items with rules/widgets.

[rkoshak]

Maybe this should be documented somewhere to make it clear for all, WDYT @rkoshak?)

I'm really not the best when it comes to UI Widgets but yes, if that's not documented it should be listed at https://www.openhab.org/docs/ui/building-pages.html#dynamically-configuring-components-with-expressions where it appears to indeed be missing. It makes me wonder what else might be missing from the supported expressions list.

But there might be a misunderstanding - your users aren't supposed to do that. I just wanted to show that you as the openHAB admin have a way to create the individual items quickly.

My main point is my users are the admin users. My users are the folks installing UI widgets and rule templates from marketplace, instantiating them and configuring for use by their end users. And based on the support I've been asked for, creating a bunch of Items with custom metadata is a burden and just challenging enough to cause a surprising number of users to need to ask for help.

Given that most of the people who are bothering to even look at the marketplace are UI only users, tend to not be programmers, and are on the lower end of the technical skills level of ability, knowing about single column selection is well beyond their knowledge and skills. And at least on a Mac, the UI text editors built into MainUI do not support it.

But even still in this thread, we, the widget and rule template developers, are basically being told the problems we face aren't real problems. "Just use format your .items file and use your text editor skills." "JSON is arbitrary, you don't need that in Rules DSL."

Everyone participating on this are openHAB subject matter experts. Maybe believe us when we say we have a problem? That we know what we are talking about? And if I'm off base and putting words into other's mouths, at least believe me?

Imho it's important to align the behavior but while missing complex datatypes are sometimes annoying and inelegant it's (imho) not that critical.

You've never tried to help a cargo-cult programmer configure something like https://community.openhab.org/t/aircondition-controller/127249 or https://community.openhab.org/t/open-weather-api-day-widget/136142 or https://community.openhab.org/t/time-based-state-machine-4-0-0-0-4-1-0-0/144208. It's a pretty big deal for us though and we face it with distressing frequency.

Auto generation will be very complex both to use and to implement. I'm not sure if it's worth the effort over a simple copy paste.

That's the same sort of thing that was said about Things back when 2.0 was being developed. I guess all that work wasn't worth it since copy and paste is so easy (yet so many still manage to screw it up).

I'll shut up now. As a rule template developer, I at least can build my templates to create the Items it needs and I've already added a ton of error checking to help them configure their Items. I suppose the UI widget developers are on their own.

Note: I don't care if we split the topic, but ultimately it's covered so many varied topics because there is a root problem that isn't being addressed and bulk editing Items is only part of the problem.

[hmerk]

I'll shut up now. As a rule template developer, I at least can build my templates to create the Items it needs and I've already added a ton of error checking to help them configure their Items. I suppose the UI widget developers are on their own.

No Rich, please don't. Your input is really really valuable.

Coming back to the suggestion for having as JSON type Item, cause JSON can easily pe parsed witin widget expressions. I agree, reading that data is easy for display, but how can this be used for editing data within the JSON object ? Wouldn’t you need more than one action ?

• Parse the JSON data into different vars
• Change the value of the desired var
• Build a new JSON from the vars
• Send new JSON to the Item as new state. AFAIK, this is not possible within widget code, otherwise, please prof me wrong.

Another thing missing would be some kind of magic to automatically create needed Items and rules when installing a widget from the marketplace. This would be a really big improvement for us widget developers (If I may count myself into that group).

[J-N-K]

"JSON is arbitrary, you don't need that in Rules DSL."

I didn't say that. I said that having a JSONItem will not improve the support for arbitrary data stored in JSON format because we still can't do what JS can do: use the . operator for accessing the child or [] for accessing array elements. If it's just about getting a single element from the then JSONPath transformation is your friend - you don't need a separate file for that and it's already available without adding additional code on our side. In fact it is very similar to your second example. There will of course - as you pointed out - always be corner cases. But if we come up with our own implementation we'll have those, too.

In fact what I did is speaking up against introducing features where we already know that they can't be supported by Rules DSL because it would be a PITA from the support POV if we add something as fundamental as a new item type - even if it is well known that I would prefer to kill Rules DSL and textual configuration.

[spacemanspiff2007]

are basically being told the problems we face aren't real problems.

If you read thoroughly through this thread than you'll see that nobody said that.

That's the same sort of thing that was said about Things back when 2.0 was being developed. I guess all that work wasn't worth it since copy and paste is so easy (yet so many still manage to screw it up).

Now you're comparing apples to oranges. Things introduced new features and new issues - some of which still aren't solved two (and soon three) main releases later. Here we're talking about something feature identical.

It's a pretty big deal for us though and we face it with distressing frequency.

I understand the issue however this can already improved now with a documentation update. You're great at helping users out in the forums but if it's not in the docs it's more or less a "lost effort". Because the next person will struggle again and has ask / search the forums, too. So how about a part in the docs "installing from marketplace" where there is a step by step guide on how to create items, how to set metadata, etc..

• from *.items file example
• from yaml example

The next time someone asks you can just post the link as a response. I also skimmed through your provided links and an improvement would be the separation of *.items files and the gui examples (a separate highlighting box) This is just meant as a hint and by no means an attempt to belittle your great work for the community.

I at least can build my templates to create the Items it needs and I've already added a ton of error checking to help them configure their Items.

Tbh. this should be possible through the configDescriptions and you shouldn't have to do the validation in the rule. There could be an additional entry metadata where you could provide e.g. a json schema. Maybe add an issue with a feature request?

[rkoshak]

So how about a part in the docs "installing from marketplace" where there is a step by step guide on how to create items, how to set metadata, etc..

from *.items file example from yaml example

I have that documented actually in my rule templates (see the time based state machine for example). And I still have to help people because they just blindly copy and paste the example, despite lots of bold warnings these are for illustration purposes only. It's not the creation of the Items themselves, it's the configuration of the Items. Helping them get the metadata right and setting up the relationships between the Items is where it falls down. It's not the creation of the Items, it's the creation of the right Items with the right configurations.

Believe me, I have tried to address this with documentation since that's all I really have.

As an aside, a YAML example for Items is a no go because MainUI still doesn't support a code tab for Items (though I've added that to @florian-h05's brain storming list).

I also skimmed through your provided links and an improvement would be the separation of *.items files and the gui examples (a separate highlighting box)

Unfortunately that's a limitation of the Marketplace entry parser. The second code block gets interpreted as being the rule template itself instead of just another example. It's quite annoying. I don't remember if an issue was filed for that but it's still an issue.

Tbh. this should be possible through the configDescriptions and you shouldn't have to do the validation in the rule.

We are already using configDescriptions. configDescriptions are what define the rule template and UI Widget parameters. So I'm already able to say "this option named "triggeringGroup" must be an Item of type Group" and in the configuration the end users will get an Item selector dialog showing only the Groups. I make heavy use of that already as do UI Widget developers. For example, the following is most of the form for my Open Door Reminder rule template.

But that only gets me so far. In the rule itself I still need to verify lots of these parameters (e.g. there's no config parameter to define a duration so I have to check that myself, the user may disable or remove the rule that this template calls, the Item metadata may be bad if present (in this case it's optional), the version of the third party libraries it depends on may be too old or too new, etc.).

For example, my Debounce rule template as currently written requires the users to:

• create a proxy Item for each Item they want to debounce
• add metadata to the "raw" Item linked to the sensor they want to debounce
• add the raw Items to a Group.

The configuration description lets the user select the Group and the name of the metadata namespace (using "debounce" as the default), and a flag to define openHAB startup behavior. I don't need to test that the triggering Item is a Group and I don't need to test that that flag is a boolean. But, in my rule I need to additionally test:

• is the version of openhab-js high enough?
• is the version of openhab_rules_tools high enough?
• do all members of the Debounce Group have "debounce" metadata
• is that metadata valid (i.e. all expected fields are present and usable)?
• do the proxy Items defined in that metadata actually exist?
• are there any Items with "debounce" metadata that are not members of the Debounce Group? (this is just a warning, not an error)

I'm not sure how any of those could be handled with a configDescription given it's not a one-to-one type check. I'm open to the ideas though. For example, it would be wonderful if I could define that a given parameter is:

• a Group Item with a type and aggregation function
• all members of the Group are the same type or compatible (e.g. Switch, Dimmer, and Color, Number and Dimmer, etc.)
• all members of the Group have a given metadata (name of the metadata needs to be defined by one of the parameters)
• all members of the Group's metadata is valid:
  • required values and configuration parameters are present
  • all values are valid (e.g. Item exists, matches a REGEX, parsable to a number or boolean or date or time, etc.)

It would be great to have. a wizard that lets a user create the Group and add the Items they want to that Group right there in the rule template instantiation. It would be awesome if that wizard would then guide the user through the creation and addition of the required metadata, using configDescription to define the form used for this.

For a rule template like Time Based State Machine, it would be awesome if as part of the template configuration there was the ability to "Add Times of Day Set" where a user could create and configure Items and the metadata for all the DateTime Items for a given day type right there in the form. Then when instantiating the rule, the user will get a configDescription defined form to help them enter the metadata for each and maybe even initializing the state of the Item right then and there.

I'm not naive enough to think these are fully possible or, if possible would take way too much work to implement. But if we can take baby steps I'd be overjoyed. Even if we had the ability to bundle multiple rule templates and UI Widgets together, we could probably figure out how to do all of the above ourselves.

[splatch]

I really beg you to allow us to do something like that pray. I am actually working towards exactly since weeks with all the metdadata implementation I did.

I have nothing against way you do it. Given advertisement of different approaches I'll outline that we won't have proper handling of schedules unless they become part of core framework. I was pointing back in past that this it is a missing spot which could be filled in: https://community.openhab.org/t/flexibility-of-framework-supplied-types/89443.

I did a prototype back in 2020: https://github.com/ConnectorIO/openhab-schedule, pushed it a little bit further over last christmass for bacnet binding: https://github.com/ConnectorIO/connectorio-addons/tree/de4c7debcd327cbc7eee035eb7d1764f753df0ed/bundles/org.connectorio.addons.temporal/src/main/java/org/connectorio/addons/temporal which brings schedules into system.

A primary use of schedule in bacnet is to use state of one item (schedule) to control state of another item.