RFC: Change Number item type for improved UoM handling

[J-N-K]

There have been various discussions (#3167, #3247, #3248, #3258, forum about the handling of UoM in Number items. Let me first summarize the current state, then the proposed state and finally the migration path for existing installations:

The 3.x situation

There are two "types" of Number items: plain Number and Number:dimension (like Number:Temperature). In contrast to the documentation they both accept DecimalType and QuantityType state updates and commands. The user needs to know the correct dimension, which is very likely the case if we talk about length or temperature but maybe not so obvious for other cases (take a second and ask yourself if you know the correct dimension for lx, in German it's "Helligkeit" which has more than ten different translations according to dict.leo.org and Google translator returns a wrong translation for our use-case).

State updates

State	Number	Number:Temperature
DecimalType	as-is	add default unit (see below)
QuantityType	strip-unit	as-is (if unit compatible and same measurement system)
		convert (if unit compatible and other measurement system
		reject (if unit incompatible

Command handling

Command	Number	Number:Temperature
DecimalType	as-is	as-is (if linked to a channel with accepted item-type Number)
		add default unit (if linked to a channel with accepted item-type Number:dimension)
QuantityType	as -is	as-is

Persistence

Persistence only stores plain numbers

State	stored	restored
DecimalType	as-is	as-is
QuantityType	convert to default unit and strip unit	add default unit

The default unit is determined (in that order)

• from the state description
• from the list of system default units

If the state description contains %unit% the unit is used as provided by the binding.

The 4.x proposal

The Number:dimension item type is removed. Instead a new metadata unit is introduced. Setting unit enables the item to handle UoM. The user does not need to care about the dimension, he just configures a unit he expects for this item which is much more convenient for those of us that are non-native speakers and don't deal with units in foreign languages in professional work.

State update

State	Number without unit	Number with unit
DecimalType	as-is	add configured unit
QuantityType	strip-unit	convert to configured unit (if unit compatible)
		reject (if unit incompatible

Command handling

Command	Number without unit	Number with unit
DecimalType	as-is	add configured unit
QuantityType	strip unit	as-is (if unit compatible)

Persistence

State	stored	restored
DecimalType	as-is	as-is
QuantityType	convert to configured unit and strip unit	add configured unit

Examples

ItemStateEvent	unit= metadata	stateDescription	internal value	persisted value	restored value	displayed value
5.0	none	none	5.0	5.0	5.0	5.0
5.0	none	%.0f	5.0	5.0	5.0	5
5.0	none	%.0f °C	5.0	5.0	5.0	5 °C
5.0	°C	none	5.0 °C	5.0	5.0 °C	5.0 °C
5.0	°C	%.0f K	5.0 °C	5.0	5.0 °C	278 K
5.0	°C	%.0f %unit%	5.0 °C	5.0	5.0 °C	5 °C
5.0 °C	none	none	5.0	5.0	5.0	5.0
5.0 °C	none	%.0f	5.0	5.0	5.0	5
5.0 °C	none	%.0f °C	5.0	5.0	5.0	5 °C
5.0 °C	°C	none	5.0 °C	5.0	5.0 °C	5.0 °C
5.0 °C	°C	%.0f K	5.0 °C	5.0	5.0 °C	278 K
5.0 °C	°C	%.0f %unit%	5.0 °C	5.0	5.0 °C	5 °C
5.0 K	°C	none	-268.15 °C	-268.15	-268.15 °C	-268.15 °C
5.0 K	°C	%.0f K	-268.15 °C	-268.15	-268.15 °C	5 K
5.0 K	°C	%.0f °C	-268.15 °C	-268.15	-268.15 °C	-268.15 °C
5.0 K	°C	%.0f %unit%	-268.15 °C	-268.15	-268.15 °C	-268.15 °C

The "internal value" is used in ItemStateChangedEvent and (if accepted ItemStateUpdatedEvent).

The migration

Plain Number items

There is nothing to to for plain Number items. They work like they did before.

Managed Number:dimension items

With #3268 (which could also be run automatically on upgrade, in contrast to what is written in the PR) the dimension could be removed from the item-type. If a dimension is found, the unit metadata is added. In case a state description is present that contains a unit, that unit is used, if not, the system default unit is used. IMO an automatic conversion is desired here, because the user expects plug-and-play for managed entities.

Unmanaged (textual) Number:dimension items

This needs to be discussed. @spacemanspiff2007 offered to provide a script that can be run during update to offer a similar functionality. The question is whether this should be run automatically or on demand, because usually users of textual configuration do not expect any changes to their configuration during upgrade.

UI

UI needs to be adapted to only present Number instead of a list of many items. It should show an additional field unit on creation only, later the unit should be editable like other metadata.

The decision

@openhab/maintainers and @openhab/architecture-council:

Since this requires quite some refactoring, I would like to hear your opinions on that before I start the implementation. IMO the new proposal is easier to understand and much more consistent than the current situation. It allows decoupling of internal state and presentation (because state description is no longer taken into account for determining the unit). To allow extensive testing we should reach a decision here within the next three weeks (i.e. until 2023-01-15) and have the PR merged latest 2023-03-31.

[spacemanspiff2007]

Based on our logic we assume the specified unit when no dimension is provided from the channel. To be consistent it should also be possible to send a value with a dimension to the channel and the dimension should then be stripped silently.

This could be mitigated by adding a system:strip-unit profile that user can add to support such links.

I think this profile should be available non the less. E.g. if there is a buggy device that sends the wrong unit (e.g. 0 °F instead of 0 °C) The user can then strip the unit and use the unit to °C with the unit=°C metadata. Also I think a profile that will force the unit to a certain value will also be a good idea for completeness (e.g. system:force-unit)

Only after I wrote this I realized you probably meant in the outgoing direction (e.g. item -> thing). Aren't all profiles currently inbound (a.g. thing -> item). Because in this case the direction is vital.

[J-N-K]

Profiles work in both directions, the standard profile forwards state updates and commands from the handler to the item and commands from the item to the handler.

[rkoshak]

Shall we keep this logic or remove it?

Only considering this from the end user's perspective, it feels off to require the users deploy a profile to fix a limitation of the binding. If we are to allow a user to link a Number:dimension Channel to a Number and vice versa I'd say we should make that work as easy, transparently, and consistently to the end user as possible.

So my vote would be to keep it as is, but that doesn't account at all for the complexity involved in keeping that "fix" in the CommunicationManager. And it's not a strong vote; it's not something I'd fight for.

[spacemanspiff2007]

Profiles work in both directions, the standard profile forwards state updates and commands from the handler to the item and commands from the item to the handler.

The question is do they only process state updates or also commands. I think my question becomes more clear if we use the offset profile. Thing reports value 10, offset adds 10 so item has now state 20. If I send a command 30 to the item and the profile is truly bidirectional the thing should actually receive a command 40. So bidirectional in a sense that it gets applied both ingoing and outgoing and not that it passes through. This might be important for setting and stripping item units but probably is discussed best in a different issue.

I think for now if we have a profile to strip the unit and a profile to set the unit that'll be a good start.

[rkoshak]

The question is do they only process state updates or also commands.

From what I've seen they can process either or both updates and commands based some of the implementations I've seen creating custom profiles using rules languages (I've seen Jython and jRuby, I once tried to make one using Nashorn JS but gave up). And the Profile can have some "memory" or else the button profiles wouldn't work (can't detect a double press if you can't remember that first press).

So they can process in both directions I believe and what they do depends on the profile developer. I could be totally wrong though and it's been awhile since I've looked into that.

[ccutrer]

That's correct. At the Java level, a new profile instance is created for each channel<->item link with said profile, and there are 4 methods - one for update, one for command, for each direction - that need to be implemented: https://www.openhab.org/javadoc/latest/org/openhab/core/thing/profiles/stateprofile. (Note also there are StateProfiles and TriggerProfiles).

The ScriptProfile can have both a toHandlerScript and a toItemScript configured, so that you can handle each direction appropriately (note that onStateUpdateFromItem is ignored and not passed to the toHandlerScript, the same as the default profile).

As for persistence (like the button profiles), I'm not sure if that will work with ScriptProfile or not. And ScriptProfiles definitely do not get access to the additional state a pure profile gets (https://www.openhab.org/javadoc/latest/org/openhab/core/thing/profiles/profilecontext), such as the per-link configuration, or the callback object (https://www.openhab.org/javadoc/latest/org/openhab/core/thing/profiles/profilecallback) which provides access to the link itself in order for the profile to adjust based on something like metadata in the item.

Shameless plug: the Ruby helper library allows you to implement your own profiles, with access to all of the above: https://openhab.github.io/openhab-jruby/main/OpenHAB/DSL.html#profile-class_method.

[spacemanspiff2007]

Quote from @J-N-K from the linked issue which should be discussed here instead of the linked issue:

The dimension will not be stripped from the item, if we do that, then UoM improvements will not make it into OH4. It requires too many refactoring also in openhab-addons.

Advancing this feature has already been very slow because of missing feedback of other maintainers and the many misconceptions around it. Maybe it makes sense to push back the release of OH4 by a couple of weeks? That way there would be more time and this feature could be implemented properly. Maybe there is someone we could additionally ask for help?

[jimtng]

Pardon me, I haven't been following this. Just curious what are the misconceptions? To clarify, as I understand it, this pr/proposal will remove the dimension from the item?

[spacemanspiff2007]

I would very much like that but the current state of the PR introduces two fields that are used to determine and configure the same thing. We will have an item type Number:Length which indicates that this item is a UoM item with the dimension length. Additionally we will have metadata unit="m" which also indicate that his is a UoM item with the dimension length and that the internal and external value representation of the item value should be in m. The important part of the definition is unit="m" because that's what will be used to report the value to persistence and that's also the unit and scale that will be assumed if no unit is provided (e.g. in rules, channels, persistence and the rest API) All desired features can be implemented only with unit="m" but @J-N-K is hesitant because it's lots of work someone has to do and we're already on M2 for OH4.0 (to be clear it's a valid concern and I am in no way blaming him in any way).

The misconceptions I mentioned are mainly because the originating issues and this issue are a huge wall of text now and it's a mixture of how things are now, how things should behave in rules, how things should be represented in the UI and a little bit of how things should behave in the future. Additionally there is now a differentiation of internal/external value representation (what is defined with metadata unit) and value representation used in gui/charts (defined in the state description). So now it's possible to persist a value in feet (metadata unit="ft") and display it in cm (state description "Lenght [%.1f cm]").

It's a much more a complex topic than one would think at the first glance.

[lolodomo]

It is important IMHO to propose a solution that will be transparent for existing bindings and will not require a refactoring of around 500 official bindings!

[jimtng]

Would it work if we kept the dimension and simply add the unit metadata?

Items without the unit metadata continue to work like before.

Items with the unit metadata behave as proposed.

[J-N-K]

@jimtng That's exactly what is implemented in #3481 (except they will use the system default instead of the state description unit, because we decided to decouple that).

[jimtng]

And what if you don't change the default behaviour when no unit metadata is defined, and continue using the state description, at least in that pr.

Aren't the two unrelated?

[spacemanspiff2007]

They are unrelated but you have to pick a metadata unit when creating an uom item. The whole idea is to make the internal representation "static", so persistence etc. will not be broken if you change the state description.

@J-N-K : You make it sound like it's a dynamic fallback to the system default. I thought it's a one time migration where the metadata will be added to the item accordingly?

It is important IMHO to propose a solution that will be transparent for existing bindings and will not require a refactoring of around 500 official bindings!

@lolodomo As I outlined above the dimension of the number item does not add anything functionality wise. On the contrary - it even obfuscates that the important part is metadata unit. So keeping it around is both confusing to the end user (why do I need to set it to Temperature if I set it to °C) and to developers (if metadata is the important part why is there dimension Temperature) and will definitely lead to code issues down the road.

We've been talking about this issue since November and there would have been enough time to think of a good solution how the migration pain could be mitigated. @J-N-K asked for guidance in December on how/if to proceed but unfortunately he was left hanging. This will be one of the most important features for OH4 because it will simplify and streamline so many things and it will probably have the biggest impact for the users. Will the it be better than what we have currently? Yes - definitely. Will it be as good as it can be? Definitely not. It'll be patched on and it will show.

[J-N-K]

It seems no consensus can be reached and keeping everything as it is now is the most important thing. So let‘s close this. It‘s sometimes confusing how out UoM behaves, but in most cases it seems to work.

[spacemanspiff2007]

It seems no consensus can be reached

Is that a hint in my direction? Because I especially stated above that the PR will be better than what we have now.

[J-N-K]

I have spent far more than 100 hours on discussions in issues (this one and related) and the creation of the associated PRs (#3248, #3268, #3367, #3481 and additions in openhab-addons and openhab-webui). Still it seems that no one is satisfied and we are discussing the same issues over and over again (and sorry, that also includes you) and every change is considered a bad thing. I'll not again be the bad guy that "breaks everything" and "recklessly moves forward much too fast" like with the Java 17 change.

Maybe we can give it another try for OH5 in some years.

[spacemanspiff2007]

(and sorry, that also includes you)

No need to say sorry because it's a fact. Obviously I'm also heavily invested and want OH to have good concept so it can have a sustainable development. Nonetheless I understand that you are frustrated because I would be (and am) too. Anyway - thanks for you effort.

we are discussing the same issues over and over again

It's time consuming to read the whole history and as I stated it seems (almost) nobody is willing to read and understand the issue, yet has an opinion and makes suggestions. That in evidently leads to discussions which run in circles. Imho this issue should have been at some point a call/conference which freezes the concept.

At least we now have a decision and can move forward from that.

I've been in limbo what to do about the UoM items in HABApp but now I have something I can work with.

Maybe we can give it another try for OH5 in some years.

Let's see if we'll still be motivated and have the time and energy. 😉

[lolodomo]

iMHO, last @J-N-K proposal keeping dimension was a good compromise, it was not breaking current setups too much and if my understanding is good, it was not requiring a code migration in all existing bindings. And it was apparently adding the features requested by few advanced users.

The question is not whether we want or not to move. Just be pragmatic. If a change in core framework would break all exishing bindings (I suppose removing dimension from Number will?), who will be volunteer to fix all of them? Probably neither @J-N-K nor any existing addon maintainer, this will be done only on well maintained bindings. This is acceptable when the changes are bringing a lot of new things like when we moved from OH1 to OH2. Doing that just for the fun to remove dimension from Number, I am not sure at all it's worth the required effort. Just my opinion.

[mherwege]

I have been following this discussion for quite a while and looked forward to the outcome of it. I think the proposal was a good compromise. While I do understand all the arguments for dropping the dimension, I do see dropping it has a major impact on bindings. And it will require more explaining to the existing users (even it is a simplification). I do not consider doing anything at all an option. At least a separation between default unit and unit as conveyed in a state description should be made. It has hunted me a number of times and is difficult to explain/understand to someone that is less in the details. I still believe the logic: dimension has a default unit and the default unit can be overriden in the state description for representation is easy enough to understand, even when dimension is not strictly required. But the current logic: dimension's default unit is a bit of black magic based on the presence or not of state descriptions, is not acceptable. I call it black magic, and I do unstand there is clear logic behind it, but it is not always easy to understand the consequences of changes. So I would vote for implementing @J-N-K proposal.

[J-N-K]

@lolodomo The question is exactly that: If nothing can be touched without people complaining, then the decision is not to move. And when I originally suggested to remove the dimension from the item and channel (#3367) and let it automatically determine the correct dimension by the unit, I also added a PR in addons (https://github.com/openhab/openhab-addons/pull/14389) that would have solved far more than 90% of all issues, except those that use dynamic channel types (channel types not channels!).

@mherwege It would have been good to have read some kind of support for that proposal before. All I read is either "that's too much" or "that's not enough". Feel free to pick up #3481, maybe you'll be more successful in finding someone who reviews and merges it.

[spacemanspiff2007]

And it was apparently adding the features requested by few advanced users.

This is again a misunderstanding. The PR was a fix for an issue that affected users who used UoM items and persistence (or rest api events). If you used those you were affected and if things worked as expected it was just by chance. This is by no means a feature for super professional users (and I should know since I've requested those) but rather a feature that would have affected almost all users because it would have brought huge simplifications. This is why I think such a huge change would have benn justified because it would have been a huge gain (e.g. this would also have simplified the use of units in dsl-rules, etc.).

I call it black magic, and I do unstand there is clear logic behind it, but it is not always easy to understand the consequences of changes.

There's always one thing you can do: Not use UoM items. UoM provides benefits when switching units/scale between states but it provides nothing that can't be done with a rule and a proxy item (albeit not as elegant). With this PR closed UoM is dead for OH4 and I can (and will) only recommend any user not to use it because the dangers outweigh the benefits by a huge margin.

[rkoshak]

As someone who fights with UoM problems more than anything when helping users in the forum I'm very much saddened to see it come to this. Fighting with UoM "black magic" as @mherwege is probably the number one issue even advanced rules writers have to fight against.

I thought we had come to a good compromise too with keeping the dimensions part of the Number Item type and adding the unit metadata. I was just waiting for a review, a merge, and a party. This was a huge missed opportunity I think and it's very unfortunate.

[lolodomo]

Except @spacemanspiff2007 , I understand all others discussing here were ok with @J-N-K proposal. So why not validating @J-N-K 's simplest option?

[J-N-K]

@jimtng Also has a different opinion, from what I read above. Even after reading the whole issue again, I did not find the broad support for my proposal that is now expressed. If we agree not to discuss the concept and concentrate on the code, I can re-open the PR.

[lolodomo]

@jimtng Also has a different opinion, from what I read above. Even after reading the whole issue again, I did not find the broad support for my proposal that is now expressed.

Ok, I haven't read all the discussion again. So maybe you re right and there are more disagreement with your proposal than I imagine.

[splatch]

Well, my points weren't taken into consideration (making clear separation of Decmial vs Quantity types), as I think it will sooner or later become a problem. it is part of problem right now, however way you go is up to you. I am not a core contributor, actually I am not a nominated contributor to any of repos, so I'm perfectly fine if you treat my voice as an independent observer and ignore it with final implementation. ;-)

[rkoshak]

If we agree not to discuss the concept and concentrate on the code, I can re-open the PR.

I've been satisfied with the concept for some time, which is why I've not been commenting (perhaps I should have been more vocal in my support). Keeping Number:Dimension around doesn't give me too much concern (would it be better to get rid of it, or create a new QuantityType Item, perhaps, but that's not where users are having problems in my experience and the disruption across repos could be huge). Separating the unit of the Item from the state description makes me very happy. Making the unit consistent in all the places the Item is used (REST API, persistence, etc) makes be very happy. The rest (as far as I can tell) are arguments over implementation details which as a non-developer/maintainer on OH I don't have an informed opinion to provide.

And even were I not happy with the current approach, it's a heck of a lot better than what we have now. I'd rather see that and readdress the remaining concerns for OH 5 than abandon all attempts to make UoM work better.

[J-N-K]

@splatch Regarding data types DecimalType and QuantityType are already separated. IMO we should probably add some sort of deprecation warning if the "implicit conversion" logic (i.e. adding/stripping units) is called. Users can then add the UoM profile (also in #3481) to the link if necessary and we allow for a smooth transition when we remove these implicit conversions later. This seems to be a much less disruptive approach than breaking these conversions immediately (IMO that should never have been introduced at all, but that decision has been made in OH2 days).

[ccutrer]

I've been satisfied with the concept for some time, which is why I've not been commenting

This has basically been my position for some time as well. I've only been tangentially following, since I don't have enough time to get super involved with this. I figured once it all shakes out, I would make any changes (if necessary) to improve support for the new approach in the addons I maintain.

[splatch]

@splatch Regarding data types DecimalType and QuantityType are already separated. IMO we should probably add some sort of deprecation warning if the "implicit conversion" logic (i.e. adding/stripping units) is called

@J-N-K My point was that same separation should follow up with NumberItem and QuantityItem. Then unit handling would be specific only to QuantityItem. Personally - effectively any improvement, even if its partial, its better than what we have now with state description driving everything. From my point of view, while I consider problem still be present, its good to go as-is and make a follow up changes and clarifications after 4.0. I understand necessity to provide smooth transition, even if it will double the pain, but span it over longer time.

[J-N-K]

But that is only an implementation detail, not a user-facing change. If we have a clear separation Number is for DecimalType and Number:.... is for QuantityType we can refactor the code so that Number is handled by NumberItem and Number:.... is handled by QuantityItem. This is nothing the user will ever notice, so it's far less critical.

I'm far less concerned about changes that require developers to adjust something than changes that require the end-user to adjust something. If we change that in core, most add-on developers (except probably scripting and persistence) will not notice.

[rkoshak]

IMO we should probably add some sort of deprecation warning if the "implicit conversion" logic

I really like that! If you add a warning log statement, users will freak out and move mountains to get rid of it (especially if it happens a lot like on every update/change to the Item). If the warning tells them what to do to fix it, so much the better! :-D

[jimtng]

I'm fine with the latest pr keeping the dimension. Tbh I don't know what's best so I was just asking questions trying to understand. I'm sorry if it came across as being negative.

I think the general consensus, including my own is to support the proposal.

[spacemanspiff2007]

Except @spacemanspiff2007 , I understand all others discussing here were ok with @J-N-K proposal. So why not validating @J-N-K 's simplest option?

@J-N-K is doing all the work so he gets to say what he is implementing. I reported the initial issues and have a pretty good understanding what the feature (imho) should look like however I am not the one to decide what and how. I've tried to make my arguments clear countless times.

Besides - and correct me if I am wrong - even @J-N-K thought that it would be a good idea to remove the dimension from the NumberItem (see #14389). However some people made some of out of context claims or made misleading statements because the misunderstood the issue/PR which finally caused him to backtrack and close the PR.

The problem with this issue has never been the lack of feedback and opinions, it has been the lack of qualified feedback. It's a reoccurring theme if you skim through the issues and PR and you'll often read "I didn't fully understand the issue, but ..." or "I haven't read all the issues, but ...". Obviously it has been the main reason why we are running in circles and I would have expected someone from the openHAB core maintainers (don't we have some volunteers for architecture, not sure) to provide feedback on the core concept - even if it would have been only a "I don't fully understand it - do what you think is best". Instead this issue and the concept has been left in limbo so even more opinions accumulated.

So now the confusion is perfect and we're already behind M2 and time is running out. Yet trying to bring clarity still feels like the fight against windmills. Even I don't know the current state of the PR any more and if it aligns to a consistent user experience.

However pulling at @J-N-K to steer him in my envisioned direction has been very exhausting both for me and him because I'm not the only one pulling him. So I'll stop meddling with him and this issue because after creating the issue and countless comments trying to get everyone onboard now I am the one preventing a solution? Nothing can be further from the truth.

[mhilbush]

@J-N-K the time you spent on this (and other) PRs is not lost on me. I very much appreciate your contributions.

I thought this PR had reached a good point, which is why I remained silent. Anything than can make the internal representation consistent while allowing the user to define the external representation is a step in the right direction. It's also a good thing that this PR minimizes the impact on bindings.

My earlier post simply pointed out that when it comes to dealing with units, the time-consuming part for most users is when writing rules. And I thought it was important to point out that this PR will not change that much, if at all. I hope that was not considered misleading or out of context.

It would be good to get this finalized and merged soon as we are getting pretty deep into the release cycle for 4.0.

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/dimensionless-state-description-pattern-unit-vs-rpm-ppm-etc/148723/5

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/dimensionless-state-description-pattern-unit-vs-rpm-ppm-etc/148723/6

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/how-to-change-unit-of-quantity-types/150772/4