Introduce `js-quantities` for units of measurement handling

florian-h05 commented 1 year ago

I propose to add the js-quantities library to openhab-js, because:

We need a good unit of measurement handling in the JS layer to work with UoM returned from the Java layer
Blockly would heavily profit from js-quantities, as we can have something like: 1 W + 1 kW and it calculates 1001 W
It seems to be a much requested feature in the community (I can't provide links, but I guess others can)
js-quantities has NO dependencies and is only 30 kB bundles, so we don't have to loose anything if we add it
js-quantities covers many units (if you know all of them, respect), see https://github.com/gentooboontoo/js-quantities/blob/master/src/quantities/definitions.js

/cc @digitaldan @jpg0 @rkoshak @stefan-hoehn @ghys

stefan-hoehn commented 1 year ago

Based on a design session just now with florian, we came up with the idea that we then could provide blocks that can do the following

stefan-hoehn commented 1 year ago

So @digitaldan @jpg0 @rkoshak if you give us a thumbs up we would further proceed with a Proof of Concept for that.

By the way, we even found a way to provide blocks that would only available if jsscripting is installed which allows us to provide more functionality for those who do have installed jsscripting.

stefan-hoehn commented 1 year ago

Sorry, we could't stop it - it was just to tempting. It already works on our machines 🥇

generates

and delivers

var ctx = this;
var runtime = require('@runtime');
var itemRegistry = runtime.itemRegistry;
var events = runtime.events;

console.info((Quantity('1W').add('1 kW')));

2022-12-20 20:53:19.634 [INFO ] [tion.script.ui.test_jsscripting_math] - 1001 W

or

2022-12-20 20:56:43.586 [INFO ] [tion.script.ui.test_jsscripting_math] - 1001.001 W

Intuitively we used other units as well and with a bit of more tweaking the code even this works almost out of the box

(though temperatures are pretty nasty on absolute and relative values ;-) )

rkoshak commented 1 year ago

This solves #102. I am very much in support of this.

I do recommend setting up a pretty extensive set of unit tests around this though to make sure that we can handle all the usual combinations of modifiers and such. For example density can be something like kg/m³. That's probably the most complex unit I can think of since it combines a modifier k, weight g, ratio /, length m, converted to a volume ³. It's not clear from scanning the code that the JS library supports that.

It'll be awesome if it does, and we need to document the limitations and provide an alternative if it doesn't.

florian-h05 commented 1 year ago

I do recommend setting up a pretty extensive set of unit tests around this though to make sure that we can handle all the usual combinations of modifiers and such.

js-quantities already has a large set of unit tests (1600 lines!!): https://github.com/gentooboontoo/js-quantities/blob/master/spec/quantitiesSpec.js.

It'll be awesome if it does, and we need to document the limitations and provide an alternative if it doesn't.

You are right, we need the docs for that, but first of all we need the implementation. (See https://github.com/florian-h05/openhab-js/tree/js-quantities for my current implementation.)

rkoshak commented 1 year ago

js-quantities already has a large set of unit tests (1600 lines!!):

I'm more concerned with testing the conversion between OH QuantityType and this library. If there isn't a one-to-one conversion between all the OH units we need to know that. And off it's not possible to convert between them, I'm not sure I get the point.

florian-h05 commented 1 year ago

I'm more concerned with testing the conversion between OH QuantityType and this library. If there isn't a one-to-one conversion between all the OH units we need to know that.

@rkoshak @stefan-hoehn Based on the list of units at the openHAB UoM Docs, I created over 170 unit tests that check for each unit that is listed in the openHAB docs whether js-quantities accepts them: https://github.com/florian-h05/openhab-js/blob/js-quantities/test/quantities.spec.js

For example density can be something like kg/m³. That's probably the most complex unit I can think of since it combines a modifier k, weight g, ratio /, length m, converted to a volume ³. It's not clear from scanning the code that the JS library supports that.

I added a test for this as well, js-quantities doen't throw an error which means that it accepts this as a unit.

I polyfilled support for some unit notations, because js-quantities for example doesn't accept VAh but VA*h (Volt-Ampere hours) instead. I'm not sure if those unit modifications are correct in terms of physics, would be great if one could have a look at them: https://github.com/florian-h05/openhab-js/blob/js-quantities/quantities.js

There is a small list of units that js-quantities doesn't support (or they are in the wrong unit notation): Minute angel is unsupported as well. Means: 6 of 71 SI units listed in the openHAB docs are unsupported, which is quite a small number.

IMO those are not very common, so I don't see a problem here. We should note that they are unsupported, but I don't think that many users will miss them.

As a side-note: Looking at the docs, I am wondering whether some units aren't written wrong:

Energy > Volt-Ampere Reactive Hour is varh bit IMO it is better written as VArh
Power > Volt-Ampere Reactive is var but IMO it is better written as VAr

rkoshak commented 1 year ago

I'm not sure if those unit modifications are correct in terms of physics, would be great if one could have a look at them: https://github.com/florian-h05/openhab-js/blob/js-quantities/quantities.js

I don't either. They feel right but I'm no phycisit.

IMO those are not very common, so I don't see a problem here. We should note that they are unsupported, but I don't think that many users will miss them.

Agreed but we should also document how to work around that by using the raw Java Objects in this one case to convert down to a plain number or do math using QuantityType.

Looking at the docs, I am wondering whether some units aren't written wrong:

Most but not all of the units come from an upstream library. These are common enough I would guess they are that way in the upstream. I don't know why. I agree, volts and amps are usually capitalized. Maybe QuantityType isn't case sensitive?

stefan-hoehn commented 1 year ago

The only unit I may feel could be missed is the Deutsche Härtegrad which is e.g. provided by decalcification system. However, I have no clue how many of these actually have a smart interface that is used by any openHAB user and even then, how many would actually compute values with those. So, most likely we can omit that, too.

maniac103 commented 1 year ago

Power > Volt-Ampere Reactive is var but IMO it is better written as VAr

var is correct though: https://en.wikipedia.org/wiki/Volt-amperes_reactive ... Same for varh.

florian-h05 commented 1 year ago

@maniac103 Thanks! I think I will check Wikipedia myself to verify whether my unit modifications are correct: https://github.com/florian-h05/openhab-js/blob/js-quantities/quantities.js

@stefan-hoehn Regarding Grad Deutscher Härte: Based on what is written down there, it seems like we could convert it to something that works: https://www.internetchemie.info/chemie-lexikon/d/deutscher-haertegrad.php

florian-h05 commented 1 year ago

After thinking a little bit about the physics, I'm pretty sure that my additional unit notations (https://github.com/florian-h05/openhab-js/blob/js-quantities/quantities.js) translate correct:

Given that P = U * I, we know that 1 W = 1 VA
Given that 1 Joule = 1 Watt * second, we know that Ws is W*s and VAh is VA * h and varh is var * h

dBm as dB*mW is also correct, since the following code always prints 0.03 W:

console.log(Qty('30 dBm').to('W').toString())
console.log(Qty('30 dBmW').to('W').toString())
console.log(Qty('30 dB*mW').to('W').toString())

florian-h05 commented 1 year ago

@rkoshak @stefan-hoehn I have polyfilled support for Deutscher Härtegrad, Dobson Unit, Minute Angle and Second Angle. The only two missing units are octet for DataAmount and mired for Temperature.

I have added some extensive unit tests to also verify that my polyfilled units do the correct math.

digitaldan commented 1 year ago

I propose to add the js-quantities library to openhab-js

This sounds great, i don't work much with UoM , but i think anything we can do to make this easier would be great. Sounds like the library (and all the tests @florian-h05 wrote!!!) works with a large sample of common UoM patterns in OH? If so this seems like win to me.

jpg0 commented 1 year ago

I left a comment on the PR which should probably be here instead:

@florian-h05 I think that it's a great addition to incorporate unit handling. The main question I have is why it's not possible to use the native openHAB logic to do this? Presumably it also needs to handling parsing strings into unit representations? It seems weird to have the core handle this, but also include additional logic in a JS library to do the same thing (or similar, which isn't great). It also means that other scripting languages don't benefit.

I'm guessing that there are fundamental reasons why this cannot be handled in core, but was interested to know why?

florian-h05 commented 1 year ago

I haven't really checked, but in theory it should be possible to construct a new QuantityType from a given string like "1.56 km", but I see the problem that we would need to wrap the QuantityTypes to avoid that the end user has raw Java objects in his JS environment. Given that, I thought that it is probably better to introduce a straight-forward JS library to handle quantities, and comparing the functionality of openHAB Core's QuantityType and js-quantities, they provide nearly the same functionality. FYI: For working with the Java time package, we also introduced a JS library (js-joda), js-joda's ZonedDateTime is converted to a Java ZonedDateTime inside the addon. The same should be possible for js-quantities and QuantityType, so that you can create a js-quantities Quantity and send it to some core API that needs a QuantityType.

jpg0 commented 1 year ago

I remember having lots of conversations about this kind of thing when I first started building the JS libraries. I certainly don't want to slow down the development of the scripting libraries, and I do expect each language to have it's own libraries, styles and idioms and hence use different libraries. Over time I would hope that core, shared functionality does migrate into core though, as otherwise having implementations for each language + for core, for each capability like this could turn into an ongoing source of bugs and frustration.

My belief is that the pattern should be to bring in 3rd party libraries precisely like this to accelerate the development of scripting libraries, but have a plan (and no blockers) to allow longer-term transition to cross-language implementation in core. I'm not sure exactly what this looks like from a technical standpoint (for example it could be auto-generated wrapper code in each language to avoid handling raw Java types, or polyfills that translate dynamically etc.)

Saying this, I'm not completely against the multiple-implementations approach, or even a mixed approach where we move functionality into core iff it is causing problems. Ultimately I think that we should make a conscious decision on the overall approach to this general problem, as it seems that we are gradually slipping into an approach without making a specific decision (and I have no idea on the views or progress of the other scripting languages). I would love to get the opinion of @kaikreuzer on this question.

florian-h05 commented 1 year ago

I don't see the problem in core: IMO core provides all the functionality that is required to work with quantities/units.

My big problem with using QuantityType from core, is that this would require us to deal much with the Java types. If the problem with the Java types is easy to solve and requires no extensive wrapping, we can directly use QuantityType from openHAB Core. What I mean by dealing with types is, for example you have a Quantity and want to convert it to another unit. If the toUnit method of QuantityType accepts a simple string like km/min, it's okay. But if you need to provide a javax.measure.Unit to convert a QuantityType to another unit, I am not happy with it.

jpg0 commented 1 year ago

@florian-h05 I haven't used QuantityType's so I don't really have any experience in the area. Usually these things are possible to make work in cases like this - for instance my generating all the wrapping JS code based on the Java code (maybe even requiring annotations on the Java code etc). I agree that having to handle raw Java types would be a deal-breaker.

florian-h05 commented 1 year ago

It might be possible to use the QuantityType with (extensive) wrapping. I understand your concern that each scripting language will have it's own APIs when introducing libraries for cases like the one here, but IMO even using the openHAB Core functionality does not guarantee that the APIs are the same because most languages will have to wrap them, and when wrapping there might be some additional functionality or different method names etc. I can have a look at wrapping QuantityType.

jpg0 commented 1 year ago

does not guarantee that the APIs are the same

I am not suggesting that the API presented to script writers are the same, rather that they wrap the same logic. I doubt that manual wrapping is a good idea for a large API, because they are likely to diverge as logic ends up in the wrappers.

Saying this, if it's possible to retain the API of the imported library if you do create wrappers, there's no need to do it yet.

florian-h05 commented 1 year ago

I doubt that manual wrapping is a good idea for a large API, because they are likely to diverge as logic ends up in the wrappers.

IMO that's a reason to use a native JS library and translate it's objects to their Java equivalents when sending to the Java APIs. I agree that we then don't use the same logic as the other ones, but as long as the logic is technically/mathematically correct, I don't see a problem here.

digitaldan commented 1 year ago

So i have been flip flopping my opinion here quite a bit, and this is likely due to not using Quantity Types much in my own setup. That being said, i initially assumed this was similar to our us of js-joda, wrapping the java time api was going to be awful, in that case both were based on a common ancestor (joda time), so it was a good choice as they had a similar api.

That being said, i was just going through the QuanityType source code started to think it might be not as bad as i thought to wrap, but looking a bit more, I'm concerned about the use of javax.measure.Unit which it exposes publicly on quite a few methods, i have not looked much into this JSR, but i'm assuming its huge (like most java things) and might be very hard to wrap in a maintainable way. I also could be wrong here, i need to dive in a little deeper.

If wrapping the openHAB api is not possible, then ideally it would be nice to shim js-quantities to feel like the openHAB api, so we could eventually change the core to be more scripting friendly (or at least as much as possible). And if thats not possible , then i think there is still merit in including this in, as the alternative is to use native methods which is not nice.

rkoshak commented 1 year ago

I haven't really checked, but in theory it should be possible to construct a new QuantityType from a given string like "1.56 km"

I can confirm this is possible.

If the toUnit method of QuantityType accepts a simple string like km/min, it's okay.

This I too can confirm that you can pass a string representation of the unit.

I am knee deep in QuantityTypes (I have to be, they cause so many problem for people on the forum).

And @jpg0 is right, there is a lot about them that can and should be fixed in core. But the main issue with using QuantityTypes in a rule (no matter the language) is you can only do math or boolean logic on QuantityTypes with other QuantityTypes or you have to explicitly convert them to a plain number. And sometimes you really do want to use QuantityTypes because the whole point is you should not have to know or care what the units are you are doing math with, all the conversions and operations happen automatically. So you can compare a °C to °F and get the right answer or add 20 feet to an Item holding meters.

Rules DSL invented a new operator shortcut to quickly create a QuantityType constant: 20 | °C. For example:

if(20|°C < MyTempItem.state)

In JS Scripting today we either have to just completely throw out the units, losing the benefit they bring:

if(20 < Number.parseFloat(items.getItem('MyTempItem').state)) // assuming that it's not °F

convert it to our desired units and then throw out the units if we are not sure it's what we expect:

if(20 < Number.parseFloat(items.getItem('MyTempItem').rawState.toUnit('°C').toString()))

or create a Java QuantityType Object to compare with. But because JavaScript doesn't support operator overloading (as far as my research has been able to tell me) we have to use the methods on QuantityType.

var QuantityType = Java.type('org.openhab.core.library.types.QuantityType');
var currTemp = items.getItem('MyTempItem').rawState;
var thresh = QuantityType.vaueOf('20 °C')
if(thresh.compareTo(currTemp) < 0)

For reverence, here's the javadoc for QuantityType: https://www.openhab.org/javadoc/latest/org/openhab/core/library/types/quantitytype

Note, I've tried it before to call valueOf() on the rawState but because the method is static, it's not available on the Object, only the class, so I had to import it.

Notice, the only way to actually use the units the way they are intended is to use Java Objects.

I know that wrapping these with the js-quantities library won't solve all my QuantityType woes, but it will go a long ways towards helping I think.

florian-h05 commented 1 year ago

@jpg0 @digitaldan I was able to implement the functionality of js-quantities by wrapping the openHAB QuantityType, see #206.

openhab / openhab-js

Introduce `js-quantities` for units of measurement handling #200