metadata: no place to indicate vessel wide display units

[bkp7]

Within meta there is a untis property which explicitly defines the (SI) units which the associated value is specified in.

There is no-where to indicate the vessel level preferred units. Without such a value each consumer will have to be configured individually to show values in typically understood units, as unfortunately the SI units are not often aligned with common units.

I propose an additional property within meta of displayUnits. This being the vessel level units that the value should be displayed in. Clearly individual users with their own consumer can set alternative display units which will take precedence over the vessel wide default units. Consumers added to a vessel (ie not personal devices) would not have to all be set up individually.

[sbender9]

Do we really want to have to specify this per data element?

Wouldn't it be better to have a way to specify more generic units preferences. "Speed", "Depth", "Pressure", etc.

I guess we could do both, but I wouldn't want to have to manually specify the units for every available path on my boat.

[sbender9]

What about having a units type for individual paths. This would be an enum. Current selection from WilhelmSK:

"Long Distance" "Short Distance" "Wind Speed" "Speed" "Depth" "Fuel" "Temperature" "Engine Pressure" "Atmospheric Pressure" "Position" "Rate Of Turn" "Flow Rate" "Fuel Economy"

And then provide a place to specify display units for each of these (and any others that I'm missing)

[tkurki]

See #130. How about

• annotating the schema with semantic classification
• adding a home for preferences, either dispersed in the metadata objects or as a top level entity. Latter would allow distinct preference sets, like personal prefs

[tkurki]

Oops, I am also falling to the ’one home for everything’ trap.

We could have a separate api to manage preference sets and the values set there could be also available in metadata objects, next to units. A default set, that you could override and the one in metadata could use logged in user to serve the vessel/system default or your pref if one exists.

[bkp7]

I also would not want to have to set units for each data element, but I think it should be possible to do so.

I would envisage that the server designer come up with some plan to arrive at the displayUnits for each branch. This could be hard coded, based on country, based on longDistance/shortDistance like Wilhelm, or something else, or even not done at all (ie. displayUnit is optional).

At the data interchange level we just need to describe what the preferred unit is without worrying about how it was decided upon.

Also, there would need to be a Signal K list of possible units so that consumers know which conversions they should implement.

There should be no limit on what additional units a consumer might wish to implement in addition to the list provided by Signal K.

[tkurki]

Also, there would need to be a Signal K list of possible units so that consumers know which conversions they should implement

Even more important is that we need to document how the different units are encoded in the data.

[bkp7]

I started working on this but on reflection I think including a list of possible units in the Schema may not be the best approach.

The meta already includes information about the schema (units, description), I believe so that the consumer doesn't have to have a copy of the schema. Following the same logic it would seem sensible that the consumer also be relieved of knowing all possible units and conversions.

So as an alternative how about including the conversion data in the meta? That way we don't need to specify any additional units within the standard. It would be up to the server to implement any conversions if so designed (this should all be optional data).

So adding a displayUnits object along the lines of:

"meta": {
  "description": "Tachometer, Engine 1",
  "units": "Hz",
  "displayUnits": {
    "display": "RPM",
    "description": "Revolutions per Minute",
    "factor": 60.0
  }
}

"meta": {
  "description": "Engine temperature",
  "units": "K",
  "displayName": "Coolant Temperature, Engine 1",
  "shortName": "Temp",
  "longName": "Coolant Temperature",
  "displayUnits": {
    "display": "°F",
    "description": "Degrees Fahrenheit",
    "factor": 1.8,
    "offset": -459.67
  }
}

Where: displayValue = SIValue * factor + offset

I have been through all the units currently in the Signal K specification and I think all conversions can be accommodated using just factor and offset.

However there are some units we might add in future where this would not be sufficient, and if we add such units we would also need to add an additional factor. I am thinking here of logarithmic conversions such as signal strength for example. In which case, if in future we decided to include such a measure it could look like:

"meta": {
  "description": "Receive power",
  "units": "W",
  "displayName": "Receive Power %",
  "shortName": "RSSI",
  "longName": "Receive Signal Strength Indicator (-96dBm to -10dBm)",
  "displayUnits": {
    "display": "%",
    "logFactor": 1000,
    "factor": 11.6279,
    "offset": 111.6279
  }
}

Where: `displayValue = log(SIValue logFactor) factor + offset

[bkp7]

Could someone add this to the metadata project please.

[fabdrol]

@sbender9 @tkurki: @bkp7 did a PR on this (https://github.com/SignalK/specification/pull/468), which looks good from a convention point of view but I don't want to hasten the pull, because there are no more replies here after @bkp7 final two comments. Could (one of) you have a look if it needs more discussion, or if you guys feel that this is the right way to go.

Personally, I agree with @bkp7's argument for putting it in a data point meta, given the way the model is structured right now. To prevent the problem of having to set it everywhere, an implementation could provide an easy way to set it once for a given SI unit, then apply it to every similar data point (which would cause a server to apply the relevant displayUnits block to each data point of that unit). Since the displayUnits block is still stored in that data points meta, it'd be trivial to implement a mechanism to overwrite a single data points' displayUnits.

Edit: after thinking about this some more, I stand by my point above if this feature is added to meta. However, I'm not sure it should be. Please see my comment on #409 about domain overlap between server and consumer, below are the most important points:

There's another issue to consider here: adding this to the specification just adds to the amount of bloat, and it has the potential of increasing the work a user has to put into server setup. It also raises a question in the development of consumers: if server defines a gaugeType, should we follow it or allow it to be overwritten? In the latter case, what's the point of the gaugeType in the first place (assuming one accepts my argumentation above).

Personally, I feel that we should aim to keep the domains server and consumer pure (at least from a specification point of view) and try to prevent overlap as much as possible. If there is valid use case (I do see some merit in @bkp7's scenarios) to "sync" consumers of a similar type (or consumers on the boat in general), I feel that this should be handled by an implementation-level API, separate from specification.

[fabdrol]

@bkp7 wrote a very complete response to my argument here: https://github.com/SignalK/specification/issues/409#issuecomment-396511153. I'm - personally - not entirely convinced that it's the best way to go, but the argument that 'centralised defaults' are a powerful feature does make sense to me - after all a consumer could simply choose not to follow the defaults.

Most of my worries, apart from the "purity"/separation of concerns, have to do with the mechanisms available to the user to set this stuff up. That is not necessarily in the scope of specification, but to take a leaf from @tkurki's book: does it make sense to accompany this PR with a plugin of some kind to make this actually available to the end-user?

[bkp7]

Following the discussion and rejection of PR #468, now taking a centralised approach to conversions in the schema, how about:

"resources": {
  "displayUnitConversions": {
    "m": {
      "default": {
        "display": "km",
        "factor": 0.001,
        "timestamp": "2015-03-06T16:57:53.643Z",
        "source": "??? - as used for routes?"
      },
      "urn:mrn:signalk:uuid:c0d79335-4e25-4248-8892-54e8ccc8020a": {
        "name": "Long Distance",
        "description": "Nautical Miles",
        "display": "Nm",
        "factor": 0.000539956803455,
        "timestamp": "2015-03-06T16:57:53.643Z",
        "source": "??? - as used for routes?"
      },
      "urn:mrn:signalk:uuid:c0d79335-4e25-4248-8892-54e8ccc8020b": {
        "name": "Depth",
        "description": "Feet",
        "display": "ft",
        "factor": 3.280839895,
        "timestamp": "2015-03-06T16:57:53.643Z",
        "source": "??? - as used for routes?"
      }
    }
  }
}

In meta:

"meta": {
  "units": "m",
  "displayUnits": "urn:mrn:signalk:uuid:c0d79335-4e25-4248-8892-54e8ccc8020b"
}

The rationale for having the SI unit above the UUID is so that the consumer can get a complete list of conversions relevant to that SI unit via the REST API, without having to get a complete list of all conversions for all SI units. It also allows a default for each SI unit.

If displayUnits is not present in meta it is assumed to be "default". If "default" is then not present in displayUnitConversions use SI (unconverted). This allows global preferred units to be set eg. "I want temperatures shown in Fahrenheit".

Implementation of quantities eg. depth/short distance/long distance would be a server implementation issue. ie. the server could set displayUnits for all "m" branches with depth/short distance/long distance, as required. When we allow the consumer to update the server it can change the conversion definition.

When updates to the server are implemented we would need to consider whether propulsion.*.exhaustTemperature.meta.displayUnits = foo is allowed and whether it applies to all existing engines only, or all engines including those as yet unseen.

[sbender9]

Some conversions are more complicated than just a factor. Do we actually need to support some kind of expression language (like JSONata)?

[sbender9]

@bkp7 I missed that you proposed logFactor, offset, etc.

That takes care of some conversions, but I don't think it makes any sense to specify conversions this way.

We need to support an expression language like JSONata or leave conversion definition out completely.

Let's just specify the preferred units and leave the conversions up the clients.

[rob42]

Since the conversions are fixed, eg the factor etc doesnt change over time I dont see the need for a source or timestamp. Likewise i would even consider creating a unique id by concatenating the to and from units.

My logic is that 'm2ft' is only needed once, and will always have the same factor etc. Or maybe 'm.ft'.

I also think we might be able to just store the maths expression. The usual ones are pretty easily parsed. Maybe use some std format, even spreadsheet formula '=LOG(x,2)'

[rob42]

This is a java lib that reads and executes arbitrary maths expressions written in human form https://lallafa.objecthunter.net/exp4j/. Must be similar libs in most major langauges

[fabdrol]

@bkp7 maybe the key of the units should be written out, to avoid confusion. E.g. nm may refer to distance (nautical miles) or force (newton meter). In regular meta, at the moment, the use of the shorthand names isn't an issue because every single key must be an SI unit (which are all unique).

[bkp7]

@fabdrol the display would hold the string representing the unit which is postpended to the converted unit. description would be longer text which not only describes the conversion but can also deal with any ambiguity eg. statute, international, survey mile, etc. If we define a key for the unit we then need to make sure each is unambiguous and unique. Are you thinking that a key be used rather than a GUID, in which case doesn't that bring us back to having to define a list in the specification?

@rob42, the problem with assuming that m.ft always has the same value is ensuring that the name m.ft is always unique. ie. we would need to define what ft meant (international, survey, etc), and that brings us back to having to define all possible units in the specification. My proposal brings the functionality of allowing a conversion to be defined as say 'shortDistance' which can then be referenced. The rationale for having source and timestamp is mainly to follow the example set by charts, notes, routes, etc. in resources for consistency. Along with other resources we will probably need a delta specification in due course and we need to clarify what happens when there are mutiple servers (sources). I'm not sure why timestamp should be there for the other resources? Maybe it's helpful as a quick check via REST API to know if something has changed without having to read every property?

@sbender9

Let's just specify the preferred units and leave the conversions up the clients.

Because of the ambiguity in identifying non SI units, I think if we did this we would need to list all possible conversions within the specification, and this is a list which will be very long and we would end up adding to over time. I'm not keen on this approach.

@sbender, if there is a suitable standard which allows us to state the conversion as an expression that would be great, provided it is cross platform, language agnostic and secure. I took a brief look at JSONata which is described as a query and transformation language, does it have a way to encapsulate an expression within the JSON itself?

@rob42 my concern with anything passed as data, but which has to be evaluated/run is security. Could this form the basis for an injection attack vector?

[fabdrol]

@bkp7 that's what I was thinking; at the end of the day even with a GUID the implementation would have to make sure that it's unique - using a name of a unit is slightly more semantic. Obviously, that does limit the number of possible unique display units - however I don't think that there will be that many. I reckon that in most cases, users will set up a bunch of standard conversions if they don't like the included defaults - but that won't be 1000s of them.

I reckon the main question is wether we need to design for a greater number of possibiilties or not. If you guys think, as I do, that using proper names for the units is unique enough I'd say we should choose that option over something less readable/more obscure.

For example: I don't believe that a vessel will include more than one conversion for each of the nm's: newtonmeter, nanometer, nauticalmeter (that would imply unnecessary duplication of the same conversion).

[bkp7]

@fabdrol fair points. So we would have some (common) conversions defined in the specification and allow additional ones too. Previously I think the discussion has been looking at either, or.

If we insist on all consumers natively implementing the specified list of conversions (and we describe each fully in the docs) this will ensure consistency of presentation across all consumers. We would still have to deal with the presentation side of the conversion. Using Gallons as an example, which has ambiguity and a material difference, an American would not wish to see 'International Miles per US Gal', any more than a Brit wants to see 'International Miles per Imperial Gal'. We could take the view that both units be always presented as 'MPG' even though they are actually different, but I would be reluctant to make that part of the specification. This is an example of two different units sharing the same presentation, but there is also the case of the same unit being presented differently eg. kn, nd, kt or kts for knots.

Alternatively, we could make consumers dumb (as regards conversions) and insist on servers implementing the specified list. Where a server can 'multiplex' info from other servers it is required to implement the whole list, but where it is stand alone it only needs to implement those relevant to its design. ie. a Signalk fridge only implements temperature conversions.

However, neither approach solves the issue of wanting to specify groups of units eg. shortDistance/longDistance/depth, which my proposal does, at least for already known branches, and maybe across the board depending on how we define updates made to a server. We could implement this with another section so that for example shortDistance maps to feet and so on.

[tkurki]

Just a short note: nm ambiguity is much less of an issue if you consider it in combination with the base SI unit: meters don’t convert to Newtonmeters (whose abbreviation is btw Nm).

But the issue with for example different gallons still stands.

[rob42]

I wonder if signalk is the proper domain for the conversions at all. Early on we avoided the mess that other protocols have with units by using defined SI units per key everywhere, which I think has been a win.

The conversion factor itself is constant math(eg the same for every-one), even for obscure ones. Maybe we should just provide a way to define target units per locale/vessel, and assume the client or implementation provides the conversions. A custom library for various languages would make that easy to use.

That brings me back to a top-level locale tree, using the hierarchical EN_NZ_motu.navigation.speedOverGround.units concept.

Copied here from #468 co I cant work out how to link a specific comment :-)

I think we need a signalk structure for international 'translations', based on a hierarchical tree similar to the java ResourceBundles concept. Basically we create a tree of properties with increasing specialization. The top level is Locale, eg locale.EN (english), then locale.EN.NZ (new zealand),

The tree is recursively read from bottom up, so if I dont specify locale.EN.NZ.navigation.courseOverGroundTrue, I get locale.EN.navigation.courseOverGroundTrue

I can then specialize that to locale.EN.NZ.nelson.motu for my region, vessel, or other arbitrary criteria. To get the applicable values for my user they select the locale from the locales available on the server, and do the conversions on the users device.

This has the extensibility of adding new nodes to the tree FR (french), and being able to transmit locales around the network as signalk messages. It allows for all or only relevant locales per installation.

[fabdrol]

A side note: this discussion is a good example of why before I've advocated keeping this sordid stuff in the consumers ;-)

Anyway, more to the point @bkp7 I feel that we're over-complicating it. If we simply define a resource for units and provide a plugin so people can actually set it up, then consumers could choose to use these - or not. The SI base units remain the actual unit of the value so any consumer that works today will keep working.

I reckon the model could be evens simpler - as somebody pointed out: the conversions are not likely to ever change, so what's the point of a timestamp? Regarding source: I think this may make sense if the source is a slave SK server (think about the ECU example) - but since all of that needs more work I would say we defer that for now (develop use cases in a separate issue, etc).

I believe that a simple schema, with references to it in data nodes' meta, will suffice (for now, in any case). Simple is usually better. What I'm unsure of, however, is how we handle the calculation in JSON. JSONata seems like an option, but requires a custom parser which may not be available everywhere and/or add (unacceptable?) overhead. Consider that it should be acceptable for basic embedded hardware as well (down the line, IMHO).

const resources = {
  displayUnitConversions: {
    meter: {
      default: {
        shortName: "km",
        longName: "kilometer"
        calculation: ????  // some language-independent way of describing the conversion formula
      },
      'nautical-mile': {
        shortName: "nm",
        longName: "nautical mile"
        calculation: ????  // some language-independent way of describing the conversion formula
      },
      // ... etc
    }
  }
}

As for translations/supporting different locales; I feel that is a discussion that should be deferred to a next major version (reasoning: the current nested structure is "ugly" once you mix in the locales. Whilst I don't dislike @rob42's proposal per se, it seemingly duplicates the tree. Integrating locales into a graph model where nodes are a collection of edges to other nodes (as suggested in various discussions on Slack and here by @tkurki) combined with a HTTP header or query parameter makes much more sense to me, since one could simply "link" to another meta node when the language changes. But that's besides the point)

[bkp7]

@fabdrol could you just clarify whether in your example you envisage 'nautical-mile' etc. to be defined/listed in the spec? If so I would agree that it shouldn't ever change. If not, do you envisage it must be hard coded into the particular server, in which case it also never changes, or is it something that a user can set up, in which case it would change when the user changes it?

There is also the issue of the units themselves which I think you would want to change. So in the example of knots (k, kt, kts, nd, etc) whilst the calculation doesn't change, the units can.

BTW, I agree about keeping this stuff in the consumer. All I'm after is a way to set vessel wide display units for values when the consumer hasn't been set up to do something else. ie the consumer can choose to use the vessel units or not.

[fabdrol]

@bkp7 I don't think the data should be in the spec; the spec describes how this stuff is presented by an implementation; the implementation can take care of actually implementing it. I'd say that node server should provide the main units + a plugin to add other conversions, as a guide to server implementors

[bkp7]

I don't think the data should be in the spec

@fabdrol, yes I agree.

But if users/installers can add/modify other conversions we need to keep the conversion info up to date (timestamp). Also a multiplexing server will have to deconflict the names used, which could be done by concatenating source or something to the name, but actually wouldn't it be better to use a UUID as with waypoints, routes, etc.? That's how I ended up with the proposal above which is also consistent with the other resources.

[rob42]

The dependency on using http/ws specific solutions will only satisfy web browsers, and force that on any UI device. What about a simple LCD readout?. What about an existing chartplotter? We need to make sure there is a way to propagate these things over raw signalk, or it will bite us later.

[tkurki]

Resources and metadata are currently only accessible via http. Specifying how to access all the data from non http/ws devices is a separate issue.

[fabdrol]

@rob42 good points, can you create an issue for that? I reckon that point is valid for all resources.

@bkp7 I'm sensitive for the consistency argument, the other points I feel are not that of a problem. If a server encounters two definitions of a meter -> nautical mile conversion, they'll most likely be the same (and if not, one is wrong). Regarding the timestamp: even if the conversions change, they won't change very often and I fail to see what the consumer wants to do with that information.

[bkp7]

@rob42 agreed, I've created new issue #479

@tjkurki meta and resource are available via the websockets get, but can't be subscribed to.

@fabdrol there are so many non SI units many of which share the same local name. eg. litre to gallon. You cannot assume 'they'll most likely be the same'. Even if the conversion is fixed the display unit could be changed, so if the user changes kt to nd they would expect consumers to pick that up and change too. I'm not certain what the timestamp is for in the other resources (undocumented) but I'm assuming that the consumer can read just the timestamp, if unchanged do nothing, if changed ask for the resource again.