Schema should include metadata about sources

[fabdrol]

The current spec doesn't include a good place for metadata about data sources, such as N2K devices on the network, 0183-enabled devices, other SK devices, DIY devices (e.g. arduinos) etc. At the moment, there is a "sensor" group which doesn't fit this need at all.

This new place (e.g. a group called sources) should hold device metadata from various N2K PGNs, 0183 sentences (a list will follow) and retrieved from a SK-enabled device via the REST API or web socket. At least the type of device, identity information, capabilities and how to connect to/reach the device should be included.

Possible other properties are the various other fields defined in the PGNs/sentences, priority/weight, user defined information (friendly label, for instance), placement on the vessel (in case of GPS devices, depth transducers), etc.

[rob42]

Ive used sources as " the source of adata value" so that's different to a physical source or a sensor It includes virtual, remote, physical or calculated " sources"

I see " inventory" as a Collection of stuff on the boat. And "sensors" are different again. Then a boat has a "manifest" as well,

So lots of stuff that we need to get clear in terms of what we are trying to acheive

Needs some new ideas or better problem definition maybe

[fabdrol]

I'm not sure we should have so many different categories; a "value source" could be a source in a future sources list, but doesn't have to be IMHO. Having a manifest, inventory, sources and sensors just makes it more complicated to implement; I'd vote for a single "sources" list that is flexible enough to include data sources of various types. Wether the data source is a sensor, a SK gateway, a N2K device or a 0183 device is immaterial.

[sumps]

Very few people (and certainly no dealers/installers) are going to manually create a list of sources, sensors, etc, so the schema we define should give priority to the fields that can be automatically populated from the data being received.

With serial or networked data the only source info you can get automatically is the Connection info i.e. COM1 or 169.254.1.1:2000 the protocol i.e.TCP, UDP, SeaTalk, 0183(HS), etc. the type of equipment sending the data (from the Talker ID) i.e.SD, GP, AI, etc. and the types of data available i.e. Wind, Boatspeed, GPS, etc. and this must be gradually discovered as the data is read, there is no "Discovery" mechanism.

With NMEA2000 a Gateway could "Discover" within a few seconds a "map" of the network and a list of not just the data sources but the physical equipment on the bus, with Manufacturer code, Product code and even firmware version. For the data sources, it is probably still going to be a case of gradually discovering the data that each device transmits, because although there is mechanism to query a device on the NMEA2000 network to discover what PGNs it transmits (or receives), I am not sure if every manufacturer has to support this feature.

[fabdrol]

@sumps I think that's more a server issue; but I do agree. Maybe with the exception of diy and 0183 sources, the discovery shouldn't be such a big problem.. SK devices have mDNS, n2k has it's own process if I understand correctly.

[sumps]

Multiple GWs/Servers need to be considered so sources also have to be linked to a GW or Server

[fabdrol]

GWs/servers (at least, SK ones) have mdns, so they should be easy to discover. It ties in to an earlier conversation we had regarding the "model number" or ID...

[sumps]

Do you want the GW, Server or Consumer to decide the primary source of say GPS position ?

[fabdrol]

Good question. I'd say that if there is ambiguity a server should be authority, perhaps via a config page or something. But, that's maybe too much implementation, first we need to figure out what is needed to be able to store this stuff in the first place (spec wise)

[sumps]

In an ideal world, it should be possible for every single piece of data to come through to the consumer, if the user wants it to. In this case there might be many different POSITION Objects being received and each must have a traceable Source.

Currently for every type of data object i.e. Position, Depth, etc. we have a Source and this Source currently has four fields; Label, Type, Src, PGN

I assume the intention is for the Label field to be manually entered, so most likely will not be filled in, so that leaves the Type which should be NMEA2000, NMEA0183, SeaTalk, etc. the Src which should be the Signal K device (UUID or ?), possibly a new field "Interface" which will be COM1, 169.254.1.1:2000, CAN1, ttyusb0 and a new field "Device" which could be the NMEA2000 bus address of the device that sent the data or the TalkerID of an NMEA0183 device and finally a new field called "message" (that replaces PGN) which is the NMEA2000 PGN number or the NMEA0183 Sentence ID of the original massage.

Would this work ?

[sumps]

At the end of the day we need enough info in the Source fields for a GW/Server/Consumer to automatically decide which is the best source to use or enough info to allow the user to identify the source and manually select which source to use. Is there anything else we need from the Source data ?

[rob42]

@fabdrol - I agree about too many categories, the point was they are not clearly defined, and have too much overlap. That makes it confusing as to what is in there and why.

I think @sumps is close, the key thing is what we need now - not what we might need, or might like. So to me the useful data is where a reading came from, and maybe a list of choices for a given value like depth.

To explain:

I sail a catamaran, and its 6M between the bows. I have had 2M of water under one hull, and the other aground!. So two depth gauges are handy, and if i get a depth alarm, I need to know which one!. In NMEA0183 thats really hard, better in N2K but i dont have that. Also if I map data for crowd source depth, then 2 feeds 3M off center matters too.

Also I may receive wind from the masthead, and from the weatherbouy here, and from metservice wind modelling feed. They are all just wind speed and dir in signalk, but their source matters. In order of reliability, its masthead, local weatherbouy, metservice. If I am in the marina, about to leave, and see 25knts on metservice, and 15knts on the local weatherbouy and masthead that will influence my decision.

So for version 1 I think we should record what we can gather in the course of normal data reception in 'sources', using a suitable logical tree.

We can add to this when we have a better use case for the infomation.

[sumps]

We need to make decision on this source issue. For the Gateway, I propose one config setting that is... Sources

• Auto (Gateway auto selects best source and only outputs one source for each data type)
• All (A delta file is created for each piece of data, from every source)

We need to check and standardise the location of the Source field in each data type, it is currently different for say Position compared to Wind Speed.

I propose that we have one Sensor field for each data type and this basically says that this Depth Value came from "Sensor XYZ" and then in the Sensors section we have a list of each of the sensors and for each sensor full details (metadata) of what it is, what protocol it is using, sentence/PGN, gateway or interface reference. etc.

In this way a consumer or server can use the restAPI as follows...

/signalk/api/v1/vessels/self/sensors

This returns a list of the sensors (sources) and then the server/consumer can easily present this list to the user and get them to prioritise the sources, fill in any blank information (name, location, etc.) or the server/consumer can automatically apply its own priority or filtering of sources.

[sumps]

If the consumer or the server for that matter (connected to a gateway like iKommunicate) is to make a reliable choice of the best source from multiple data sources, then you will need to improve the metadata related to Sources. We currently have just four fields and in my opinion this is not enough.

I have studied this quite a lot over the last week or so and I think we should replace the "Sensor" object with a new "Source" object that would cover all sources of electronic data that comes in to a Signal K system. Each source will have a Unique Local Name i.e. "Depth Sounder 1" and all applicable data objects that need to be traced to a source will have a source field that stores the Name of its source e.g. vessels//environment/depthBelowTransducer would have a field "Source" with value of the source's Unique Local Name.

An NMEA to Signal K gateway would be able to automatically create this Sources list from the NMEA data it is receiving and an application would be able to query and display all sources using the restAPI "vessels//sources". These could then be presented to the user for manual priority selection or sorted by Interface type, N2K source address, instance or some other method.

I would strongly recommend that we set a field called "Priority" in each traceable data object so that even the simplest of consumer can always be displaying the automatically or manually set highest priority data source.

I think the following fields would be required for a new "Source" object and nearly all of these can be automatically populated by a suitable NMEA to SK gateway....

• uniqueLocalName
• productModel (i.e. N2NET NMEA 2000 AIS)
• productFunction (i.e. AIS)
• type (i.e. Combined GPS/GLONASS)
• protocol (i.e. NMEA2000)
• signalKSource (i.e. iKommunicate 1)
• priority (i.e. 0 highest, 99 lowest)
• nmea2000SourceAddress (i.e. 23)
• sentenceID (i.e. VDO)
• distanceBow (i.e. distance in Meters from Ship's Bow)
• distanceStarboard (i.e. distance in Meters from Ship's Starboard side)
• installationNotes (i.e. "Mounted behind chart table with Splitter and GPS antenna on Mizen")
• manufacturersInfo (i.e. "support@digitalyacht.co.uk www.digitalyachtamerica.com")

[timmathews]

May I suggest adding:

• firmwareVersion
• modelVersion
• serialNumber
• pgnList (an object containing all of the transmitted PGNs and their default and current transmit rate, perhaps also the PGNs which the device is capable of receiving)
• replace sentenceID with sentenceList (same as pgnList, but for 0183 sentences)

[sumps]

Yes, I had meant to add Software/Firmware version and Hardware Version which are both easy to extract from NMEA2000 systems. Serial Number can also be added.

I am not convinced that a PGN List is useful enough to justify the additional overhead that it would place on a gateway/server.

I think I put SentenceID in by mistake and should be removed. A sentence list would also be additional overhead for the gateway and is even less useful that a PGN list.

[fabdrol]

I'll post a PR today with a first pass at this

[sumps]

nmea2000SourceAddress should be called canbusSourceAddress, as I am sure Signal K will be connected to other Can Bus based networks i.e. J1939, CANOpen, etc.

I am not sure if we have any other SIgnal K objects or fields that have NMEA2000 in the name, but these should also be changed to canbus.

[timmathews]

The sentence and PGN lists are important. We need to maintain a list of available data points in the system somehow. I think grouping it by device is as good as any other option.

[fabdrol]

@timmathews if the point is to maintain a list of data points, I'd argue for a list of available SK paths. N2K and 0183 are merely sources, it could be anything (sensors on the computer itself, arduinos, non-NMEA weather stations, IoT hardware etc)

[sumps]

Bear in mind that this is just a snap shot of the data at the moment and in the case of PGNs just the TX and RX PGNs that the device supports, it may not have the transducers to actually send data out on these PGNs.

It really is better to use the restAPI to see what the latest data is, rather than worry about data points that may or may not be correctly listed.

[timmathews]

But then we can only show things which are already mapped. I believe there is utility in keeping a list of available data which isn't currently mapped to SK.

As a consumer developer, I want to be able to query a server to get an object containing all of the available data points and some metadata about them like data rate, source device, device type, maybe other stuff. How that happens isn't particularly important, but I think it needs to be simpler than getting the whole current tree (which may not include every available data point anyway) and walking it to determine what's available.

I know this is more or less how Instrumentpanel works right now, but maintaining this state centrally seems like a better option.

[fabdrol]

You're talking about two things right? A list of available data (= already mapped) + metadata, is something I believe should be available on request both via WS and REST. Something like this:

{
  "/vessels/self/navigation/position/latitude": { ..meta },
  "/vessels/self/navigation/position/longitude": { ..meta },
  "/vessels/self/navigation/position/altitude": { ..meta },
  etc
}

could even be scoped (like /signalk/v1/available?scope=vessels/self)

Then the second thing, data points that are not mapped - i.e. not available on that SK server. I'm wondering why this is important to know? If you're connecting to a SK server, and that data isn't mapped, you can't do anything with it anyway. So why would a consumer need to know?

[timmathews]

So that as an end user I can write the mapping and add it to the server. Especially useful for proprietary sentences/pgns.

[sumps]

How is a gateway to handle a proprietary PGN ? I am not sure proprietary PGNs and an open source data format are comfortable bed fellows 😉

[timmathews]

Well, at first it doesn't handle them. It just shows that there is a device with proprietary (or otherwise unknown) data, and provides a way to get the raw bytestream.

Then someone might could write an app which takes that stream of data and lets the reverse engineer play with field widths and types until data begins to make sense. Then the app could generate PGN definition and SK map files which could be uploaded to the gateway. et voila, what once was hidden is now plain to see.

Anyway, I'm firmly against hiding data from users. So the spec should support it. If gateway manufacturers choose not to include that functionality, well that's up to them.

[tkurki]

See sources.json.