jboynes commented 7 years ago

Summary

SignalK provides an open model that allows for both commercial and amateur implementations. This RFC defines how each implementation can describe its data model, without needing a central repository for such definitions. It enables simple, config-free devices to be integrated into a holistic vessel-wide configuration in an automated manner.

Motivation

To allow interoperation between independently developed systems, a typical approach is to define a common standard to which implementations must conform. Such a standard may be controlled by a single vendor, by committee, or by a formal standards body. Regardless, any new functionality must be approved by the controlling entity and integrated into existing implementations before it can be used. This limits the pace of change and discourages innovation.

This RFC describes an open, federated data model that can be used by devices to describe the data they can provide and the commands that they can support. By defining a standard for a meta model it provides a mechanism that allows any other device to integrate with the new device's data using standardized configuration entries.

Although the model is fully open, we expect common patterns of use to emerge, and that implementers will prefer to use those patterns over proprietary definitions when possible. These common patterns will initially be defined by community working groups before maturing into group patterns defined by SIgnalK.

This issue was initially identified as a TODO item in the sensors group. This RFC refers to devices in preference to sensors to allow for other types of unit such as actuators or more complex systems.

Detailed Design

Put simply, this RFC defines patterns for describing the representation of JSON data using JSON Schema. A system need only support these patterns in order to integrate with any other system that uses them to describe its data model. Integration is performed by introspecting a JSON schema provided by a device that describes its data model, combined with schema annotations defined by this RFC that supplement the raw schema; an existing example of such an annotation is the units property used to specify the unit of measure for a JSON number.

Device Data Model

In an open model, each device may potentially support an arbitrary set of data and messages used to exchange information with other systems. To support this, each device is required to provide a JSON schema that describes the data and messages; this schema may be defined by the device's creator, or may a reference to a common schema defined by other implementers or this project for use by similar devices.

The schema is responsible for defining:

the atomic data values the device supports
the format of messages used to interact with the device

Atoms

An atom is a non-decomposable fragment of data used for interaction with a device. This may be a single value (such as a voltage), or may be a structure containing multiple, closely coupled properties (such as a position comprised of latitude, longitude and an optional altitude). A atom is the smallest unit of information that may be exchanged between systems, although higher level messages may contain multiple atoms.

In addition to the actual value, an atom may contain meta information describing the format, source, timestamp, validity, or accuracy of the value. An atom is therefore defined to be a JSON object type whose schema is decorated with the atom boolean property. A parent or child definition of a definition annotated with atom MUST NOT be annotated with atom.

Atom Metadata

All property names beginning with _ are reserved for atom metadata. Implementers SHOULD begin device property names with characters from the set [a-zA-Z0-9].

Format

The content of an atom may be specified using a JSON schema reference i.e. a property named $schema containing valid JSON schema or a reference to a JSON schema definition.

Example:

{"$schema":"http://devices.acme.example/signalk/bme280.json"}

Atom ID

The _id property should contain a unique identifier for this atom. How this identifier is created is undefined; applications may use a sequence number, structured UUID, or random token. Systems receiving this id must treat as an opaque token.

Examples:

{"_id":"1234567"}
{"_id":"acme-bme280-f1df70-1234567"}
{"_id":"4556-gdgdfg-fge45-5646"}
{"_id":"dsgflwj56tdsufisdt50wtgd"}

Device ID

The "_deviceId" property may contain a unique identifier for the device that created this atom. Atoms should contain a device id when sent over a shared channel such as a UDP multicast group or message topic.

Example:

{"_deviceId":"acme-bme280-f1df70"}

Timestamp

The _timestamp property may contain an estimation of the time at the source when the atom was created, expressed as an absolute time or a duration since the Unix epoch (1970-01-01T00:00:00Z).

Examples:

{"_timestamp":"2016-11-26T21:23:28.454Z"}
{"_timestamp":"PT1480195408.454S"}
{"_timestamp":1480195408.454}

Non-normative Examples

The atom for a voltage value could be defined as:

{
  "type": "object",
  "atom": true,
  "allOf": [{
    "$ref": "https://signalk.github.io/specification/schemas/definitions.json#/definitions/atomMetadata"
   },{
    "properties": {
      "voltage": {
        "type": "number",
        "units" : "V"
      }
    },
    "required": ["voltage"]
  }]
  }
}

with a sample data point of

{"voltage":12.25}

A position as

{
  "type": "object",
  "atom": true,
  "allOf": [{
    "$ref": "https://signalk.github.io/specification/schemas/definitions.json#/definitions/atomMetadata"
   },{
    "properties": {
      "latitude": {
        "type": "number",
        "units" : "deg"
      },
      "longitude": {
        "type": "number",
        "units" : "deg"
      },
      "altitude": {
        "type": "number",
        "units" : "m"
      }
    },
    "required": ["latitude", "longitude"]
  }]
  }
}

with a sample data point of

{
  "_deviceId":"acme-GPS01-5d34e7",
  "_id":"1480195408454",
  "_timestamp":"2016-11-26T21:23:28.454Z",
  "latitude":40.345,
  "longitude":-122.765
}

where the "Acme GPS01" device with serial number "5d34e7" uses time as a unique id and is able to provide an absolute timestamp.

Device Messages

Systems interact by exchanging messages containing zero or more atomic values. Each message is structured as a JSON object, described by a JSON schema supplemented with SignalK specific annotations.

Each message should include a $schema property at the root corresponding to the content of the message. Receiving devices may validate the message against that schema and may discard non-conforming messages without further processing.

The message types a device can send and receive are listed in its device metadata.

Device Specific Model Space

Every SignalK device is allocated a portion of the model tree unique to it under the context path /devices/{deviceId} where deviceId is the globally unique id for that device. From the example above, this location would be under the path /devices/acme-bme280-f1df70/...

This property must contain an object conforming to the SignalK type "#/definitions/device" including a "_type" property that contains the device metadata, either as a JSON object or as a reference to another JSON object.

For example, the device model

{
  "devices": {
    "acme-bme280-f1df70": {
      "_type": "http://devices.acme.example/signalk/bme280.json#/metadata",
      "serialNumber": "f1df70",
      "lastReading": {
        "_timestamp": "2016-11-26T21:23:28.454Z",
        "temperature": 293.15,
        "pressure": 99845,
        "humidity": 45.65
      }
    }
  }
}

gives the unique id, serial number and last reading from this device. The linked metadata http://devices.acme.example/signalk/bme280.json contains:

{
  "metadata": {
    "manufacturerName": "Acme Devices, inc",
    "productName" : "BME-280 Environmment Sensor",
    "model": {
      "$schema": "http://json-schema.org/draft-04/schema#",
      "id": "http://devices.acme.example/signalk/bme280.json#/metadata/model",
      "allOf": [{
        "$ref": "https://signalk.github.io/specification/schemas/definitions.json#/definitions/commonDeviceProperties" 
      }, {
        "properties": {
          "lastReading" : { "$ref": "#/data" } 
        }
      }]
    },
    "sends": {
      "hello": { "$ref": "https://signalk.github.io/specification/schemas/messages.json#/hello"},
      "data": { 
        "type": { "$ref": "#/data"},
        "interval": 10
      }
    },
    "receives": {
    }
  },
  "data": {
    "$schema": "http://json-schema.org/draft-04/schema#",
    "id": "http://devices.acme.example/signalk/bme280.json#/data",
    "type": "object",
    "atom": true,
    "allOf": [{
      "$ref": "https://signalk.github.io/specification/schemas/definitions.json#/definitions/atomMetadata"
     },{
      "properties": {
        "temperature": {
          "type": "number",
          "units" : "K"
        },
        "pressure": {
          "type": "number",
          "units" : "Pa"
        },
        "humidity": {
          "type": "number",
          "units" : "ratio"
        }
      }
    }]
    }
  }
}

that shows the device sends a standard hello message, as well as its device specific data message every 10s.

Device to Vessel Integration

This RFC describes application-level integration that occurs after a device has been installed, authorized to join a vessel's network, discovered, and been discovered, by other SignalK applications. It assumes that the messages defined by this RFC can be exchanged.

Examples of such integration include:

a voltmeter may need to be associated with the circuit location that it is measuring (e.g. house vs. engine battery)
an environment sensor may need to be associated with a living space (e.g. aft cabin vs. saloon, lower vs. upper deck, inside vs. outside)
a relay may need to be associated with the circuit it operates (e.g. main cabin lights vs. engine start)
a compass sensor may need to be associated with a deviation calibration set

Each device announces its capabilities by sending a standard hello message containing a minimal copy of its device model including the device metadata and, optionally, the current device state. The minimal model is defined by the https://signalk.github.io/specification/schemas/messages.json#/hello schema:

{
  "type": "object",
  "properties": {
    "devices": {
      "type": "object",
      "patternProperties": {
        "[a-ZA-Z][a-ZA-Z0-9._-]*": {
          "type": "object",
          "properties": {
            "serialNumber": {
              "type": "string"
            },
            "_type": {
              "type": "object string",
              "description": "Device type definition, either as an inline object or the URI of resource containing the definition" 
            }
          },
          "required": ["serialNumber", "_type"]
        }
      }
    }
  },
  "required": ["devices"]
}

The device responsible for integrating the new device into the vessel's data model creates an entry in its own model containing the new device's metadata, and performs any mapping needed to include the devices data into the vessel's model. For example, the an installer might enter location information indicating the environment sensor was located in the master's cabin thereby associating its readings with the /vessels/2309999/environment/inside/masterCabin path (or wherever the installer deemed most appropriate).

Schema Distribution

The JSON schema documents associated with a device may be distributed in any way that allows them to be read by the integrating system.

A simple, albeit verbose, mechanism is to include them, by value, in messages sent by the device. This may be done during initial negotiation, for example, as part of a hello message.

More typically, the schema would be referenced using its unique id (from the JSON schema definition). If this id is a dereferencable URL, any integrating device with internet access would be able to retrieve it directly; such devices should cache the copy locally for offline access. Other alternatives include:

a device with an embedded web server could support download from the device itself
a copy of the schema could be provided on installation media shipped with the device (e.g. a flash drive)
common schema may be pre-installed on the integrating device; for example, a manufacturer may pre-install schemas defined by this specification, or for other devices they produce

Drawbacks

Although the device specific information is kept to a minimum, more work must be performed by the integrating device to retrieve and process the metadata objects and schemas. For example, the environmental sensor need only deal with simple messages containing its serial number (unique id) and atomic values. The rest of the information is static content that could kept on the device or sourced elsewhere. However, the integrating server needs to be able to retrieve or otherwise install that information.

A device manufacturer is responsible for producing the device metadata needed for automatic integration. However, it seems better to place the burden there rather than on end users.

Alternatives

We could maintain a central registry of device types and metadata groups. However, that means any new device type implementations would need to be defined here first, entered in the registry, and would not be usable until a new version of the schema and keys was published.

We could require every device to allow configuration that defined how it was associated with the vessel-wide model. That would require some form of user interface for the installer to use (although that could be some form of remote client) and would require all clients to have local persistent storage (e.g. EEPROM or flash) to store the association mapping. For example, the environment sensor would need to be configured to send its readings as part of the /vessels/{vesselId}/{inside}/{cabin} context.

Unresolved Questions

The schema entries for this need still need to be fully defined, including more verbose descriptions.

jboynes commented 7 years ago

PR for comments if that works better

I also started on a proof of concept, using a sketch for a Wifi-capable Arduino to send environment data to a UDP multicast group. Listening on that is a simple server application that processes hello messages from the device to identify atoms, and then associates updates with other atoms in the vessel-wide portion of the model.

{"devices":{"acme-bme280-0ffbf1":{"_type":"http://devices.acme.example/signalk/bme280.json#/metadata","serialNumber":"0ffbf1","lastReading":{"temperature":295.60,"pressure":99752,"humidity":44.71}}}}
Processing update of [devices, acme-bme280-0ffbf1, lastReading]: {"temperature":295.60,"pressure":99752,"humidity":44.71}
Setting [vessels, 2309999, environment, inside, masterCabin] to {"temperature":295.60,"pressure":99752,"humidity":44.71}
Processing update of [vessels, 2309999, environment, inside, masterCabin]: {"temperature":295.60,"pressure":99752,"humidity":44.71}

{"devices":{"acme-bme280-0ffbf1":{"lastReading":{"temperature":295.60,"pressure":99752,"humidity":44.71}}}}
Processing update of [devices, acme-bme280-0ffbf1, lastReading]: {"temperature":295.60,"pressure":99752,"humidity":44.71}
Setting [vessels, 2309999, environment, inside, masterCabin] to {"temperature":295.60,"pressure":99752,"humidity":44.71}
Processing update of [vessels, 2309999, environment, inside, masterCabin]: {"temperature":295.60,"pressure":99752,"humidity":44.71}

pod909 commented 7 years ago

It looks like it would be sensible for the hello message to be the same as the record of the device under the "device" node.

There is a lot of cross over with the thinking around sources as well which should probably be resolved.

Should there be standard strings for the SK specified message objects? something like "$schema": "_delta" "$schema": "_value" "$schema": "_full"

rob42 commented 7 years ago

See http://json-ld.org/ We should try to incorporate existing standards as much as possible.

Also worth noting that adding to the schema should not be onerous for new use cases. Something like: 1) devise extension 2) publish on your own webserver, so others can access if required. 3) submit RFC or pull request 4) ...discussion...change..repeat 5) accept and merge in to dev 6) tag next sub version.

One of the key goals in signalk was to be extensible without breaking older clients - they simply ignore what they dont recognise, so adding a v1.2 device to a predominantly v1.1 system is fine.

SignalK / specification

RFC 0006: Device Data Model #297