Sending SK values in messages.

[jboynes]

The way values are represented in the full model and in delta messages is different. Based on discussions in #291, #288 and #289, the transport for these could be unified. We should also consider sensor and actuator devices that provide and act on changes in the model but do not have the resources to keep a full copy.

[tkurki]

Why would you not use the delta format for sending updates?

Full tree is different, because it allows hierarchical HTTP query access to the latest snapshot of data, so I don't really see it as a transport.

[jboynes]

Proposal: now we have the value type, replace the update data structure with a sparse model containing only leaf values of that type. Omitting for clarity the source tracking, the linked delta message would become:

{
  "vessels": {
    "230099999":{
      "navigation":{
        "logTrip": {"value": 43374},
        "log": {"value": 17404540},
        "courseOverGroundTrue": {"value": 172.9},
        "speedOverGround": {"value": 3.85}
      }
    }
  }
}

The recipient would update their copy of the model, preserving any other properties of the value object. This would include the source property if it was sent. To remove a value, set the key's value to null.

This has the advantage of using the same structure for representing a full, sparse, or change to the model. The structure can now be described by and validated using the schema, allowing a consumer to introspect the content.

[jboynes]

@tkurki I hope I covered the difference to the current delta format - the idea is to have a single representation.

I don't see how HTTP query access fits into this. This is covering the representation used to transfer state, not the underlying entity. A query would execute against that entity in the same way.

[pod909]

For value updates I don't think a nested structure like this works. All that matters is the path to the key/property/attibute and this will be carried in different places depending on transport. The nested structure is only relevant for responses over REST.

Multiple keys in the same message is also a bit of a pain.

Need to see how multiple values would be sent. For me this can not be keyed on source as multiple values may be sent from the same source. Nor can you key on timestamp.

@tk it's important that the value structure is the same in full and delta as ageneric broker will want to store an in coming delta and then provide it over REST with out needing to do any translation.

[jboynes]

Another application for this is to allow devices (sensors, actuators) to provide information about the values that they can produce without needing to be configured with their role in the model (i.e. which key their value maps to).

For example, an Acme THM-1 air sensor could announce itself with the message:

{
  "devices": {
    "acme.THM-1.12345": {
      "manufacturerName": "acme",
      "productName": "THM-1",
      "serialNumber": "12345"
    },
    "data": {
      "$schema": "http://devices.acme.example/signalk/thm-1.json"
    }
  }
}

The linked schema would describe the values it could emit, in this case a temperature and humidity:

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "id":"http://devices.acme.com/THM-1/signalk.json",
  "type":"object",
  "description":"Values produced by an Acme THM-1 Temperature/Humidity Sensor",
  "properties":{
    "temperature":{
      "description": "Air temperature",
      "$ref": "https://signalk.github.io/specification/schemas/definitions.json#/definitions/numberValue",
      "units": "K"
    },
    "humidity":{
      "description": "Relative humidity",
      "$ref": "https://signalk.github.io/specification/schemas/definitions.json#/definitions/numberValue",
      "units": "ratio"
    }
  }
}

A server receiving the announcement can retrieve the schema (either over the WAN, from installation files, or from the user installing it), determine what the value fields are, and prompt the installer for a mapping from the device to it's role on the vessel (e.g. that this sensor was actually installed in the master cabin and not the saloon).

The device would then send updates in structure conforming to the schema e.g.

{
  "devices": {
    "acme.THM-1.12345": {
      "data": {
        "temperature": {
          "value": 291.15
        },
        "humidity": {
          "value": 0.39
        }
      }
    }
  }
}

And the receiving server would be able to map that value to the vessels.23009999.environment.inside.masterCabin.temperature key.

[pod909]

Excellent

[pod909]

An aside but that final key is bit messy tho as it goes context + attribute + more context + attribute.

Vessels.2300000.inside.mastercabin.environment.temperature

... Would be better

[jboynes]

@pod909 said in #291 that this would not work with mixed attribute groups, "specifically where there is a node that sends values in more than one attribute group." I was thinking that would be covered by a message like this:

{
  "vessels": {
    "23009999": {
      "environment": {
        "inside": {
          "masterCabin": {
            "temperature": {
              "value": 293.15
            },
            "humidity": {
              "value": 0.39
            }
          }
        }
      },
      "navigation": {
        "logTrip": {
          "value": 43374
        },
        "log": {
          "value": 17404540
        },
        "courseOverGroundTrue": {
          "value": 172.9
        },
        "speedOverGround": {
          "value": 3.85
        }
      }
    }
  }
}

[pod909]

IT all makes sense but having multiple values in a single message leave us with no chance of using a standard broker out of the box.

By using the features brokers have for adding a prefix to the topic the context can be added to the topic automatically. The broker can be context aware.

This can be done when the messages are received or when they are shared between brokers. So long as a broker represents a single context that should be job done with out the need for a Signal K Server .... provided we stick to 1 value per message. Looks like some of them can add a timestamp to the topic as well.

I'm just looking to see if there is a broker that can configured in real time to choose a prefix by client id.

[tkurki]

The current delta is essentially key-value structure with a bit of metadata.

I have found that a lot easier and more definitive to work with than arbitrary hierarchies.

I believe @rob42 changed java server's internal structures from hierarchy to key-values following the same reasoning.

Are you proposing (a) replacing current delta with a hierarchical structure (b) adding a third format for SK values in messages ??

(a) would mean an extensive rewrite of all existing servers, gateways and applications. I don't quite see the point in that, as I fail to see the upside.

What value would we gain from (b) compared to using current delta message or an extension thereof?

[tkurki]

All that matters is the path to the key/property/attibute and this will be carried in different places depending on transport. The nested structure is only relevant for responses over REST.

Agree.

[jboynes]

@tkurki Agree on the key-value with metadata concept. However, the keys are not independent, there is a semantic hierarchy in the names of the keys used that is defined by the JSON-schema. That allows a user to synthesize key names for values not yet defined; for example, the information in the schema allows me to know the that key for the master's cabin temperature should be vessels.2309999.environment.inside.masterCabin.temperature rather than vessels.2309999.inside.mastercabin.environment.temperature

I disagree that "the nested structure is only relevant for responses over REST." A REST response is simply transfer of a representation, which is precisely what we are doing when transferring the same information using messages. The nested structure described above is the one defined for the Sparse Format.

Putting it another way, I'm proposing (c) Deprecate the Delta Format and use the existing Sparse Format to transfer updates

What we gain is:

• Only one message format for devices and servers to handle
• Specification of the well-known key structures using the JSON schemas
• Specification of value types using JSON schema
• Ability to validate messages against schema
• Smaller message size for networks with limited packet size (e.g. ZigBee, MQTT-SN)

When used in conjunction with the device metadata announcement above, it also provides a way to extend the keys and values in the model without a central registration authority and in a way that does not require custom code on a SK server for key mapping.

[pod909]

Agree that the key is valid signal K. Problem is the schema as defined is going to make things difficult when it comes to differentiating context from super attribute name

On Nov 14, 2016 17:06, "Jeremy Boynes" notifications@github.com wrote:

@tkurki https://github.com/tkurki Agree on the key-value with metadata concept. However, the keys are not independent, there is a semantic hierarchy in the names of the keys used that is defined by the JSON-schema. That allows a user to synthesize key names for values not yet defined; for example, the information in the schema allows me to know the that key for the master's cabin temperature should be vessels.2309999.environment. inside.masterCabin.temperature rather than vessels.2309999.inside. mastercabin.environment.temperature

I disagree that "the nested structure is only relevant for responses over REST." A REST response is simply transfer of a representation, which is precisely what we are doing when transferring the same information using messages. The nested structure described above is the one defined for the Sparse Format http://signalk.org/specification/master/data_model.html#sparse-format.

Putting it another way, I'm proposing (c) Deprecate the Delta Format and use the existing Sparse Format to transfer updates

What we gain is:

• Only one message format for devices and servers to handle
• Specification of the well-known key structures using the JSON schemas
• Specification of value types using JSON schema
• Ability to validate messages against schema
• Smaller message size for networks with limited packet size (e.g. ZigBee, MQTT-SN)

When used in conjunction with the device metadata announcement above, it also provides a way to extend the keys and values in the model without a central registration authority and in a way that does not require custom code on a SK server for key mapping.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/SignalK/specification/issues/292#issuecomment-260396006, or mute the thread https://github.com/notifications/unsubscribe-auth/AF0KFxjF7b7NpzRoJq4UM_WFQNiRTCBLks5q-JUqgaJpZM4Kw9sy .

[jboynes]

I don't follow - could you provide an example of the problem.

[jboynes]

Or are you referring to how context is defined, where in the Delta Format it's:

{
  "context": "vessels.230099999",
    "updates": [
      ...data goes here...
    ]
}

in the Sparse Format it's:

{
  "vessels": {
    "230099999": {
      ...data goes here...
    }
  }
}

[pod909]

Yep

The vessel is a context for an attribute or a sensor. Inside mastercabin is also a context. Environment is part of the key for an attribute. So is temperature.

The delta some seems to recognize this separation of context and attribute but some where along the line the 2 concepts have got mixed.

The problem is, for a sensor that doesn't know about the vessel or master cabin, what is the delta path/MQTT topic?

On Nov 14, 2016 17:34, "Jeremy Boynes" notifications@github.com wrote:

Or are you referring to how context is defined, where in the Delta Format it's:

{ "context": "vessels.230099999", "updates": [ ...data goes here... ] }

in the Sparse Format it's:

{ "vessels": { "230099999": { ...data goes here... } } }

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/SignalK/specification/issues/292#issuecomment-260403370, or mute the thread https://github.com/notifications/unsubscribe-auth/AF0KF_BUypfYfVERCzSnxOj3Tfb7sb-zks5q-JuHgaJpZM4Kw9sy .

[jboynes]

Ah. I agree that figuring out what the value of the context should be is a problem for both formats. I think that's a different issue than this one though as both Delta and Sparse formats provide a way for that to be included.

I think the proposal here is simpler though as the context is naturally included in the JSON.

For the Delta format, the receiver needs to concatenate the value of the top-level context property with the value of the path property in each update. There is syntactic ambiguity in that, for example, how to handle path components containing a . character. It is also harder to use a streaming parser as JSON does not guarantee that the context property will come before the updates.

[jboynes]

This also clears up ambiguity about how to handle values that are objects e.g. navigation.position which in keysWithMetadata is listed as three keys but in the JSON schema is a single position object. The schema says the source information is associated with the position type at path ...navigation.position but the documented key is navigation.position.latitude Further, for regular values the value of the value property in the update is used to set the value property of the key but the position type does not have such a property.

IOW, with the current format special handling is needed for values that are object typed which prevents portable use of unregistered keys (like the masterCabin one) unless they are scalars. My proposal allows any key/value combination, and allows it to be portably documented using JSON schema.

[pod909]

This causes a direct problem in delta as soon as there is a messy nesting and a sensor that doesn't know where it's installed.

For the full key vessels.12345.environment.inside. mastercabin.temperature whats the value is the context and what value is the path supposed to have?

On Nov 14, 2016 19:20, "Jeremy Boynes" notifications@github.com wrote:

Ah. I agree that figuring out what the value of the context should be is a problem for both formats. I think that's a different issue than this one though as both Delta and Sparse formats provide a way for that to be included.

I think the proposal here is simpler though as the context is naturally included in the JSON.

For the Delta format, the receiver needs to concatenate the value of the top-level context property with the value of the path property in each update. There is syntactic ambiguity in that, for example, how to handle path components containing a . character. It is also harder to use a streaming parser as JSON does not guarantee that the context property will come before the updates.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/SignalK/specification/issues/292#issuecomment-260433573, or mute the thread https://github.com/notifications/unsubscribe-auth/AF0KF3ypySOzvyeafV7n9RESsFnbykf4ks5q-LR5gaJpZM4Kw9sy .

[jboynes]

Not sure what you mean by a 'messy nesting' - remember this is a sparse structure that leaves other nodes unchanged.

A device that does not know where it's installed sends out messages in the devices sub-tree i.e.

{
  "devices": {
    "acme.THM-1.12345": {
      "data": {
        "temperature": {
          "value": 291.15
        },
        "humidity": {
          "value": 0.39
        }
      }
    }
  }
}

It is the vessel's SK server that knows that device is installed in the master cabin so using that knowledge it sends out the update to the vessel-wide model:

{
  "vessels": {
    "23009999": {
      "environment": {
        "inside": {
          "masterCabin": {
            "temperature": {
              "value": 291.15
            },
            "humidity": {
              "value": 0.39
            }
          }
        }
      }
    }
  }

[pod909]

My question was about delta

On Nov 14, 2016 21:33, "Jeremy Boynes" notifications@github.com wrote:

Not sure what you mean by a 'messy nesting' - remember this is a sparse structure that leaves other nodes unchanged.

A device that does not know where it's installed sends out messages in the devices sub-tree i.e.

{ "devices": { "acme.THM-1.12345": { "data": { "temperature": { "value": 291.15 }, "humidity": { "value": 0.39 } } } } }

It is the vessel's SK server that knows that device is installed in the master cabin so using that knowledge it sends out the update to the vessel-wide model:

{ "vessels": { "23009999": { "environment": { "inside": { "masterCabin": { "temperature" : {

"value": 291.15 }, "humidity": { "value": 0.39 } } } } } }

—

You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/SignalK/specification/issues/292#issuecomment-260469467, or mute the thread https://github.com/notifications/unsubscribe-auth/AF0KFz8dhIPG8RaQESXrgLdtxmkF_9Bjks5q-NOUgaJpZM4Kw9sy .

[jboynes]

@pod909 OK, sorry for the confusion.

My proposal for vessel-context-free devices is based on having a devices sub-tree in the global model so each one can establish its own context. I think you could achieve that in the Delta model in the same way, except that for that approach to work each device has to be able to publish the structure of its keyspace. I'm doing that with a reference to JSON schema in the discovery message; we'd need to invent a mechanism for that for the keysWithMetadata approach (and address the object value problem).

[pod909]

Totally agree with the separate list of devices. It's the keys with meta data that I disagree with, or the position the meta data has been given in the key I suppose. The meta data is really part of the context and should come between the vessel and the key, rather than muddying the key. It forces us down the route you've taken which is messy in terms of forcing nodes to deal with multiple values in a single message. A vessel.meta context can be given to devices with out any problems where as vessel.2039999.environment.inside.mastercabin can not as not all the devices in the master cabin will deal with environment attributes.

Stick to what's defined in the group schema and change the keysWithMetadata from environment..temperature to .environment.temperature) and there is no problem with sensor knowing that it's publishing to environment/temperature ....

.... or the configuration of device to context mapping on device discovery as it's limited to the location of the device + meta data with out requiring the user to remember that they have to insert part of the key in the middle.

Since the device has to implement Signal K JSON for the hello and update I don't see there being an issue with it also knowing the topic names to send each value update too.

I don't see a lot of value in the linked schema here: { "devices": { "acme.THM-1.12345": { "manufacturerName": "acme", "productName": "THM-1", "serialNumber": "12345" }, "data": { "$schema": "http://devices.acme.example/signalk/thm-1.json" } } }

When that arrives at the server the server will need to recreate each data property from the schema so that a "context" property can be added to hold the mapping.

The hello might as well detail them out with out the need to pull down an additional document (with context as added through the discovery process)...

{ "devices": { "acme.THM-1.12345": { "manufacturerName": "acme", "productName": "THM-1", "uuid": "12345" }, "data": { "environment.temperature":{ "description": "Air temperature", "$type": "https://signalk.github.io/specification/schemas/definitions.json#/definitions/numberValue", "units": "K", "context":"vessels.20309999.inside.mastercabin" }, "environment.humidity":{ "description": "Relative humidity", "$type": "https://signalk.github.io/specification/schemas/definitions.json#/definitions/numberValue", "units": "ratio", "context":"vessels.20309999.inside.mastercabin" } } } }

[pod909]

My main point is still that using the sparse format to carry the update stops a generic broker from being used. Carry the value updates individually and a generic broker can be used. The ActiveMQ broker Rob highlighted also provides the REST interface which would, I think, provide consumers to extract aggregated information in effectively sparce Singal K.

That may I suspect need some changes to the value object(s) but, in terms of adoption, a solution based on out of the box enterprise strength solutions is going to be a winner every time.