MQTT sensor connection example implementation

[toqduj]

Based on a request by @tkurki (requested in this comment in #288 ), here's one possible implementation for letting sensors send information over Wifi & MQTT to the SK receiver.

A short summary: I've created a straightforward GPS sensor (for deck mounting) by connecting an Adafruit Ultimate GPS (based on the MTK3339 chip) to an Arduino MKR1000 and a LiPo back-up battery. The code for running the sensor is available here. It is not advanced code, but it is simple and works.

The code uses the following key libraries alongside standard ones:

• PubSubClient for MQTT, modified to handle larger messages
• ArduinoJson for constructing the SK deltas
• PString for handling the JSON string before sending.
• TinyGPS++ for interpreting the GPS messages

The sensor module "sender" connects to a "receiver", a Raspberry Pi running OpenPlotter V0.9 (OP) in the following steps:

1. The sensor connects to the OP-instantiated Wifi access point, currently using hardcoded information (SSID, WPA password).
2. The sensor connects to the OP-instantiated Mosquitto MQTT server, currently using hardcoded information (IP, port, MQTT username, MQTT password). Once I find a compatible mDNS discovery library, the IP and port can be autofound.
3. The sensor subscribes to an MQTT channel, "gpsget", set up for this purpose.
4. The sensor outputs SK deltas to this channel, updating one variable per message (five messages in total for five navigation keys: position.latitude, position.longitude, position.altitude, speedOverGround, and courseOverGroundTrue). Due to the design of ArduinoJson, I could not get it to put multiple key:value combinations in a single message. Each message is about 160 bytes long, and contains the following information (which adheres to the SK delta format as far as I could understand):

{ "updates": [{ "source": { "timestamp": "2016-11-09T09:15:36Z", "$source": "deck.gps001" }, "values": [{ "path": "navigation.speedOverGround", "value": 0.8385444 }] }] }

The timestamp is the time which the GPS returns, i.e. the time when the reported values were valid, and so lags a little behind real time.

Currently, OpenPlotter does not parse incoming SK deltas yet, as explained by @sailoog in this comment. For now, I am waiting for that to be implemented, as MQTT appears to be a nice way of sending information.

There is a power-saving feature: if the GPS does not have a valid fix, the Wifi disconnects. Every time before sending a message, it is checked if Wifi and MQTT are connected (otherwise it reconnects). As such, it uses on average about 50-60 mA at 5V (0.3W). This can be reduced in the future by exploiting the built-in RTC as a watchdog timer.

[tkurki]

Still nitpicking here: timestamp is totally relevant for position, which needs structured value anyway, not so much for speedOverGround, which is an average over time.

[jboynes]

The broker can be discovered via mDNS and MQTT-SN adds the ADVERTISE and SEARCHGW messages as an alternative for non-IP networks.

It would be easier if the sensor could just be added to the network and have data show up without additional configuration of the device. Including a device-specific id in the topic or in messages would allow that as the configuration could be performed at the aggregator or display where there's a UI.

[tkurki]

Yes, _mqtt._tcp for discovery and then either map whatever topic the sensor is publishing to SK path or get the sensor to use SK topic that maps to the SK data model.

[tkurki]

For FWIW I created a naive SK Node server plugin that accepts plain values in topics of the form vessels/self/environment/temperature/outside.

Naive in the sense that there is no error handling or (optional?) validation against the SK data model.

https://github.com/tkurki/signalk-mqtt-gw

[pod909]

a/ { "value":3.1234 } is a valid Signal K object. 3.12345 isn't.

b/ If the string 3.12345 is better, why isn't the 32 bit real binary better again?

c/ By being consistent in the standard across transports and sending valid Signal K you're separating the transport from how the content is consumed/processed. You don't need to build a special case into the "message processor".

d/ Some nodes may transmit more verbose data. Better for the recipient not to have to work out what it's receiving.

[pod909]

"vessels/self/environment/temperature/outside"

Not sure this can be a topic. "environment/temperature/outside" yes, but the context will usually be unknown by a sensor node. That's in terms of how they are provided. How they are consumed? Different thing I guess.

If "environment/temperature/outside" is an attribute in the Signal K standard. "vessels/self" is the context for that attribute. It's also the context for the node in terms of it logical grouping, in this case being physically located on the default vessel.

For sensors sending values the topic needs in most cases to be the path to the attribute with in a context, with the context derived from a brokers knowledge of the node. For consuming a topic it needs to be the path across the context and attribute.

There's an obvious answer here but I don't think people will like it. That's to basically revers the order things appear in the Signal K and put the attribute at the top, containing the value, containing a source which has the context.

The attribute can then be carried as the topic with the "value" as a value ... augmented with source and context either in the message or by the server

[tkurki]

"vessels/self/environment/temperature/outside"

This allows the same topic structure to be used for a broker that receives data over MQTT from multiple vessels. Think using a generic cloud MQTT broker as a race server.

[tkurki]

By being consistent in the standard across transports and sending valid Signal K

As far as I know the structure { "value":3.1234 } is not used anywhere in the current SK spec software, so I fail to see the consistency here.

Better for the recipient not to have to work out what it's receiving.

I don't understand how { "value":3.1234 } accomplishes that.

[pod909]

It's defined in the definitions schema, broken out for different primitive types and it's part of the signal-delta object.

The separation of the path for the context and path for the 'attribute' is how the sk-delta has been approached.

[pod909]

"This allows the same topic structure to be used for a broker that receives data over MQTT from multiple vessels. Think using a generic cloud MQTT broker as a race server."

Agree for subscription topic. Publish topic? Not so much. See edit above.

[pod909]

Look like the steps need to be something like:

Receive a PUBLISH message from a client (sensor node). Add/synchronize the timestamp. If not present add a source based on the client id/CONNECT credentials. Look up the context for that source. Merge the value for the attribute in the topic in the context found from the source. Make the combined context+attibute topic available to subscribing clients.

[jboynes]

The latency and availability problems of a WAN link mean there would need to be a broker on the vessel to handle local message delivery. Determining which updates to forward off the boat requires knowledge of the SK data and so would be better handled by a SK server rather than a generic MQTT broker.

Taking the race scenario, there could be multiple MQTT domains, each with their own broker:

• one on each vessel for local data
• one for each team for private communication
• one for the race committee
• one or more open to the public or other parties The domains would bridged by servers that act as clients to multiple brokers. A server on a vessel would bridge between the local data and the race committee's, sending the data the race rules require when a WAN link was available.

[jboynes]

@pod909 I don't see how that works using a generic MQTT broker. The client's CONNECT is sent to the broker and the client id and credentials it contains are not visible to other clients. When it PUBLISHes a message, the recipient could not create a source entry as it would not know the sender's id. To support that would require a custom MQTT broker that was aware of SK topics and/or messages.

To use a generic MQTT broker, a SK server would be a MQTT client and do something more like:

• Send a CONNECT to the MQTT broker
• Send a SUBSCRIBE for one or more topics
• Receive a PUBLISH message from the broker with an update
• Use the topic name or message content to determine the SK key(s) to update
• Apply the updates to its model

Updates the model might result in it PUBLISHing other update messages via the broker. However, those could be triggers by any updates to the model including those coming from non-MQTT transports like 0183 or N2K buses.

[tkurki]

@jboynes while I agree with your different brokers approach you don't need to go 100% MQTT, just use it where it makes sense, like possibly the WAN connection.

[pod909]

@jb It doesn't work for a generic MQTT broker. The only way for it to work would be for the sensor to be configured with the context so that it could include it in the topic.

You've described the "receive an update" flow. I was attempting the client sends an update in terms of what the broker sees.

[jboynes]

To send an update using a generic broker, the client would do this:

• Send a CONNECT to the MQTT broker
• Create an update message for a value and select a topic
• Send a PUBLISH message containing the update to the broker (including the topic name) The broker would then publish the update message to all clients that had subscribed to the topic, and the clients would then process the updates as above.

[rob42]

Race boats are very like truck monitoring, see https://eclipse.org/kura/

It is very unlikely that a boat will NOT have an onboard messaging server of some sort. Simple sensors will just ping values, and wont work in intermittently connected environments. Adding store and forward to all sensors is too hard and expensive, so there will always be a store and forward messaging server onboard. It will probably push to the cloud, which will route to race head-quarters.

Hence the messaging server will at a minimum understand MQTT or similar, and likely understand signalk too, as the incremental cost is small.

Sending updates is a use case for simply sending valid signalk over MQTT as transport, and assumes the target device is smart enough to deal with signalk messages. In that case a generic MQTT broker just passes the message, which will work ok.

If we have an MQTT device that simply pings out values to an MQTT queue, completely unaware of signalk, then its unlikely we will be trying to communicate back to it with signalk

eg if it doesnt understand signalk, then we need to communicate to it via custom gateway that can translate signalk to sensor-speak anyway

[pod909]

Totally agree with that. The only thing is the context (the identity of the vessel) is only known to the broker. Not the individual sensors. If we are saying the context can be inferred by the consumers from the identity of the broker that also works, particularly if there is some way of for the broker to tell the consumer what it's context is.

On Nov 14, 2016 01:44, "rob42" notifications@github.com wrote:

Race boats are very like truck monitoring, see https://eclipse.org/kura/

It is very unlikely that a boat will NOT have an onboard messaging server of some sort. Simple sensors will just ping values, and wont work in intermittently connected environments. Adding store and forward to all sensors is too hard and expensive, so there will always be a store and forward messaging server onboard. It will probably push to the cloud, which will route to race head-quarters.

Hence the messaging server will at a minimum understand MQTT or similar, and likely understand signalk too, as the incremental cost is small.

Sending updates is a use case for simply sending valid signalk over MQTT as transport, and assumes the target device is smart enough to deal with signalk messages. In that case a generic MQTT broker just passes the message, which will work ok.

If we have an MQTT device that simply pings out values to an MQTT queue, completely unaware of signalk, then its unlikely we will be trying to communicate back to it with signalk

eg if it doesnt understand signalk, then we need to communicate to it via custom gateway that can translate signalk to sensor-speak anyway

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/SignalK/specification/issues/291#issuecomment-260230608, or mute the thread https://github.com/notifications/unsubscribe-auth/AF0KF6PhCGrY-kC6aQ5V41ELOtcAcWOhks5q97zugaJpZM4KtV6- .

[jboynes]

@pod909 A generic MQTT broker does not know anything about SK or its context. It is known by the SK Server which is a /client/ of the broker just like the sensor.

[pod909]

So the SK Server is a demon receiving packets from generic attribute topics, translating them to context specific topics, sorting time stamping and pushing them back to the broker?

...leaving the broker to handle all the MQTT and REST etc to the outside world

On Nov 14, 2016 04:43, "Jeremy Boynes" notifications@github.com wrote:

@pod909 https://github.com/pod909 A generic MQTT broker does not know anything about SK or its context. It is known by the SK Server which is a /client/ of the broker just like the sensor.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/SignalK/specification/issues/291#issuecomment-260247375, or mute the thread https://github.com/notifications/unsubscribe-auth/AF0KFxZid1q-SQd-mG5z92vY41lDTWw4ks5q9-bygaJpZM4KtV6- .

[jboynes]

For the SK Server's role, yes.

The local MQTT broker handles MQTT transport on the vessel; the SK Server would use (as a client) a different MQTT broker shore-side (e.g. in the cloud) when talking to remote peers (shore-based or SK servers on other vessels).

For REST, the SK server would provide the HTTP endpoint as it's a direct view of its model.

[pod909]

Works for me.

Reinforces the need to unify the value object in the delta and full sk as they are going to be the same thing on a generic broker.

On Nov 14, 2016 05:10, "Jeremy Boynes" notifications@github.com wrote:

For the SK Server's role, yes.

The local MQTT broker handles MQTT transport on the vessel; the SK Server would use (as a client) a different MQTT broker shore-side (e.g. in the cloud) when talking to remote peers (shore-based or SK servers on other vessels).

For REST, the SK server would provide the HTTP endpoint as it's a direct view of its model.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/SignalK/specification/issues/291#issuecomment-260249757, or mute the thread https://github.com/notifications/unsubscribe-auth/AF0KF4w3Xdg7F4BA8DRN3oPD7vN6b6qQks5q9-1GgaJpZM4KtV6- .

[jboynes]

Agreed on unification. I have a proposal for that but am going to create a new issue as it's not MQTT specific and this one is long enough already.

[pod909]

I get how the zero config would work now.

On the vessel side broker you just need a 'sources' topic that each sensor sends it's hello to when it join.

That just leaves discovery of the broker using a standard ip address/port plus mDNS.

On Nov 14, 2016 05:20, "Jeremy Boynes" notifications@github.com wrote:

Agreed on unification. I have a proposal for that but am going to create a new issue as it's not MQTT specific and this one is long enough already.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/SignalK/specification/issues/291#issuecomment-260250678, or mute the thread https://github.com/notifications/unsubscribe-auth/AF0KF1MtuF5qXPRr1DRXzLMSXzQUfr_Dks5q9--wgaJpZM4KtV6- .

[jboynes]

As an example of using a single topic, I set up a "device" and a "server" communicating over MQTT and had the broker log out the messages sent to that topic.

CONNECT from SKServer
SUBSCRIBE from SKServer on signalk/update
CONNECT from acme.THM-1.12345
PUBLISH from acme.THM-1.12345 on signalk/update: {"devices":{"acme.THM-1.12345":{"manufacturerName":"acme","productName":"THM-1","serialNumber":"12345"},"data":{"$schema":"http://devices.acme.example/signalk/thm-1.json"}}}
PUBLISH from acme.THM-1.12345 on signalk/update: {"devices":{"acme.THM-1.12345":{"data":{"temperature":{"value":293.15},"humidity":{"value":0.39}}}}}
PUBLISH from SKServer on signalk/update: {"vessels":{"23009999":{"environment":{"inside":{"masterCabin":{}}}}},"temperature":{"value":293.15},"humidity":{"value":0.39}}
PUBLISH from acme.THM-1.12345 on signalk/update: {"devices":{"acme.THM-1.12345":{"data":{"temperature":{"value":294.15},"humidity":{"value":0.39}}}}}
PUBLISH from SKServer on signalk/update: {"vessels":{"23009999":{"environment":{"inside":{"masterCabin":{}}}}},"temperature":{"value":294.15},"humidity":{"value":0.39}}
PUBLISH from acme.THM-1.12345 on signalk/update: {"devices":{"acme.THM-1.12345":{"data":{"temperature":{"value":294.15},"humidity":{"value":0.39}}}}}
PUBLISH from acme.THM-1.12345 on signalk/update: {"devices":{"acme.THM-1.12345":{"data":{"temperature":{"value":294.15},"humidity":{"value":0.39}}}}}
PUBLISH from acme.THM-1.12345 on signalk/update: {"devices":{"acme.THM-1.12345":{"data":{"temperature":{"value":294.15},"humidity":{"value":0.39}}}}}
PUBLISH from acme.THM-1.12345 on signalk/update: {"devices":{"acme.THM-1.12345":{"data":{"temperature":{"value":293.15},"humidity":{"value":0.39}}}}}
PUBLISH from SKServer on signalk/update: {"vessels":{"23009999":{"environment":{"inside":{"masterCabin":{}}}}},"temperature":{"value":293.15},"humidity":{"value":0.39}}

When it starts, the Acme device sends out its info (as described in #292 and then starts sending device-specific values. The SK server is "configured" to know that the 12345 device is installed in the "masterCabin" and maps the values from the device to the vessel's tree. When the temperature reading changes, it updates the tree and sends out another update for any other interested parties (which would include those not using MQTT e.g. those using WebSocket).

[pod909]

The mixed attibute/context key definitely isn't going to work. Specifically where there is a node that sends values in more than one attribute group e.g. navigation (location and speed) and environment (wind)

[pod909]

Not sure about the aggregated update going back out either. That makes it difficult for consumer to subscribe to just the information they are interested in. Perhaps both would be good. The full message going out on signal/updates and individual messages going out to value specific topics?

[pod909]

Looking at hw wildcards work in topics (http://www.hivemq.com/blog/mqtt-essentials-part-5-mqtt-topics-best-practices) makes the case for more verbose topics with the update from the Signal K server being sent one per value. Still prefer single value per message from the device as well

Topics for updates from the device: signalk/updates/devices/acme-ABC-123456/environment/temperature Updates from signal server having resolved the context: signalk/updates/vessels/203099999/inside/mastercabin/environment/temperature

[pod909]

It t seems that MQTT brokers can deal with this already using a feature called mount_point. This appends a string to the topic received from a client based on the client id.

So the sensor should send a value JSON to a topic environment/temperature and this will be automatically mapped to vessels/203099999/inside/mastercabin/environment/temperature provided that the client id has been given the mount_point vessels/20309999/inside/mastercabin/

Same with establishing a bridge between brokers. A broker can add a prefix automatically.

[jboynes]

@pod909 I am using a standard broker (Moquette) to prototype this so it does work, so far at least.

I did not find anything in the MQTT spec about mount_point and from what I can find on Google it looks like a configuration option for certain brokers designed to isolate ports. From IBM's doc:

A mount point isolates the publications and subscriptions exchanged by MQTT clients using one listener port from MQTT clients connected to a different listener port.

In their broker, it appears to be a constant and is not derived from the client id. It looks like a mechanism for running multiple virtual brokers on one server.

[pod909]

Don't doubt that your approach works and is relatively straightforward. But it requires a lot more work from consumers in terms of breaking down messages, double the message load performance issues around the signal K server. Why go to that extent when you can do it with out if the box software.

Mounting _point is documented in the mosquito config doco. You're right tho it's universal so a broker would be limited to representing a single context ... Or a single port per context anyhow.

Prefixing the top when connecting brokers is also in the doco.

On Nov 14, 2016 16:24, "Jeremy Boynes" notifications@github.com wrote:

@pod909 https://github.com/pod909 I am using a standard broker (Moquette) to prototype this so it does work, so far at least.

I did not find anything in the MQTT spec about mount_point and from what I can find on Google it looks like a configuration option for certain brokers designed to isolate ports. From IBM's http://www.ibm.com/support/knowledgecenter/SS9D84_1.0.0/com.ibm.mm.tc.doc/tc60522_.htm doc:

A mount point isolates the publications and subscriptions exchanged by MQTT clients using one listener port from MQTT clients connected to a different listener port.

In their broker, it appears to be a constant and is not derived from the client id. It looks like a mechanism for running multiple virtual brokers on one server.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/SignalK/specification/issues/291#issuecomment-260383397, or mute the thread https://github.com/notifications/unsubscribe-auth/AF0KFwih2Aw6cM-52VO3-Z7UBh-yr7_jks5q-Is-gaJpZM4KtV6- .

[jboynes]

It looks like we might need multiple sets of topics to handle different types of client e.g.

• devices (sensors, actuators) dealing with a small number of device-specific values
• SK servers (gateways) dealing with a larger context such as a vessel or race
• user agents (browser or native apps, displays) dealing with portions of the context relevant to their UI

There may also be differences based on the network communication is occurring over. For example, environmental sensors (e.g. temperature) might be limited by the power and bandwidth available to short, infrequent messages (e.g. over ZigBee). On the other hand, rapid update messages (e.g. vessel performance, radar data) may require use of a high-power, high-bandwidth, low-latency network (e.g. wired Ethernet). Remote comms will be affected by link availability, speed, latency and cost.

Neither of the two extremes, one topic for all data or separate topic for every value, seem appropriate. Instead a hybrid approach is needed. To that end, a proposal for well-known topic names and usage ...

• signalk/discovery: a low-throughput topic used for discovery messages (TBD)
• signalk/updates: a high-throughput topic for deltas to the entire model
• signalk/updates/vessels/{vesselId}: messages related to a specific vessel
• signalk/updates/vessels/{vesselId}/{group}: messages related to a specific data group
• signalk/updates/devices/{deviceId}: messages related to a specific device, throughput depends on device
• signalk/updates/user/{agentId}: messages related to a specific user agent (app, display, ...)
• signalk/subscriptions/{subscriptionId}: a subscription as per the Subscription Protocol where portion of the model is transferred like it would be over a socket, only using MQTT messages instead of a stream

The {group} for data is limited to the first level of keys in the model hierarchy. This allows reasonable groupings of information by purpose (e.g. navigation) without requiring separate messages to be sent for each individual value - for example, with navigation, position, SOG, COG, heading, ... values could be sent as a single message given most subscribers would want all of them. Conceptually, it's similar to a Parameter Group in N2K.

Using that scheme, the message exchange above would be rerouted as follows:

CONNECT from SKServer
SUBSCRIBE from SKServer on signalk/discovery
CONNECT from acme.THM-1.12345
PUBLISH from acme.THM-1.12345 on signalk/discovery: {"devices":{"acme.THM-1.12345":{"manufacturerName":"acme","productName":"THM-1","serialNumber":"12345"},"data":{"$schema":"http://devices.acme.example/signalk/thm-1.json"}}}
SUBSCRIBE from SKServer on signalk/updates/devices/acme.THM-1.12345
PUBLISH from acme.THM-1.12345 on signalk/updates/devices/acme.THM-1.12345: {"devices":{"acme.THM-1.12345":{"data":{"temperature":{"value":293.15},"humidity":{"value":0.39}}}}}
PUBLISH from SKServer on signalk/updates: {"vessels":{"23009999":{"environment":{"inside":{"masterCabin":{}}}}},"temperature":{"value":293.15},"humidity":{"value":0.39}}
PUBLISH from SKServer on signalk/updates/vessels/23009999: {"vessels":{"23009999":{"environment":{"inside":{"masterCabin":{}}}}},"temperature":{"value":293.15},"humidity":{"value":0.39}}

Alternatively, the SK Server could have subscribed up front to signalk/updates/devices/# if it knew it wanted to receive all device updates. Similarly, if the device wanted to receive messages (e.g. it was an actuator) it would SUBSCRIBE to signalk/updates/devices/acme.THM-1.12345

[jboynes]

I added that to my prototype and tweaked the message formats a bit giving:

CONNECT from SKServer
SUBSCRIBE from SKServer on signalk/discovery
CONNECT from acme.THM-1.12345
PUBLISH from acme.THM-1.12345 on signalk/discovery: {"devices":{"acme.THM-1.12345":{"manufacturerName":"acme","productName":"THM-1","serialNumber":"12345"},"data":{"$schema":"http://devices.acme.example/signalk/thm-1.json"}}}
PUBLISH from acme.THM-1.12345 on signalk/updates/devices/acme.THM-1.12345: {"data":{"temperature":{"value":292.15},"humidity":{"value":0.39}}}
SUBSCRIBE from SKServer on signalk/updates/devices/acme.THM-1.12345
PUBLISH from acme.THM-1.12345 on signalk/updates/devices/acme.THM-1.12345: {"data":{"temperature":{"value":292.15},"humidity":{"value":0.39}}}
PUBLISH from SKServer on signalk/updates/vessels/23009999/environment: {"inside":{"masterCabin":{"temperature":{"value":292.15},"humidity":{"value":0.39}}}}
PUBLISH from acme.THM-1.12345 on signalk/updates/devices/acme.THM-1.12345: {"data":{"temperature":{"value":292.15},"humidity":{"value":0.39}}}
PUBLISH from acme.THM-1.12345 on signalk/updates/devices/acme.THM-1.12345: {"data":{"temperature":{"value":292.15},"humidity":{"value":0.39}}}
PUBLISH from acme.THM-1.12345 on signalk/updates/devices/acme.THM-1.12345: {"data":{"temperature":{"value":293.15},"humidity":{"value":0.39}}}
PUBLISH from SKServer on signalk/updates/vessels/23009999/environment: {"inside":{"masterCabin":{"temperature":{"value":293.15},"humidity":{"value":0.39}}}}
PUBLISH from acme.THM-1.12345 on signalk/updates/devices/acme.THM-1.12345: {"data":{"temperature":{"value":293.15},"humidity":{"value":0.39}}}

The primary change is that this uses the topic name as the context for the message so the outer parts of the model do not need to be sent every time, which makes the messages simpler and easier to understand. I think it will make source attribution harder though.

[pod909]

Now I can select topics for the vessel and environment but I can subscribe to just inside or temperature

[jboynes]

Not on the common broadcast topics, no.

That was the idea for the signalk/subscriptions/{subscriptionId} topics where two peers would use the subscription mechanism to select which SK keys they were interested in, just like they would over a streamed connection (TCP or WebSocket). That allows arbitrary groupings of values rather than being tied to the structure of the hierarchy.

This also avoids the need to send every value update as a separate message, avoiding problems with congestion, additional work needed to correlate messages that could arrive out of order, etc.

I'll follow up with an illustration of subscription but it might take a couple days to pul together now real life has kicked in.

[pod909]

MQTT is a small packet protocol with brokers tuned to handle lots of small packets. Either its up to the job and doesn't require any additional engineering or its not.

No size is saved with the larger messages. If anything they are heavier and stop load balancing and self optimisation by the broker.

E.g. you might want to give extra resources to carrying temperature and less to humidity.

On Nov 15, 2016 06:46, "Jeremy Boynes" notifications@github.com wrote:

Not on the common broadcast topics, no.

That was the idea for the signalk/subscriptions/{subscriptionId} topics where two peers would use the subscription mechanism to select which SK keys they were interested in, just like they would over a streamed connection (TCP or WebSocket). That allows arbitrary groupings of values rather than being tied to the structure of the hierarchy.

This also avoids the need to send every value update as a separate message, avoiding problems with congestion, additional work needed to correlate messages that could arrive out of order, etc.

I'll follow up with an illustration of subscription but it might take a couple days to pul together now real life has kicked in.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/SignalK/specification/issues/291#issuecomment-260561082, or mute the thread https://github.com/notifications/unsubscribe-auth/AF0KFyuFe98E-xXWEaxjJ_eKFJ9FWIvkks5q-VUqgaJpZM4KtV6- .

[pod909]

*can't ... That's forcing me to take data I don't care about, which increases the number of packets sent and their size

[toqduj]

I'm wondering if network congestion (at least within the boat) is really going to be that much of a problem. We have the option of using MQTT QOS in the sensor-sent data, to ensure the critical data is delivered.

As for the multiple values or single values per message, I didn't get any further with incorporating multiple values in a single delta JSON, so I'll be sending five deltas instead of one. Each delta is only 160 bytes, though, so it's hardly movie streaming.

@jboynes regarding your proposed MQTT topic set, would I publish my sensor SK deltas (as described way above) in

• signalk/updates/vessels/{vesselId}, or
• signalk/updates/devices/{deviceId}

and is deviceID an arbitrary ID set by the sensor? Right now, I only have unidirectional MQTT communication going from sensor --> broker. If I need to negotiate some ID with the broker, the code will be (quite a bit?) more complex.

B.

On Tue, Nov 15, 2016 at 8:34 AM, pod909 notifications@github.com wrote:

*can't ... That's forcing me to take data I don't care about, which increases the number of packets sent and their size

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/SignalK/specification/issues/291#issuecomment-260568214, or mute the thread https://github.com/notifications/unsubscribe-auth/AFMoydtw22h1DaCK4nS-ScP3BExgJbyQks5q-WCFgaJpZM4KtV6- .

[jboynes]

@toqduj You could do either depending on how much configuration your sensor supported:

• the {deviceId} option was meant to support sensors that did not need configuration, or which did not have a UI for doing so. The SK server defines how the sensor is associated with the boat. In my example I had a simple thermometer that derived the {deviceId} from its serial number. It needs to provide a unique client_id to use MQTT so used the same id for both. An installer configured the SK server with the information that it was installed in the "masterCabin"
• the {vesselId} option requires configuring the device itself during installation with both the vessel id and where the sensor is installed. I thought that was hassle for a simple thermometer but might be something a more complex device like a MFD or AIS transponder could support as they have a UI/need configuration anyway

No negotiation of the id needed in either case.

Re congestion, I was primarily thinking of sensor networks like ZigBee with much lower bitrates than Wifi, IIRC around 250kbit/s raw or 120kbit/s effective. If your GPS sent that 160 byte message at 10Hz it would consume ~10% of the effective bandwidth. Even with Wifi I can imagine contention in a crowded marina.

edit: each message was 160 bytes and it was sending 5 of them at a time. At 10Hz that would use ~60% of the ZigBee bandwidth

[pod909]

Which is why there is continued support for wired installations from manufacturers .

On Nov 16, 2016 04:42, "Jeremy Boynes" notifications@github.com wrote:

@toqduj https://github.com/toqduj You could do either depending on how much configuration your sensor supported:

• the {deviceId} option was meant to support sensors that did not need configuration, or which did not have a UI for doing so. The SK server maps knows how the sensor is associated with the boat. In my example I had a simple thermometer that derived the {deviceId} from its serial number.It needs to provide a unique client_id to use MQTT so used the same id for both. An installer configured the SK server with the information that it was installed in the "masterCabin'
• the {vesselId} option requires configuring the device itself during installation with both the vessel id and where the sensor is installed. I thought that was hassle for a simple thermometer but might be something a more complex device like a MFD or AIS transponder could support as they have a UI/need configuration anyway

No negotiation of the id needed in either case.

Re congestion, I was primarily thinking of sensor networks like ZigBee with much lower bitrates than Wifi, IIRC around 250kbit/s raw or 120kbit/s effective. If your GPS sent that 160 byte message at 10Hz it would consume ~10% of the effective bandwidth. Even with Wifi I can imagine contention in a crowded marina.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/SignalK/specification/issues/291#issuecomment-260845569, or mute the thread https://github.com/notifications/unsubscribe-auth/AF0KF5aPqWtaxJ1PzqP8mmVQ9mtjRPdkks5q-nuXgaJpZM4KtV6- .

[pod909]

Perhaps there is a compromise here if we say that part or all of the key may be carried in the topic with the remainder carried as a fragment in the message?

The device publish would work as proposed by in.

May be with an addition of a time sync topic and message so that the server can resynchronize any locally based line timestamps from not RTC nodes.

[pod909]

(God damn predictive text)

[pod909]

Some proposals for MQTT are included in RFC 0005. Topics are suggested that are consistent with the URLs for the delta REST API

[toqduj]

This discussion is carried forward to RFC 0005.