[WIP] Proposal: SSE API

[neftaly]

This is a proposal to add server-sent events on top of the current REST API, in a backwards-compatible manner.

Advantages over WS streaming API:

• Allows for metadata updates
• Simpler client implementation
• Uses established web standards (WHATWG SSE + RFC-7396)
• Complements the existing REST API (as opposed to RPC-style WS API)

[sbender9]

Note that metadata can be (and is in node server) updated via WS.

[sbender9]

You would need to do full deltas, not just simple key/value pairs. Clients need the context and source information.

[sbender9]

I would think it would also need a way to subscribe to specific paths and contexts.

[neftaly]

How do WS metadata updates work? I remember being told it wasn't possible.

[neftaly]

Deltas apply directly to the JS object. The example I used is just cribbed off http://signalk.org/specification/0.0.1-12/doc/rest_api.html (i.e. GET /signalk/v1/api/vessels/123456789/navigation/speedThroughWater). I should probably clarify this, or update the example to be applied to the root of the full model.

[neftaly]

Path/context subscription is done in the standard REST manner (eg the GET request path above).

[sbender9]

I think context would still be an issue. If you get /signalk/v1/api/vessels/, then you won't have any way to know which vessel the data is for. Right?

[neftaly]

The vessel ID would be part of the patch. For example:

GET /signalk/v1/api/vessels/

# Initial state
data:{"urn:mrn:signalk:uuid:c0d79334-4e25-4245-8892-54e8ccc8021d":{"electrical":{"batteries":{"1":{"voltage":{"meta":{"label":"Battery Voltage","units":"V","min":0,"max":15,"zones":[{"upper":10.5,"state":"alarm","message":"Fault: very low voltage"},{"lower":10.5,"upper":11.5,"state":"warn","message":"Low voltage, battery near empty"},{"lower":11.5,"upper":12,"state":"alert","message":"Low voltage, battery charge low"},{"lower":12.8,"upper":14.5,"state":"alert","message":"Charging"},{"lower":14.5,"upper":15,"state":"alarm","message":"Voltage too high"},{"lower":15,"state":"alarm","message":"Voltage too high, out of range"}]},"value":14.56,"$source":"n2kFromFile.129","timestamp":"2014-08-15T19:08:24.376","pgn":127508},"capacity":{"timeRemaining":{"value":null,"$source":"n2kFromFile.129","timestamp":"2014-08-15T19:08:22.231","pgn":127506}},"current":{"value":33,"$source":"n2kFromFile.129","timestamp":"2014-08-15T19:08:24.376","pgn":127508},"temperature":{"value":299. ...

# Update delta
data: "urn:mrn:imo:mmsi:258858000":{"navigation":{"speedOverGround":{"value": 7.48}}}

[tkurki]

A novel idea and warrants some thinking.

For individual data items this would probably work pretty well. Fire a request, get metadata and the latest values and then act on the events. But do you create one http connection per item? A typical case might be that the user has 10 to 20 data items on the dashboard, which simply would not work in browser environment.

It would be a pretty clean way to have a client side full tree up to date, even if merging complex object structures is computationally pretty intensive.

But at least I have grown to value the full model less and less. Just having a client side full tree is not much in itself: you need to take action, usually update some UI component or take some other action like raise a notification. After the merge you have all the data, but what do you do with that then? You need to dispatch an action based on what changed, but the updated full model doesn't give you that. The incoming data item has that information, it has just the new data. But if the contract is that the incoming data may contain arbitrary changes, dispatching an action based on it is a lot more complex than on the explicit ws delta data structure.

The client software that I've written work on dispatching some action based on the string valued path. The logic is very straightforward: here's the path, here's new value, here's the metadata.

In contrast the full model has caused a lot of confusion and debate, witness the still open and some closed specification issues. The most notable one is "what is a value". Sending full structures sort of bypasses that nicely and on the other hand ignores the problem totally. What is the value that actually changed in the incoming data? If you request self's data and get updates you need to figure out what actually changed and what part of the UI should respond to that change, unless you rerender the whole ui for every received event, which often is prohibitively expensive.

I'd like to hear other's views on this.

One practical way forward would be to write a server plugin/version for this and implement a dashboard-like simple UI against it and see what kind of complexity you end up with.

Opening a ws connection and getting a feed for self's data is not very complicated either, I don't see much difference in simplicity to your SSE example:

var ws = new WebSocket('ws://' + window.location.host + "/signalk/v1/stream")
ws.onmessage = function(event) {
  console.log(event.data)
}

Simpler client implementation

So I'm not yet convinced, when you've built the whole thing. The goal is not to have an in memory full model in the client, but do something with the data.

PS. if you want the full tree in the client it is very easy to run the same full construction logic that the server uses in the client on the ws deltas.

[neftaly]

I have a module that more-or-less gets you this idea. The client that uses it could be swapped out to SSE without making any changes.

The mindset is to remove as much messaging-specific stuff as possible from clients (protocol handling, data resolution, etc), and just focus on doing stuff with the data. In a React sense, the component doesn't care about the sK protocol or update format or anything. It just renders a portion of the full model, and re-renders whenever it changes. Re-rendering the whole UI on every event is exactly the intention, and is specifically how things such as Redux and Cerebral (and React in general) work. The module above uses immutable js; it's very cheap to diff and track changes, and can be done in a super performant way without even needing a state management library (and will work just fine in Angular etc). I challenge your view that it's at all expensive! :)

Your example doesn't include retrieving initial data + metadata, applying events to that data, etc. Is it fair to say that https://github.com/neftaly/kumara/blob/master/src/index.js is a more realistic example, within 80% of the minimum LOC? Also, is it possible to receive metadata updates via ws (genuinely curious, I can't find anything about it in the documentation)? Another simplicity argument is that the sK message protocol is proprietary, whereas this protocol is literally just RFC7396. Also, RPC is arguably more complex by definition than REST.

It's probably worth noting the distinction between protocols (proprietary sK vs RFC7396) and transports (WS vs SSE).

[tkurki]

Huh, I think we are after exactly the same thing:

The mindset is to remove as much messaging-specific stuff as possible from clients (protocol handling, data resolution, etc), and just focus on doing stuff with the data.

Our approaches are just a bit different, at least mine colored by stuff I'm familiar with. While a bit old fashioned and not idiomatic React https://github.com/SignalK/instrumentpanel/blob/master/lib/ui/widgets/digitalposition.js#L13-L17 shows an example of what I'm thinking of: all a component needs to do is subscribe by SK path + source and then update the UI on incoming data.

Re-rendering the whole UI on every event is exactly the intention, and is specifically how things such as Redux and Cerebral (and React in general) work.

While in theory a good idea witness the amount of guides on how to optimise by avoiding total rerenders. A major part of Redux connect's logic underneath the hood does that for you. Then again Immutable.js pretty much solves that problem or at least makes diffing cheap enough.

But I don't think current front end frameworks or how you code the UI should be the guide here. I believe I mislead us in my earlier message.

You have a point about retrieving the initial state with metadata and allowing also metadata updates.

So far I have not found initial state to be a problem: most data will either update frequently or be hopelessly out of date and not worth displaying. Maybe someone else can comment on this? I am sure there is data that is not updated frequently but worth displaying.

is it possible to receive metadata updates via ws

Ws carries only data, not metadata.

My main concern with your idea is that it ties us very tightly to the hierarchical, single dimensioned full model, which I think is keeping us from moving forward with a more rich data model and a api/protocol to go with it. Personally I would like to find the time to figure out how GraphQL and GQL Subscriptions would solve our problems of having multiple ways at looking at things: is the alternator part of the electrical subsystem or engine? Why not both?

Another point is SK specification surface area. While a new way to do stuff might be marginally better and solve some existing problems it is a new, alternate way to do things. Should clients be rewritten to support this? What of old servers that only support ws? When writing a new server / SK API to an existing system should you support both ws and this? More apis, more bugs and idiosyncranies.

But I am sure you've already thought about all this. As I said the way to move this forward would be to write an implementation and see how things look. Signal K v1 will most probably be just ws, but I won't make any guesses this way or that for the future at this stage.

[neftaly]

I've made a plugin eventsource-sk. Wanted to name it sk-sse, but plugin load order seems to be by name, so it was injecting after the built-in rest plugin.

If there is any possibility of data that never change (such as vessel ID or metadata), you'll still need to get the initial state on connect.

The hierarchical model has a lot going for it; namely, simplicity. Being able to get a DIY client up-and-running with a tiny amount of work (not even reading the docs) is a big deal.

Stuff like GraphQL/Falcor is cool, and I wouldn't discount it, but at some point you'll still need to define what data exists where. If I get what you're thinking right, you're trying to describe the difference between a directed acyclic graph (DAG) and a cyclic graph? You could actually implement this on top of any sK v0.x API by using LD-JSON without making any breaking changes (so on this, the REST endpoint, and a ws endpoint, etc).

SSE can actually be done in an easily backward compatible manner - the connection is closed early (non-SSE), just re-connect in 10 seconds (poll). Whether you'd support this and ws in v2 is not relevant, as you could easily make the same argument about ws/REST and GQL etc (but that's what semver is for).

[tkurki]

Rest is not a plugin and loaded separately - did the naming really make a difference? Bumped against this with the charts plugin just now - another case of server core functionality vs a plugin.

[neftaly]

The behaviour might have changed when I moved it out of the interfaces folder into its own, I didn't test extensively sorry.

On 24/01/2018 7:00 PM, "Teppo Kurki" notifications@github.com wrote:

Rest is not a plugin and loaded separately - did the naming really make a difference? Bumped against this with the charts plugin just now - another case of server core functionality vs a plugin.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/SignalK/specification/pull/402#issuecomment-360029680, or mute the thread https://github.com/notifications/unsubscribe-auth/ABGDnpsghXIxq-G2Z7RJtxWkjhfDdQQmks5tNscUgaJpZM4Qv8ZA .

[tkurki]

I don’t think being able to bootstrap development has much to do with hierarchy - if the data model were flat you could still eyeball the response and pick the data you need. And I am talking about hierarchy within a single vessel. Being able to retrieve data per vessel by url clearly has value.

But I’ll try to cook up a separate issue and share my thoughts on the sk data model.

[fabdrol]

@neftaly @tkurki bump. This PR hasn't seen any TLC since January, is it still relevant/should the discussion be continued?

[tkurki]

I have not seen any discussion other than this issue & the plugin about this. Closing for inactivity.

[neftaly]

It's a shame to see this die with no traction whatsoever :(

[fabdrol]

@neftaly it doesn't need to die, it's just that we feel that a PR is not the right place to discuss this - especially if the discussion doesn't seem to go anywhere.

Therefore, what I would suggest, is this: create an issue referencing this closed PR and continue the discussion there. You could kick-start this again by summarising the discussion in this PR and putting in your thoughts as to how to move on with this. Hopefully, that'll restore traction and if the discussion goes somewhere, we can re-open this PR or you (or someone else) can submit a new PR.