Open universal json format

[tkurki]

Navgauge uses its own 'universal' format for sending basic navigation data to the browser. 'universal' and 'basic' mean that data from different sources (NMEA 0183, N2K, different sources within a N2K network) are combined on the server and sent in the same format and same fields so that the UI doesn't need to know anything about where the data is coming from. The combining is done in a boat specific configuration file in Navgauge.

This 'universal' way is something I just came up with when making the UI work universally with the data sets from three different boats that I had available. For example units are a mixture of metric & marine unit.

Creating a more universal and standard format would support sharing innovation & functionality between different efforts.

[ktuukkan]

@timmathews those look pretty straight-forward to add so no problem, will add them in my backlog. @canboat, @rob42 I like the hierarchical model too and REST was also my first thought, fits nicely in that. Also +1 for SI units as this stuff may also be running in trains, planes and automobiles. Seeing the other's data would be really cool feature and in fact I've been thinking about starting new project around that idea, but haven't had time to get it going. Made a quick proto for Google Latitude -like server, though.

[timmathews]

If we think 'distributed' from the start then when two suitably equipped boats come within wireless range, or stream data via mobile internet, then they can connect and merge their trees AIS style, so you can see their data too!

If we can get ActiveCaptain involved, we could automate route sharing (and have a nice prebuilt "cloud" to store all of our collected data). Social boating?

I think the next step is then to collect all the current keys, which we can extract from NMEA0183, and NMEA2000. Plus the ones we know from common sense. Then we arbitrarily assign to collections and sub-collections. Fairly soon we will have a tree, which we will probably rehash a couple of times.

We could start by mirroring the official n2k categories from their online PGN list. Quantity in parens.

• AIS (21)
• Environmental (8)
• General and/or Mandatory (15) - Break these up. Mandatory stuff is mostly housekeeping.
• Navigation (43)
• Other (0) - So why is this a category?
• Power (13)
• Propulsion (7)
• Steering (6)

I wouldn't get too carried away with a huge tree, let's try to keep it to 3 or 4 levels. After all, you Java geeks might not mind model.vessel[123456789].systems.galley.toaster[0].temperature, but to a Unix guy like me that's a lot of typing.

Otherwise, I'm totally on board with this plan. I will work on putting together a cross reference of PGNs and fields to 0183 sentences and fields this weekend and then we can start categorizing. I'll also try to include SeaTalk.

[timmathews]

I pushed a commit to the argo metadata API to support filtering by category. It also adds a 'Category' field to the PGN object.

So http://pyxis.openseasproject.org/api/v1/messages?category=entertainment returns a list of known "Entertainment" PGNs. This isn't a NMEA category, and they are all manufacturer-specific, but it was an obvious enough category.

There are a lot of manufacturer-specific PGNs that are not currently categorized. For instance http://pyxis.openseasproject.org/api/v1/messages?category=airmar returns a list of uncategorized Airmar PGNs. I also broke up general and mandatory, but I am not sure I got the categorization correct for all of these.

[rob42]

@canboat - yes MMSI would work if you have one - in NZ they are MMSI's are not mandatory - but a similar concept would work. Which brings up DSC - we should store data for DSC, and also a branch for trip reports - just record persons-on-board, destination, ETA, etc and a background send/maintain updates is done via mobile internet.

Also REST interface - I just finished a commercial project using MongoDb. It takes JSON natively, and exposes data by REST by default. You just mongo.save(tree)', and it allows full index/search/query/update of any part of the structure. So that would solve history if we stored the tree every so often. Its very fast, so getting wind history etc would be easy. There are probably other NoSqls that suit too.

@timmathews - that category list is a good start - '...to a Unix guy like me that's a lot of typing...." BTW I have lots of spare keystrokes here - you are welcome to some if you dont have enough :-)

Entertainment! - also Charts, Cruising notes, Routes - this model concept opens a whole raft of progress in boats systems! Especially when you can merge with another boat.

[rob42]

@tkurki My chart applet (like yours) is based on Leaflet, using TMS. This is also used by Openstreetmap, openseamap, and other common map providers. TMS uses a tile pyramid with globally consistent and unique value for every tile at every zoom. So if we record the tile stack root url in the model, it would be fairly simple to swap charts or part there-of by downloading the tiles of interest from the root url. (Maybe have a smarter service that zipped the branch and sent the zip)

Given that at least the US, NZ, and Europe (openseamaps) are pretty well covered by licence free charts that would allow a global server (openseamaps?) to serve charts on demand, and cache locally on the boat.

[askpatrickw]

There's a CAN Cape for the beaglebone black worth checking out here: http://www.towertech.it/en/products/hardware/tt3201-can-cape/

[tkurki]

@rob42: The chart Navgauge uses is official and free for use, but not really usable for navigation, because the symbols are weird. This is the only decent free chart source for Finnish waters that I am aware of. I added it to make sense of where the boat in playback mode is. I probably won't use it for real as it is online, but I did try installing Mapproxy for WMS caching, but ran out of interest/time with the seeding configuration.

Like I said, charts are what this is about for me but the data.

@peaboy: If there had been a practical, cost efficient solution for CANbus on Raspberry last year I wouldn't have bought NGT-1, as the price is pretty steep. I actually have one PICAN CANbus board for Raspberry Pi sitting unused in the drawer. Got as far as a booting OS with custom kernel for it, but ran out of steam and went for NGT-1, which looked like a lot less hassle. If somebody has interest for the PICAN board let me know.

[tkurki]

I read over the previous stuff about the data model and started to write another comment, but then I realised we can all evolve the spec on wiki, so I put up a page right here: https://github.com/tkurki/navgauge/wiki/Open-Universal-Rest-API-and-Json-Data-Model

@timmathews decided to go for Github, since I am more used to Github and since it is Git underneath you get the potential benefits of offline editing etc. I was thinking about setting up a separate Github repo for this, since this is way beyond Navgauge and really a community effort.

Anyway, feel free to change anything. Maybe we can continue discussion here and accumulate the real stuff in the wiki?

I put model and data after MMSI in the uri, added the more specific uri structure and the notion of 'default'/'best' value and tried to come up with a single example showing data from different sources (2 n2k buses plus nmea from a standalone autopilot).

Thinking about this in terms of Navgauge I see the UI elements registering listeners for certain data items and then reacting as data becomes available. A concrete example would be the compass rose of the multigauge: subscribe to navigation.heading and as updates are received the compass rose turns. Btw this is fully reactive, so there may not be a hierarchical model representation of the data at all on the client side. How data from the different sensors/sources is combined to create a heading value does not really concern the client.

[keesverruijt]

Hi all, I merged @tkurki and @timmathews pull requests, but I (hopefully) improved on the CamelCase changes. First of all, nothing is changed if you set just -json, and there are new -json -camel and -json -upper-camel options. This makes the code backwards compatible but still allows us to experiment with the new output. Furthermore, instead of setting 'description' within the JSON array per line it generates an CamelCase":{....} record per PGN line. This is still not strictly conforming JSON, as every line should be separated by a ',' but like before I leave that up to n2kd and other servers. Let me know if you see any further improvements that can be made, as I haven't tested this in the field yet.

[keesverruijt]

@tkurki I don't think that making a distinction between uri and uri/ is an allowed distinction, both should give you the default document according to the HTTP specs. I think we will be forced to either come up with paths (e.g. .../all) or parameters (?all=1).

[rob42]

Added a draft model to think about https://github.com/tkurki/navgauge/wiki/Draft-Universal-Data-Model

Also I was looking into nosql dbs, and the easiest way to spec a query protocol is use one of their methods. CouchDb has a nice REST interface, but the overhead of HTTP on every save/get. Mongo has a rest interface, but also API, but the query language is harder, a javascripty like method call.

[tkurki]

@canboat /foo and /foo/ are definitely different resources, it is just convention that foo is often redirected to foo/ and interpreted as the same url.

See for example http://en.wikipedia.org/wiki/URL_normalization ("may result in equivalent URLs, but are not guaranteed to by the standards") and http://tools.ietf.org/html/rfc3986#section-6.2.4 .

[tkurki]

@canboat I think that for a streaming model a complete json structure per line without the trailing comma is the way to go. A trailing comma and { in the beginning of the stream would indicate 'wait, there is more to come, it is not complete yet'. As I understand it the main difference between raw sockets and websockets is that with websockets you're guaranteed to receive chunks, eg. complete entities sent by the sender. With raw sockets you need to do the parsing yourself,eg. you need to know the structure of the data.

For example Navgauge parses analyzer output and assumes that the input is message per line. Navgauge server to client uses websockets with a Javascript library, where the client is guaranteed to receive full json objects with no special attention given to delimiters.

[keesverruijt]

Nevertheless, we found many issues with this at work when customers were accessing sites through proxies.

I also think it violates the robustness principle "Be conservative in what you do, be liberal in what you accept from others” (http://en.wikipedia.org/wiki/Robustness_principle)

On 23 Mar 2014, at 09:57, tkurki notifications@github.com wrote:

@canboat /foo and /foo/ are definitely different resources, it is just convention that foo is often redirected to foo/ and interpreted as the same url.

See for example http://en.wikipedia.org/wiki/URL_normalization ("may result in equivalent URLs, but are not guaranteed to by the standards") and http://tools.ietf.org/html/rfc3986#section-6.2.4 .

[tkurki]

@canboat: Been there, had the problems. IMHO the real reason for the problems is the too loose convention. Here my thoughts were that 'for once we could use the distinction between a single resource and hierarchy level for something meaningful'. But the point is really minor, definitely not something I insist.

On the other hand Navgauge has now a tiny beginning of support for the new hierarchical API: http://navgauge.herokuapp.com/api/0.1/vessels

Furthermore the websocket interface can now provide heading values in the 'new' format and I changed the multigauge compass dial to use this format. See http://navgauge.herokuapp.com/treedata.html

[timmathews]

So, I'm still working on that n2k to n0183 cross reference. This turned out to be more time consuming than I anticipated.

Is there a format that would be most useful for everyone?

[rob42]

xml? - then we could publish it or transform it to other formats later? But the same probably goes for csv too :-)

[tkurki]

Xml or json. I find myself always turning to to json when previously I would have used xml. I assume you will need more structure than csv can provide.

Could you share yoru work in progress? I need to add N2k to NMEA mapping for my tridata stuff so I'm wondering if I can help and/or benefit. My goal is to have all data available for iNavX/iRegatta via NMEA on udp.

[tkurki]

@ktuukkan are you aware of json implementations for NMEA data? There are several nmea parser implementations for Node, I picked https://github.com/jamesp/node-nmea even if it is not very complete.

[ktuukkan]

@tkurki, @rob42 I would also go with JSON whenever possible. I believe it's also easieast and most efficient to produce and consume it in low-end devices.

@tkurki I think node-nmea is the only one I've seen so far, but on the other hand I haven't searched internets too much for it.

[rob42]

Ahh..I only meant XML as a format for recording the NMEA2000 / NMEA0183 mapping info - so we could use xslt to transform it to webpages/docs/etc. Definitely json for transmitting data and metadata between devices, its much more efficient and natural for the web.

[keesverruijt]

XML does has it advantages, as @rob42 says.

On 26 Mar 2014, at 08:47, rob42 notifications@github.com wrote:

Ahh..I only meant XML as a format for recording the NMEA2000 / NMEA0183 mapping info - so we could use xslt to transform it to webpages/docs/etc. Definitely json for transmitting data and metadata between devices, its much more efficient and natural for the web.

— Reply to this email directly or view it on GitHub.

[tkurki]

Yep, in agreement on all points.

For everybody: canboat's Nmea 2000 structure data is now available as XML and JSON: https://github.com/canboat/canboat/blob/master/analyzer/pgns.json https://github.com/canboat/canboat/blob/master/analyzer/pgns.xml

The master of this info is in the C code and xml and json can be generated with make json.

[keesverruijt]

On 26 Mar 2014, at 09:48, tkurki notifications@github.com wrote:

For everybody: canboat's Nmea 2000 structure data is now available as XML and JSON:

https://github.com/canboat/canboat/blob/master/analyzer/pgns.json https://github.com/canboat/canboat/blob/master/analyzer/pgns.xml

The master of this info is in the C code and xml and json can be generated with make json

The json XSLT thanks to @tkurki!

I can also change the XML output if needed, the current format is one that was preferred by OpenSkipper, but that project seems to be going nowhere.

Even better would be if I found the time to create a XSLT that generates the C code back from the XML, and then we can have the XML leading. It would also be much better if the various lookups were normalised so that they are in the XML only once. My fault for starting on a 50 MHz ARM processor with 32 MB RAM.

[timmathews]

@tkurki Right now, I have everything in SQL. I was thinking of putting a web frontend on it to make community contributions easier for all of the PGNs which we haven't seen data for. It wouldn't be too much work to then add a web service which could generate an XML, JSON, C, etc. representation of the data. I haven't finished inputting all of the NMEA 0183 stuff, but when I do I will post both the n2k and n0183 field lists on the wiki here and we can work on the mapping.

I'm thinking that document will look something like this:

Description	Units	n2k PGN	n2k Field	n0183 Sentence	n0183 Field
Wind Speed	m/s	130306	2	MWV	3 + 4
Wind Speed	knots	130306	2	VWR	3 + 4
Wind Speed	m/s			VWR	5 + 6
Wind Speed	kph			VWR	7 + 8
Wind Angle	degrees	130306	3	MWV	1
Wind Angle	degrees	130306	3	VWR	1 + 2
Wind Reference		130306	4	MWV	2

It will be pretty big, at least 1800 rows.

@canboat A C generating XSLT would be wonderful. I think it would also be a good idea to break out the n2k parsing code as a library.

[rob42]

@timmathews - needs extra fields for the universal json model var and type:

vessels[n].environmental.apparentWindSpeed float

[tkurki]

To illustrate the point about being able to connect from one piece of the mesh to another I added the capability to connect to different "Boats" to Navgauge's gauges demo page. It's live on Heroku: http://navgauge.herokuapp.com/gauges.html?

Initially it is not connected to any server and the vessel list is empty. Upon loading it fetches the vessel list from the server and populates the menu. When the use selects a vessel the page connects to the url specified in the vessels response. 'Freya' feed has just compass readings, Cassiopeia and Plaka all three readings (underneath all three use the same captured data, as I haven't implemented switching vessels on the server side yet).

As a placeholder for the interoperability feature I added a text field and a button for connecting to an arbitrary server. So if one of you guys can put up a server that talks something that is in line with what we've talked about I will add the functionality and try how things work out.

Incidentally I realised that I'm using a real time communications library called Primus, which is a wrapper on top of Websockets. This means that you can not connect to Navgauge server via Websocket as it is now, only with Primus on Javascript, but adding raw websocket support is very little work.

This raises an interesting point though. Either the vessels or the model api call should be able to list methods that the client can use to connect to the vessel's data stream. This should be extensible, as the communication methods will no doubt evolve. Url patterns aren't quite enough, as Primus url format is just http.

This is how it currently works: http://navgauge.herokuapp.com/api/0.1/vessels

[
  {
    "name": "Cassiopeia",
    "modelHref": "../vessel/byName/cassiopeia/model",
    "primusHref": "/?boat=cassiopeia&treedata=true&gaugedata=true"
  },
  {
    "name": "Freya",
    "modelHref": "../vessel/byName/freya/model",
    "primusHref": "/?boat=freya&treedata=true"
  },
  {
    "name": "Plaka",
    "model": "../vessel/byName/plaka/model",
    "primusHref": "/?boat=plaka&treedata=true&gaugedata=true"
  }
]

http://navgauge.herokuapp.com/api/0.1/vessel/byName/cassiopeia/model (not yet used in any way)

{
  "dataItems": [
    "navigation.headingNorth"
  ]
}

and then streaming messages in the format

{"id":"navigation.headingNorth","value":164,"source":{"type":"nmea","id":"Seiwa","timestamp":"2014-03-26T19:20:41.554Z","sentence":"RMC"}}

[tkurki]

@rob42 I didn't quite understand the need for float, json has just numbers.

@timmathews Hmm now I understand your csv comment. I'd still prefer a version controlled file, but once you implement exporting I can do my own version controlling ;-) Btw is there a specific need that you need this information for or is this for reference?

[timmathews]

As a placeholder for the interoperability feature I added a text field and a button for connecting to an arbitrary server. So if one of you guys can put up a server that talks something that is in line with what we've talked about I will add the functionality and try how things work out.

I think I can get argo to that point quickly. I have live wind and Fusion entertainment data available right now for testing, but if I stop by the boat, I can add GPS, depth, speed, water temperature, and whatever a Garmin chart-plotter puts out (route data?). I have a 9DOF Razor IMU from SparkFun which I should be able to generate data from as well.

Incidentally I realised that I'm using a real time communications library called Primus, which is a wrapper on top of Websockets. This means that you can not connect to Navgauge server via Websocket as it is now, only with Primus on Javascript, but adding raw websocket support is very little work.

Which plugin(s) are you using with Primus on the client side? I know I should just look at the code and answer this myself.

Btw is there a specific need that you need this information for or is this for reference?

The main reason is to have a standard cross reference that we're all using. Also, I want to collect as much information as I can about NMEA 0183, 2000, and SeaTalk in one place.

[rob42]

@tkurki - float - oh yeah, forgot that, but when java deserializes json 1.0 = float(double), 1 = integer, so that could cause me some complications. I guess I am saying all numbers should have at least 1 decimal, even if its zero.

Ive been working on actually using the draft model, and found there will be some misc changes.

[rob42]

@tkurki - played with your gauges demo - really cool, and potentially very powerful if you merge it with @timmathews ideas of dashboard on demand, eg just point to any part of the boats model to see the relevant instruments..nav..engine..music..

[tkurki]

@timmathews I just went with the default ws transport and Primus provides a prepackaged client library. Btw just found out that Heroku doesn't support custom ports and binding Primus and plain ws to the same port doesn't work. I need to set up a separate host or some other solution to support plain ws.

[timmathews]

@tkurki I don't really understand why you would need Primus and WebSocket server-side, but maybe I'm misunderstanding what Primus does. I thought it was just a wrapper around WebSocket, socket.io, sock.js, etc which provided a universal API. I didn't think it actually changed the protocol in any way.

I can still do this

var conn = new WebSocket("ws://navgauge.herokuapp.com/primus")
conn.onmessage = function(evt) {
  console.log(JSON.parse(evt.data));
}

and see JSON data come streaming in. What I did notice is that you can't use the client-side Primus object to connect to a non-Primus-using server (as it expects to connect to /primus).

That said, if you actually do need both, what I do for argo right now might work for you on Heroku. Since argo doesn't actually serve HTML, it sits behind Apache (which does serve the pyxis HTML) and I use mod_proxy_wstunnel to proxy WebSocket requests to /ws/ out to argo running on another port (8081 by default) internally. You could do something similar with nginx or varnish if you're not an Apache fan. That should let you keep everything on a single Heroku instance.

Here's the VirtualHost entry for this from my Apache config

<VirtualHost *:80>
 ServerName pyxis.openseasproject.org
 DocumentRoot /var/www/openseas/pyxis-web
 ProxyPass /ws/ ws://localhost:8081/ws/ retry=10 timeout=5
 ProxyPass /api/ http://localhost:8082/api/ retry=10 timeout=5
</VirtualHost>

PS mod_proxy_wstunnel was introduced in Apache 2.4.x, so if you're using 2.2.x like I am, you'll need to patch it.

[rob42]

@tkurki - I like the addition of source{}, but I dont fully understand why you use

{   
    "id":"navigation.headingNorth",
    "value":164,
    "source":{
        "type":"nmea",
        "id":"Seiwa",
        "timestamp":"2014-03-26T19:20:41.554Z",
        "sentence":"RMC"
        }
}

rather than

{
    "navigation": {
        "headingNorth": {
            "value": 164,
            "source": {
                "type": "nmea",
                "id": "Seiwa",
                "timestamp": "2014-03-26T19:20:41.554Z",
                "sentence": "RMC"
            }
        }
    }
}

The second version creates an object 'navigation' containing object 'headingNorth' etc. So it allows very clean js code, especially if its illegal to send headingNorth without a valid value.

if(navigation){
    if(navigation.headingNorth){
       //update whatever...
          compass.update(navigation.headingNorth);
    }
   ...other nav items
}

Saves the problems of checking the stream is not null, id!=null or zero-length, and various other problems. But I may be missing something ?

Robert

[timmathews]

I've started the mapping. There are two sentences mapped to their equivalent PGNs, but I've left the data model column blank for now.

Cross Reference NMEA 0183 Sentences NMEA 2000 PGNs

[tkurki]

@rob42 I think about it the other way around: gauges should be independent, eg. they don't need to know about each other or there should not be a single controller/dispatcher that controls what needs to be done when some piece of data has a new value.

In Navigauge's sailgauge the compass dial starts listening to a data stream identified by navigation.headingNorth. If there are several gauges showing heading they do the same or if some are dynamically added or deleted they just start/stop listening.

Incoming events are dispatched to these streams based on the id. There can also be a stream where you get all the messages. Dispatch is in swipe.html, action in sailgauge.js.

With the id=a.b.c structure the streams can be looked up like stream['a.b.c'] and when new message types appear the only place that needs to change is the key in the single gauge. With the hierarchical structure you need to add new if branches to the dispatcher.

IMHO this is more decoupled and dynamic than if-style dispatching.

Having said I confess that the gauges.html is not reactive, but the dispatch mechanism is a single switch statement and key is not part of the gauge. Gauges.html is a simple demo, whereas Sailgauge.js is a product of first having everything in a single file and then breaking it up into decoupled compontents.

The two approaches are pretty interchangeable but for one thing: the structured, hierarchical method can contain several pieces of data in one message and composite messages, even one message containing 'everything' can be constructed.

So I am not rigidly set on the id='a.b.c', but am looking for more real world cases to see how things work.

[tkurki]

@timmathews My mistake, I didn't realise that primus ws is available under /primus path. Thanks for pointing it out, I can now remove the extra code.

[keesverruijt]

I am not sure this direction of direct JS objects is going to be useful in practice. Take this exact example of heading. I’ve got two electronic compasses and two wind sensors.

I don’t think any non-trivial boat is going to be able to do without a mapping layer, and if we do it it needs to be on the server, or at least persisted there, since I don’t want to do this for every client: I want all tablets etc to show the same information.

Here is (the relevant part of) my current mapping table:

    { "Water Depth":   { "id": 0, "pgn": 128267, "field": "Depth", "src": 35, "units": "m", "offset": "0.5" }
    , "Keel Depth":    { "id": 28, "pgn": 128267, "field": "Depth", "src": 5, "units": "m" }
    , "Pitch":         { "id": 3, "pgn": 127257, "field": "Pitch", "src": "PB200", "units": "°", "round": 1 }
    , "Roll":          { "id": 4, "pgn": 127257, "field": "Roll", "src": "PB200", "units": "°", "round": 1 }
    , "Log":           { "id": 6, "pgn": 128275, "field": "Log", "src": "DST200", "units": "nm", "factor": "0.0005399", "round": 1 }
    , "Air Temp":      { "id": 7, "pgn": 130311, "field": "Temperature", "src": "PB200_Outside", "units": "° C", "round": 1 }
    , "Air Pressure":  { "id": 8, "pgn": 130311, "field": "Atmospheric Pressure", "src": "PB200_Outside", "units": "hPa" }
    , "Latitude":      { "id": 9, "pgn": 129025, "field": "Latitude", "src": "PB200", "units": "Deg" }
    , "Longitude":     { "id": 10, "pgn": 129025, "field": "Longitude", "src": "PB200", "units": "Deg" }
    , "Date":          { "id": 11, "pgn": 126992, "field": "Date", "src": "PB200", "log": 0 }
    , "Time":          { "id": 12, "pgn": 126992, "field": "Time", "src": "PB200", "log": 0 }
    , "Batt Charge":   { "id": 13, "pgn": 127506, "field": "State of Charge", "src": "MasterBus_0", "units": "%" , "logMin": "1", "alarmTest" : "< 70", "alarmDo": "powerdown" }
    , "Batt Current":  { "id": 14, "pgn": 127508, "field": "Current", "src": "MasterBus_0", "units": "A" }
    , "Batt Temp":     { "id": 15, "pgn": 127508, "field": "Temperature", "src": "MasterBus_0", "units": "&deg; C" }
    , "Batt Voltage":  { "id": 16, "pgn": 127508, "field": "Voltage", "src": "MasterBus_0", "units": "V", "logMin": "0.3" }
    , "NMEA Voltage":  { "id": 17, "pgn": 65410, "field": "Supply Voltage", "src": "DST200", "units": "V", "logMin": "0.3" }
    , "SOG":           { "id": 20, "pgn": 129026, "field": "SOG", "src": "PB200", "units": "knt", "factor": "1.944", "round": "1", "alarmTest": "> 2.0", "alarmMinInterval": 86400, "logMin": "0.3" }
    , "SOG (AIS)":     { "id": 26, "pgn": 129026, "field": "SOG", "src": "43", "units": "knt", "factor": "1.944", "round": "1", "logMin": "0.3" }
    , "Speed":         { "id": 21, "pgn": 128259, "field": "Speed Water Referenced", "src": "DST200", "units": "knt", "factor": "2.1", "round": "1", "logMin": "0.3" }
    , "True Wind Dir": { "id": 1, "pgn": 130306, "reference": "True (referenced to North)", "field": "Wind Angle", "src": "PB200_True", "units": "&deg;", "round": 1 }
    , "True Wind Spd": { "id": 5, "pgn": 130306, "reference": "True (referenced to North)", "field": "Wind Speed", "src": "PB200_True", "units": "knt", "factor": "1.944", "round": 1  }
    , "App Wind Spd":  { "id": 22, "pgn": 130306, "reference": "Apparent", "field": "Wind Speed", "src": "PB200_Apparent", "units": "knt", "factor": "1.944", "round": 1  }
    , "App Wind Dir":  { "id": 23, "pgn": 130306, "reference": "Apparent", "field": "Wind Angle", "src": "PB200_Apparent", "units": "&deg;", "round": 1 }
    , "Mast Wind Spd": { "id": 24, "pgn": 130306, "reference": "Apparent", "field": "Wind Speed", "src": "508_Apparent", "units": "knt", "factor": "1.944", "round": 1  }
    , "Mast Wind Dir": { "id": 25, "pgn": 130306, "reference": "Apparent", "field": "Wind Angle", "src": "508_Apparent", "units": "&deg;", "round": 1 }
    , "HDG (PB200)":   { "id": 2, "pgn": 127250, "field": "Heading", "src": "PB200_Magnetic", "units": "&deg;", "round": 1 }
    , "HDG (RC42)":    { "id": 27, "pgn": 127250, "field": "Heading", "src": "RC42_Magnetic", "units": "&deg;", "round": 1 }
    , "Sea Temp":      { "id": 29, "pgn": 130311, "field": "Temperature", "src": "5_Sea", "units": "&deg; C", "round": 1 }
    , "Engine Speed":  { "id": 30, "pgn": 127505, "field": "Capacity", "src": "MasterBus", "units": "RPM" }
    }

(Added code block for readability - tkurki / edit2 didn't work..)

Note that this mapping table also converts from sensor units to what I prefer on screen (nm/h or knt). The id’s need to be unique and persistent, as I use the id to save the data to history in RRD format.

On 28 Mar 2014, at 00:10, rob42 notifications@github.com wrote:

@tkurki - I like the addition of source{}, but I dont fully understand why you use

{
"id":"navigation.headingNorth", "value":164, "source":{ "type":"nmea", "id":"Seiwa", "timestamp":"2014-03-26T19:20:41.554Z", "sentence":"RMC" } } rather than

{ "navigation": { "headingNorth": { "value": 164, "source": { "type": "nmea", "id": "Seiwa", "timestamp": "2014-03-26T19:20:41.554Z", "sentence": "RMC" } } } } The second version creates an object 'navigation' containing object 'headingNorth' etc. So it allows very clean js code, especially if its illegal to send headingNorth without a valid value.

if(navigation){ if(navigation.headingNorth){ //update whatever... compass.update(navigation.headingNorth); } ...other nav items } Saves the problems of checking the stream is not null, id!=null or zero-length, and various other problems. But I may be missing something ?

Robert

— Reply to this email directly or view it on GitHub.

[rob42]

@tkurki - not sure i am on top of Bacon.Bus but the subscriber concept works for both formats. I showed a simple if statement above, but there is nothing to stop the whole message (or part) being pushed to a subscriber list (eg the gauge), and the if statement or filter occurring there. Thats roughly what I do in my current implementation.

What appeals is being able to send a snippet of the model from a sensor and simply map it against the master model (eg merge the branch and master), and likewise send out a filtered branch to whatever needs it. @canboat - not sure I follow you - by 'direct JS objects' do you mean

{
    "navigation": {
        "headingNorth": {
            "value": 164,
            "source": {
                "type": "nmea",
                "id": "Seiwa",
                "timestamp": "2014-03-26T19:20:41.554Z",
                "sentence": "RMC"
            }
        }
    }
}

I agree that they need to go via a server (which browser clients would anyway) where they can still be mapped as before. I imagine the messages would be stored as they arrive, and then merged into the universal model. That would still allow history to be recovered for any key ( eg navigation.headingNorth), and if you store 'source' you can filter by source too.

I recently spent some time using MongoDb (nosql) . It stores the json tree as a single 'document', but allows you to index or search by navigation.headingNorth.source.id, even if some records dont even have a 'source' !.

[rob42]

What attracts me to the universal model concept is its ability to solve a range of problems Ive had in freeboard. Basically freeboard currently works by routing messages to the browser immediately, and the browser then reacts (redraws) immediately. Ive found its quite possible, even with a laptop client, to receive messages faster than the browser can redraw, especially if you are also moving charts etc. and recalculating chart vectors.

Since javascript is single threaded that makes keeping up with lots of small packets over websockets complex. By keeping a model on server and browser, and sending steady periodic updates as sparse copies (changes only) of the model I could get optimal efficient replication to the browser. Then I could schedule redraws at a predictable and manageable rate using the local model data.

It also appealed for the ease of data replication between devices, and other boats - eg nav, comms, cruising notes, charts, etc. Plus at the server end I can run background processes to add calculated data, watch alarms etc, all of which is quite difficult at present.

This is why I favour sending the object tree, rather that the individual key/value messages. Its optimal to transfer and merge between devices, and efficient to access on the devices.

[timmathews]

I think the tree model is the right way to go as well. I'm still in favor of letting the client tell the server exactly what parts of the tree it needs, but this isn't strictly necessary and as @rob42 pointed out early on, may be too complex for simple servers.

@canboat brings up an interesting point viz. multiple sources for the same data or even the same data from the same devices under multiple formats (e.g. my wind device sends out environmental data under several different PGNs, but it's all the same stuff). I am curious how that mapping works in practice, I assume you do it based on the Model ID sent in the Product Info PGN? There really isn't anything like that for NMEA 0183, but I guess since each talker needs to have it's own input to the server, you could do it by serial port?

Do you have automatic failover or average the results? What is the benefit to having say two wind instruments or multiple compasses?

[timmathews]

The only thing I would change in @rob42's model would be to eliminate the source block and move some of that info to the metadata API:

{
    "navigation": {
        "headingNorth": {
            "value": 164,
            "source": "12314033",
            "timestamp": "2014-03-26T19:20:41.554Z",
        }
    }
}

All of the static info about the source would be available at /source/12314033 which would return something like this:

{
  "@href": "/device/seiwa-123456789A",
  "name": "Seiwa",
  "busType": "NMEA 0183",
  "sentence": "RMC",
  "sentences": [ "HDG", "HDM", "HDT", "RMC", "ROT", "TXT" ],
  "deviceType": "Compass",
  "serialNumber": "123456789A",
  "installDate": "2012-05-30",
  "location": "V-berth port locker",
  "power": "100mA"
}

The source identifier would specify a specific device/sentence or device/PGN combination and the @href field in the resultant JSON (could be rendered as a link in HTML) would take you to the info page of that specific device where you might have access to device configuration, see all of the data coming from that specific device, maybe have a link to the manual...

[tkurki]

@rob42 I urge you to look into either Bacon.js or RxJs. The problems you mentioned related to refresh rates are easily solvable with reactive, stream based approach. For example changing the update rate for Navgauge's wind marker is just one .throttle(milliseconds).

Combining data from different sources, for example true wind from apparent + boat speed, is in my opinion straightforward with a reactive, stream-based approach. When either source value changes you get a new reading. Adding more complex logic such as keeping a sliding time window of data for drawing a sparkline or ignoring out of range depth values are a breeze.

[tkurki]

@canboat Totally agree on the mapping layer. I think the whole json data model is really about what the target data model of the mapping layer is.

I have three mapping configurations, one for each boat I have data for. Data sources are different, but since I wanted to use the same UI for all of them I came up with an ad hoc, sloppy model just to get my own stuff to work together.

My next step is to get my mapping layer to output and UI to consume the stuff discussed here. If somebody else has a data feed to connect to, either running or code I can check out and have running locally, I'll start with adjusting the UI.

All of you: I would love to have a data set from your boat to try Navgauge against. Canboat raw/json or nmea, doesn't really matter. Separate nmea + canboat data is problematic if there is nothing to collate the feeds.

About the 'multiple compasses' use case: somewhere along the processing pipeline there needs to be logic related to what should be displayed by a 'primary compass' widget. My solution would be to put that logic to the mapping layer. Something along the line: default is compass A, but if no data is available use B. Again easy to capture in the stream configuration, which would in my case be Javascript running on the server. Of course you can have multiple compass widgets if you want to. This is actually the case that spurred my suggestion about the distinction between ...navigation/heading and ...navigation/heading/: with one you get the processed data, with the other the unprocessed.

[rob42]

@timmathews - I agree most of the source data should become meta-data, thats a good way to avoid constant repitition of static data.

@tkurki - (I will look into Bacon) The true wind is a good use case. Its derived from apparent wind, apparent dir, sog, and heading. Since each of them come in different messages, its necessary to store them, so they are all available whenever a recalc is needed. I see in your code its in awinfo, but I couldnt track that back to source. How do you store data in these cases?

Given that it does need to be stored, you end up with some sort of model. Not sure if your is on the browser or server.

In my case several other things occur. The chart shows vector lines for heading, waypoint, sog, and eventually wind. These are centered on the boat and change dynamically as data changes. So again whenever a datapoint that changes any of them occurs, the relevant data needs to be recalced, and several things redrawn.

Meanwhile back at the server the alarms need to check new values, and the autopilot needs to recalc if its following wind. Again a model is needed to store all the current and calculated vars on the server and changes to the model need to be propagated to the clients.

In freeboard the browser can also issue commands (eg autopilot adjust heading). So that modifies the server model, and must issue commands to the autopilot, plus update other clients so they show the new autopilot heading.

I currently use a streaming model (except in the arduino). It was fairly easy when it was simple, but its becoming increasingly complex as I add functionality.

If either model changes that should be propagated to all clients, as they may want to react. The autopilot algorithm actually runs on the Mega, so it constantly recalcs and updates data.

It always comes down to storing data on each device since updates are random and multiple data points are needed for calculations. You can see that a common model replicating (either partially or fully) becomes quite attractive.

[timmathews]

Speaking of calculating an accurate true wind vector, these are some good resources that go beyond the standard true wind calc.

• This diagram is a flowchart of how B&G H3000 systems operate
• Richard McCurdy's Masters Thesis - arguably the first modern paper on the subject of sailing instruments (1970)
• Arvel Gentry's Sailboat Performance Testing Techniques details what he built for Jim Kilroy's Kialoa IV (1981)
• and the Ockam Instruments blog is always a good source for interesting ways to analyze performance data.

PS. Does anyone want to write an HTML5 polar chart widget? Once we're logging all of this information, building performance polars should be easy.

[tkurki]

@rob42 That's the beauty of FRP with Bacon, you don't need worry about storing the values. Take a look at this simple example. You get a calculated value without really worrying about how to store the values and when side effects such logging or updatings something on the screen should be triggered.

You can try it out by copypasting it to browser's js console on some page that has Bacon loaded.

[rob42]

@tkurki - but how do I run it in java (or in C on an arduino)?

[tkurki]

@timmathews Navgauge has http://navgauge.herokuapp.com/swipe/polar , which plots your speed & twa in a polar plot with color indicating the wind speed. Feel free to use the code - legend and scale indication would be neat, hint hint! I've been thinking about extracting the widgets from Navgauge proper for easier sharing. Again this illustrates the need for common data model, because it expects data in Navgauge's ad hoc data model, coming in as messges via onData(message).

I also have the code for parsing and drawing csv polars in a local branch.

[tkurki]

@rob42 Sorry, I have no good solution for that. That's one benefit I get from Node, I can use the same code and abstractions on the server and client and move functionality between them.