search

tkurki / navgauge

Navigation gauges & display in your browser
MIT License
35 stars 6 forks source link

Open universal json format #4

Open tkurki opened 10 years ago

tkurki commented 10 years ago

Navgauge uses its own 'universal' format for sending basic navigation data to the browser. 'universal' and 'basic' mean that data from different sources (NMEA 0183, N2K, different sources within a N2K network) are combined on the server and sent in the same format and same fields so that the UI doesn't need to know anything about where the data is coming from. The combining is done in a boat specific configuration file in Navgauge.

This 'universal' way is something I just came up with when making the UI work universally with the data sets from three different boats that I had available. For example units are a mixture of metric & marine unit.

Creating a more universal and standard format would support sharing innovation & functionality between different efforts.

tkurki commented 10 years ago

I am able & willing to change the Navgauge server-to-UI json format, but would really appreciate input & insight from others.

Current format is pretty readable in the boat config files under https://github.com/tkurki/navgauge/tree/master/lib/boats

keesverruijt commented 10 years ago

Hi Tkurki,

A good idea to get a more formal and better JSON format amongst the nautical apps. I can adapt CANboat as needed.

timmathews commented 10 years ago

Hey all,

I too would like to work on formalizing the data format. Have a look at what I'm working on http://pyxis.openseasproject.org and https://github.com/timmathews/pyxis-design-documents. My goal with this is something akin to Maretron's N2KView.

The timing of this is rather fortuitous, as I have been thinking a bit about message specifications this weekend. At a high level, I don't want the client to have to know beforehand how data is formatted or what is available. Instead, the server should provide an API to query for this information. The only thing the client needs to know is how to actually make that query. So once the client makes a request to the server (standard HTTP GET request to a published URL) it will get back a list of all of the individual messages that the server knows how to send. Using this information, our user can build their display pages in the browser and save that configuration to the server. Then when one of those user created pages is displayed, another request is made via a WebSockets connection this time subscribing to the specific data that the client wants (essentially, whatever is needed to drive the gauges, charts, graphs, etc. displayed on the screen). We call this the 'control' channel. The server responds to that request with a URL to a WebSockets connection custom built for that client. The client connects to this new WebSockets URL (the 'data' channel) and begins displaying the data received.

I've pretty much worked all of this out and will be continuing to document it at http://openseasproject.org and on my GitHub page, but what I don't have yet is a good way to format the data going to the client. I know that I want to be able to break down the data packets which come in to the server (n2k, 0183, SeaTalk, whatever) and reconstitute them in a more friendly manner.

Right now, what I'm doing is using my Go port of CANboat to take data from an Actisense NGT-1, pack up the formatted PGNs (using the CANboat format) with MessagePack, send them via ZeroMQ to a very simple web server (again written in Go) which serves the data (this time JSON encoded) via WebSockets to whoever wants to connect. This is all kind of a hack, but it should be a lot better soon.

rob42 commented 10 years ago

Yes, Im all for a standard data format, and since the web and HTML5/websockets is my target I'm all for using JSON.

If you look at freeboard (https://github.com/rob42/freeboard-server/blob/master/src/main/java/nz/co/fortytwo/freeboard/server/util/Constants.java) you will see Ive used a 3-char key and associated value. I just create an array of them and convert it to JSON.

Its a basic format, and could be improved by sub-grouping by use (eg position, engine, etc) to more efficiently handle long arrays.

The protocol is also easily used by the arduino micro-controller as a non JSON serial stream. I just chop it [0-2][3-EOL] and I have key/value for minimal CPU. The consistent 3-char key and CR between each make it easy to send or process at the arduino end. Even on the RPi or BBB there is a minimum of spare CPU to parse data, so it needs to be easy to handle.

Also in my protocol if the label starts with # its a command, which I also required. Plus its still reasonably readable for dev and debug. Probably I should add some sort of message checksum too.

So I think the protocol should have:

At the client end I just pass the keys past the various gui objects and they use them if they are interested - that would work nicely for assembling custom displays. Maybe the display should also register to receive interesting keys?

Same with sending keys, a given device or gui component just issues a key=value whenever necessary, its passed to the webserver (via websockets) and routed accordingly to others.

Keep in mind there may be more than one 'server', we should think peer-2-peer.

I dont think there is much need to create a complex formal spec at this stage, just that we all work on a similar basis and co-ordinate our keys. I expect we will find new needs and opportunities, so things will change. And I like the idea (TimMatthews) that the device can be queried as to its capabilities - need a key for that.

Robert (freeboard)

rob42 commented 10 years ago

The key is inter-operability. While we remain unique and isolated our projects will never get critical mass ...we need to get OpenCPN into this too. Rob

timmathews commented 10 years ago

Totally agree we need to get OpenCPN onboard and also OpenSkipper if they're still actively developing (it's been a year since their last update).

There is another project that I've been watching at http://sailboatinstruments.blogspot.com which is pretty interesting. I'll reach out to him and see if he wants to be involved.

tkurki commented 10 years ago

Whoa, seems like there is some demand for this sort of thing. Making things interoperable is in my interest as well.

Some thoughts on the discussion so far:

What are we actually talking about when we want common json? CANboat and gpsd for example already produce json. In fact @canboat's comment about changing the json format sounded to me more like the tail wagging the dog - why not pick the N2K json format for everybody? Navgauge is not using it for the fact that half of my data is NMEA 0183. Furthermore when I got hold of datasets for more than one boat I realised that my first interest, a SailSteer clone gauge, needs some basic data and where they are coming from is different on every boat: Freya has several N2K sources for some of the data, Cassiopeia has both NMEA and N2K and Plaka just NMEA. All this lead me to create a basic stream of data that has just the needed fields and nothing extra.

Then again Navgauge server can serve also the raw data in json format - see http://navgauge.herokuapp.com/swipe/bus.

For N2K this is just CANboat format. For NMEA it is the dead simple format in node-nmea. Node-nmea suited my purposes most easily, so I just went with that. I need to add support for some more sentences to get my N2K tridata available as NMEA for iNavX and iRegatta.

For me the use case is really that if somebody comes up with a great display/gauge/overview I would like to use that on my boat with the minimum of hassle. Also the reverse is true: @rob42 said kind words about my multiuse gauge - if Freeboard would provide the same basic data in the same format over websocket he could just copypaste some Javascript & css and use the gauge on his system. Client side browser component infrastructure is developing quickly right now, which means that there is no one way to package a component, but I am confident something will emerge. With a little bit of extra work we could extract the gauge component and use it as part of Freeboard and Navgauge. Maybe somebody with more skills for graphics could make it prettier, now it's more functional the esthetic.

For this we need standard json format for the basic nav data or the component needs configuration options or intelligent fallback with defined precedence for the different formats.

I've also thought about providing a "list capabilities" type of call. BTW the data set is not static - if you had yesterday a certain PGN available it may not be there today. Or querying for capabilities right now doesn't give you the PGNs from the equipment that is not on.

As for the configurability evident in Pyxis plans I've decided not to go that way at all: so far Javascript/HTML/CSS is the way I configure my display. No doubt this will change, but I assume by just using javascript to instantiate more high level widgets and connect them to certain selected data from the server. Combining data items is imho pretty straightforward as well.

I am not sure about the "simple key=value structure, rather than NMEA0183 style complex combinations of values" - some items have structure and losing that structure might not be a good idea, for example yaw-pitch-roll and rates of turn for example. Also handling structure in json is dead simple.

"Human readable, Checksum field, fixed length short field labels" sound conflicting - the greatest joy of working with json as opposed to xml or some other format is that it really is human readable. I am definitely not fond of three letter abbreviations and it sounds like NMEA all over again. I plan to rely on tcp/ip instead of checking checksums and Mr Moore will take of the extra effort in parsing.

Keys on the level of "these data items I'm interested in" make sense, as in a subscription request. Something that I've run against is that the UI might be interested in a time series of some data - say STW and SOG (see me doing the three-letter-acronym thing ;-) or a single item from two different sensor - to create a plot. So having access to history data is something I'll be looking at in the client 2 server communications. I suppose my case is a bit different, since the UI code runs on the browser and the tablet or whatever might be switched off for periods and I would still like to get history graphs.

I have been thinking about in terms of client and server, not peer to peer, but can imagine scenarios for that. I suppose the json could be the same, but something other than websockets for reliable communications with queuing & retry.

tkurki commented 10 years ago

@timmathews I'm curious:

tkurki commented 10 years ago

What would be the use case for this for OpenCPN? As it is a single application with direct access to the different data streams I don't see the need, but is there something I'm missing?

tkurki commented 10 years ago

@timmathews Navgauge server has the possibility to subscribe to

tkurki commented 10 years ago

Does anybody have access to GoFree data in json format? I think we should consider being compatible with commercial stuff as well.

And btw I would be really interested in looking at other data captures. At least for me adding configurations for different boats changed the software quite a bit.

keesverruijt commented 10 years ago

Yes. Good questions :-)

On 17 Mar 2014, at 18:13, tkurki notifications@github.com wrote:

@timmathews I'm curious:

Why such a long chain: binary (N2K) to parsed (analyzer json output I assume) to binary (MessagePack) to broker to webserver to json via websockets? Why port CANboat and not use the original?

My own solution uses (as you guys probably know) uses actisense-serial | analyser -json | n2kd

which is usable directly by a HTML web client. My code predates web sockets. Web sockets is pretty cool tech, but I am glad I didn’t use it at the time as the spec kept changing.

I am still happy with json as a format, and I am also happy with choosing C for the analysis — it is not a trivial bit of code and the amount of processing power on my server is limited.

One other thing that we may want to discuss is whether we can’t come up with a common server side infrastructure. Even with this small group we already have Go, PHP, node.js and Java if my info is correct. Boy. Sure sounds like a lot of wasted effort. I am happy to move from PHP to something else as long as it is reasonably light weight and will run in a few megabytes.

One thing that kept me from publishing my PHP code is that (a) there is a lot in there that doesn’t work with other vessels, unless you want to invest in a Wago PLC running Linux. (b) You need to configure a lot in json config files, and I didn’t want to field so many questions.

What I would like is to reuse those beautiful gauges and SailSteer pages that you guys have been developing.

Kees

tkurki commented 10 years ago

Another idea might be to make a minimal generic translator from NMEA 0183 on a wifi to an open json format.

Use case is that some existing solution puts out NMEA over wifi and one would like to have a browser app connect to it. An easy to install translator that connects to the NMEA 0183 feed and provides json over http and/or websocket would make the json format more approachable.

tkurki commented 10 years ago

I went for Node because I wanted to know what the fuss was about. It turned out to be a really nice experience, @canboat I urge you to try it. And like it says on the Github page, it is CANboat and gpsd/gpspipe underneath on the data acquisition side.

I'm still not sure about parsing NMEA on the server. Should probably drop it in favor of the gpsd json format. Need NMEA encoding though for N2K to NMEA conversion.

nohal commented 10 years ago

For OpenCPN the benefit of this is not to have to reinvent the wheel - we must implement N2K (and SeaTalk) in some way. And even though I was looking at extending @canboat conversion to NMEA-0183 for now, implementing a format adopted by all of the other projects seems to be a better idea long term. But of course, we will stay away from WebSockets and alike as they bring nothing in our case...

tkurki commented 10 years ago

@canboat: speaking of peer to peer: If you replace the line that forks the actisense-serial | analyzer -json process with something that reads from your n2kd you should be able to use Navgauge with very little effort. See freya.js for an example on routing data from PGNs to the 'normalized' format that the UI expects.

Which all makes me think of running CANboat like you do and just have Navgauge server connect to it. Do you happen to have an init script for CANboat?

I think this reflects the main idea of this thread: there is already a de facto open source standard for N2K data and several for NMEA in json format. Where should translation from these formats to the representation usable by a UI component should take place and what format should communication between these components use?

keesverruijt commented 10 years ago

Some more comments (random ramblings):

I will try to run navgauge soon. (So many things to do...)

rob42 commented 10 years ago

The use case for peer2peer is multiple producers, multiple consumers.

This is a much better model than a single central server that does all. It means that actions on any one client or device is propagated across all the devices, and things still work when others break.

key=value ... yes we should use complex objects in the JSON. (I already use this for AIS data). By key/value pairs I meant that each value should be a separate key, not comma separated bundles like NMEA0183.

short keys...as a unix geek 3 letters is enough:-) but I guess that we can have normal names for keys, it does make it easy to read. Maybe we could have a 3 letter abbrev for a key for special cases?

Rob

keesverruijt commented 10 years ago

If you are going to do that you will probably want a sender (origin) in every message, and timestamps to check that the data is current, as well as something to have multiple instances of the same data type. Hey, we’re re-inventing N2K!

Maybe the 0183 JSON format should be more like a N2K message: - have an origin (= N2K src) and timestamp added by the device that translates it to JSON.

One solution is that we develop a 0183 -> N2K converter and do away with all 0183, and/or the other way around as well. Note that n2kd has the beginning of some N2K -> NMEA0183 conversion. It only needs more work :-)

Rob,

For the peer-to-peer part you may want to look into multicast. It dramatically simplifies network configuration (aka there isn’t any.) Unfortunately our browsers won’t be able to use multicast.

Kees

On 17 Mar 2014, at 20:33, rob42 notifications@github.com wrote

The use case for peer2peer is multiple producers, multiple consumers.

In freeboard I have an anemometer that reads wind data from the masthead unit. Via arduino that gives apparent wind. If it knew the boat speed it could calculate true wind. So it listens for SOG values, and when it has one it puts out true wind too. Remember this protocol will potentially be used by the sensors too (log, wind, depth, etc) plus it needs to control autopilots etc.

I have multiple clients on (potentially) multiple servers. They need to update each other. So I send the event stream (autopilot on/off etc) via JSON, or simple serial to the arduinos.

This is a much better model than a single central server that does all. It means that actions on any one client or device is propagated across all the devices, and things still work when others break.

key=value ... yes we should use complex objects in the JSON. (I already use this for AIS data). By key/value pairs I meant that each value should be a separate key, not comma separated bundles like NMEA0183.

short keys...as a unix geek 3 letters is enough:-) but I guess that we can have normal names for keys, it does make it easy to read. Maybe we could have a 3 letter abbrev for a key for special cases?

Nah, assembly is 3 letters. UNIX does it with two: cc, as, ld, nm, od, tr, cd, cp, rm, ln, ls, ...

timmathews commented 10 years ago

@tkurki So basically I reimplemented CANboat in Go (CANboatGo anyone?) in order to learn Go. I like the language and the concurrency patterns baked in make it a good fit for this sort of application. My convoluted data path from device to browser is a result of having a working application which could read N2k and spit out MsgPack and having a simple fake data generator that could speak JSON over WebSockets. It took about 10 lines of code to convert the data generator into a MsgPack/ZeroMQ to JSON/Websockets bridge, so that's what I did.

My two-cents on peer-to-peer: there's nothing in current browsers or in any W3C draft that I'm aware of that allows a browser to make any p2p connections, they're still strictly client-server. P2P between servers or devices which collect and propagate data is a good thing IMO though and should be investigated further. My recommendation is to use a message queuing system there (either brokered like RabbitMQ or brokerless like ZeroMQ) instead of multicast/anycast.

rob42 commented 10 years ago

Yeah re-inventing NMEA200 would be great - its problem is its closed proprietary nature. If we implemented it we would be forever caught up with conflicting PGM's (theirs/ours) and susceptible to them changing spec on us. So we should 'embrace and extend' it :-) A mapping of PGMs to our keys, and learning from their implementation is a good thing. Building bi-directional message converters (NMEA0183, NMEA2000, etc) to our JSON is the way to go.

Multi-cast isnt really necessary IMO although its quite interesting, I use Apache Camel to do smart processing/routing/conversion which is pretty easy and powerful. The browser will never be a full p2p client - its always going to have to be pushed by the webserver although websockets makes that pretty easy.

But the server implementation should be agnostic - we all have our preferences there. Also the transport, be it wifi, wired network, CANbus, or serial should not matter if we send the same JSON string.

Rob

timmathews commented 10 years ago

So peer pressure has made me clean up my server a bit and push it to GitHub. I called it argo. It's really pretty simple stuff, running it on a machine which has an NGT-1 connected, it will take n2k data from that and publish it to a WebSockets stream at /ws/v1/data. You can check it out now at http://pyxis.openseasproject.org/n2k.html

That's live data from a Maretron WSO100 in my basement with a fan blowing on it.

It needs to be cleaned up a bit and I need to do some housekeeping like adding a README and licensing and a nice thank you letter to Kees for doing all the hard work of figuring out just what the heck the NGT-1 was saying.

rob42 commented 10 years ago

@timmathews - nice - I can kind of see what its saying, but there is a lot of misc data in there. So in freeboard I would send something like 'WSA=20[CR]WDA=295[CR]' in the simple serial version, and in JSON it would ideally become something like: {"wind":{"apparentSpeed":20,"apparentDirection":295}}

Then any interested listener can look at a JSON message and if it has a 'wind' object, get the 'wind.apparentSpeed'

I do this for AIS data, using an array of aisinfo objects in an ais header object. each aisinfo is one vessel, and holds speed dir, lon. lat etc.

Rob

timmathews commented 10 years ago

@rob42 So yeah, there is a lot of info in each packet. The structure is something like this for wind (130306):

{"Header": {
  "Timestamp": "2014-03-17T17:34:16.978571418-04:00", // ISO-8601 Format timestamp
  "Priority": 2, // Message priority on the wire as defined by NMEA
  "Source": 128, // Device ID claimed by the WSO100 at bootup
  "Destination": 255, // Every listener on the bus
  "Pgn": 130306,  // PGN ID
  "Length": 8, // Number of bytes on the wire
  "Data":"7LcAzUz6//8=" // Raw data un-decoded
  },
  "Index": 159, // The index of the definition of the PGN in my map of PGN definitions
  "Data": {
    "0":236, // Field 0, in this case the SID
    "1":1.83, // Field 1, wind speed
    "2":112.64932725148729, // Field 2, wind direction
    "3":"Apparent" // Field 3, measurement type
  }
}

One of the features that I intended to implement originally was a message definition query API, so if you saw this data come in, you would make an HTTP GET request to /api/v1/pgn/:index: where :index: is the number from the packet above and you would get JSON back describing the data: field names, precision, range, units, etc. So that's why Index is there. As for the rest, well I already had that data from the n2k bus, so until I know what I need and what I don't, lets send it along.

tkurki commented 10 years ago

Your example shows what @canboat mentioned: it is often easier to send everything. Bandwidth on a local wifi hotspot is a nonissue and browsers running on tablets and phones have ample cpu for handling the data.

In the interest of a common json format this doesn't sound too promising though: you've come up with an alternate json structure to N2K data that is different from the one CANboat analyzer uses.

Getting data flowing and visible on the screen gives you the moment of satisfaction (at least it did for me!). When you start dealing with multiple sources and multiple formats it makes you start thinking about interoperability.

A concrete example would be that if I wanted to put your data as a sample data set for Navgauge I would have to deal with one more format. If you were using N2K json format I would just try freya.js and it would probably work with a parameter change or two. After that I would be a bit wiser, extract the common logic for pure N2K type boats and make it easier to adapt for such boats. And your data would be available to the wind history plot in Navgauge should you choose to try it out.

I don't see the need for separate metadata api, as I don't see why the format needs to be so verbose in other aspects but the data fields are just "0", "1" instead of properly named fields like they are in the CANboat format. Btw an array would more idiomatic for a list of fields.

As for the 'simple' json Rob mentioned: this is what Navgauge has for 'gaugeData' (as opposed to raw data):

{type:wind,reference:apparent,angle:54,speed:6.32}
{type:wind,reference:true boat,speed:5.172514148866067,angle:81.29917903663596} 
{type:depth,depth:13.38} 
{type:course,heading:167.6}
{type:speed,knots:5.7}

A separate 'type' field like this has works well for switch type handling logic, but Rob's proposed typed subelement would allow one top level message to contain several different types. For now I don't need timestamp - whatever is received is displayed immediately. Once I delve into fetching history from the server this will change.

Several inconsistencies that need to be fixed: boat speed item named 'knots', but wind speed is just speed without unit, depth depth..(sheesh who did this stuff..).

rob42 commented 10 years ago

Ah yes units of measure...lets just chose one consistent set for marine and convert when needed, then we dont have to mess with 'units' tags and extra code to deal with it.

So we could use knots for SOG and wind speeds, decimal degrees for angles, meters for distance and depth, UTC for time. Assume numbers may have decimals, eg floats

{
  timestamp: ,//possibly use this on any level
  source: ,//possibly use this on any level
  checksum:,
  wind:{ 
    apparent:{
      angle:54,
      speed:6.32
    },
    true: { //aackk - reserved word?
      speed:5.172514148866067,
      angle:81.29917903663596
    } 
    },
  depth:13.38,
  course:{
    heading:167.6,
    speed:5.7
    },
  position:{
    latitude:-41.5678,
    longitude: 173.23456
    },
  ais:{
    aisinfo:{
      name: ,
      latitude:,
      longitude:,
      heading:,
      speed:
      },
    aisinfo:{
      name: ,
      latitude:,
      longitude:,
      heading:,
      speed:
      },
    aisinfo:{
      name: ,
      latitude:,
      longitude:,
      heading:,
      speed:
    }
  }
}

Maybe even do meters/sec for speeds - would that be an advantage for complex analysis/comparison later?

Rob

keesverruijt commented 10 years ago

Hmm. “most convenient” for one is “hate” for others. I happen to agree with your “most convenient” units, but our set is just one of many possibilities.

There is one way out of that conundrum: use scientific units, e.g. SI.

The seven basic units in SI are: metre, kilogram, second, ampere, kelvin, mole, candela. Derived units are (limiting myself to what I think we’ll need here): radian, hertz, newton, pascal, joule, watt, coulomb, volt, farad, ohm.

If the format does not use units then the implied units should always be an SI unit. In other words: m/s, m, liters, degrees K, radians, radians/s. Floats is a good idea, it gets us out of the silly units that N2K uses for some datagram components. Time always expressed as UTC, and maybe always as seconds since 1970 (UNIX epoch) with an optional fractional part?

Oh, and we define that the locale is “POSIX", e.g. numbers are expressed as nnn.ddd, e.g. with a decimal point and not any other character, like my native comma (in Dutch we swap . and , in numbers compared to English.) And we define that the character set is always UTF-8.

@CANboat

On 18 Mar 2014, at 08:53, rob42 notifications@github.com wrote:

Ah yes units of measure...lets just chose one consistent set for marine and convert when needed, then we dont have to mess with 'units' tags and extra code to deal with it.

So we could use knots for SOG and wind speeds, decimal degrees for angles, meters for distance and depth, UTC for time. Assume numbers may have decimals, eg floats

timmathews commented 10 years ago

In the interest of a common json format this doesn't sound too promising though: you've come up with an alternate json structure to N2K data that is different from the one CANboat analyzer uses.

I'm not set on this format, in fact I pretty much like @canboat's format better. On the other hand, I think we should think long and hard before fixing our JSON representation to n2k PGN definitions. NMEA2000 is essentially an extension of SAE J1939 and the engineers who designed those standards had to make certain decisions informed by their transmission medium and available bandwidth. Our answers to those same questions will be different because we're working in a different medium with different rules.

I don't see the need for separate metadata api, as I don't see why the format needs to be so verbose in other aspects but the data fields are just "0", "1" instead of properly named fields like they are in the CANboat format.

Having a separate metadata API allows me to communicate a lot of static info to the client one time. Things like a nice display name for each field, an abbreviated version of the display name, the units that the field reports (or in the case of what lookup fields (enums) the possible values for the field), min and max values and what values are reported for errors. The field IDs in my current code are just numbers because I was being lazy at the time.

Hmm. “most convenient” for one is “hate” for others. I happen to agree with your “most convenient” units, but our set is just one of many possibilities.

As far as units go, consider this my vote for SI units, ISO-8601 timestamps, POSIX locale, and UTF-8.

timmathews commented 10 years ago

Here's the metadata API in action: http://pyxis.openseasproject.org/api/v1/messages for the whole list and http://pyxis.openseasproject.org/api/v1/messages/130310 for a specific PGN.

It doesn't work for all PGNs currently because of a limitation with Go's JSON encoder. Hopefully I'll have a chance to work that out tonight. It also only has the data that @canboat included in the original array of PGNs. I haven't added all of the other data that I'd like to see (min, max, and error values for example) mostly because I don't know what they are.

rob42 commented 10 years ago

I'm ok with SI units but we should use degrees rather than radians, and knots ( Wikipedia - The knot is a non-SI unit that is "accepted for use with the SI".).

This is mainly because of the relationship between navigational maths ( 1 knot=1degree, etc), boaties general knowledge (for support), and to save constant conversion to and from existing defacto nautical units.

Also I looked into JSON on the arduino - as I expected it uses a lot of scarce RAM, and a good slice of precious CPU. In general micro-controllers wont be able to deal with it easily. However the alternative key/value I currently use extends nicely to POS.LAT=xxxx.nn[CR] etc. And that can be easily parsed by a finite state machine concept, so its possibly more efficient than my current version.

I know just the person to build NMEA0183<>JSON conversion, I'll invite him in.

timmathews commented 10 years ago

@rob42 In my client registration document I describe how the client can request from the server which units it would like to see it's data in. This way we can let the server do all of the heavy lifting of unit conversion and the client can just display what it gets. One of the other ideas which I need to explain better is that instead of every client getting the full firehose of every message available all the time, every client would get it's own unique stream of data from the server which would only contain the data that it specifically asked for.

So say I have an iPad app which has several different pages of data displayed (perhaps an engine page, a wind/boat speed page, an environmental page, and a routing page) and each page has eight or ten data points that it displays. In my contrived example here, that's about forty different data points that the client needs to handle. In reality it may be hundreds (just look at the example pages put together by Maretron for N2KView), but at any rate the client is only displaying a small number at any one time. Therefore, we only need to listen to that small number.

Upon loading a display page the client tells the server which data it's currently displaying and the server builds a custom data stream just for it. In that stream, data is formatted exactly how the client requested it and (optionally) it arrives at a rate requested by the client.

This also helps little RAM-starved micro-controller clients, since they only have to deal with a small subset of the data available. Of course the flip side is that it pretty much makes it impossible for one to be the central server in a system like this, but I was targeting ARM-based platforms when I thought all this up.

rob42 commented 10 years ago

@timmatthews I understand the concept, its a listener pattern similar to a gui, where a component adds itself as a listener for certain types of events. Its a good pattern, but I think we should be careful to assume we have power to burn at the server.

Freeboard runs on a Raspberry Pi, and is quite capable of running a steady 50% CPU when IMU, GPS, Wind, log, et al are pushing data, and its converting/routing to a browser on websockets, and TCP port for OpenCpn. Its also running the wifi hostap, webserver for browser, and misc tasks. Its not due to slow code either, (jdk8 hard float on ARM, java is actually only about 8% cpu), a lot of the CPU is USB IO stuff.

The point is that to get low power consumption you need efficient systems on small CPUs, hence we cant assume a laptop or PC etc. In that vein I agree with tkurki that the clients have more CPU than the server, so pushing out tasks to them is a good move.

The only filtering I would use would be at the 'wind.*' level, especially for arduinos over USB serial.

Rob

rob42 commented 10 years ago

@timmatthews ...request from the server which units... Sorry I missed addressing this...the problem is tracking who wants what in which units, and routing/converting it an efficient way. The complexity it adds to implementations and the protocol is substantial. In terms of interoperability with other projects a simple protocol is a big advantage. Rob

timmathews commented 10 years ago

Let me apologize in advance for a longish post (with a bit of a rant towards the end).

Freeboard runs on a Raspberry Pi, and is quite capable of running a steady 50% CPU when IMU, GPS, Wind, log, et al are pushing data, and its converting/routing to a browser on websockets, and TCP port for OpenCpn. Its also running the wifi hostap, webserver for browser, and misc tasks. Its not due to slow code either, (jdk8 hard float on ARM, java is actually only about 8% cpu), a lot of the CPU is USB IO stuff.

If I had to hazard a guess here, the root cause of that level of CPU consumption is that on the rPi the Ethernet interface is a USB device connected to the onboard USB hub. You'd probably see significantly reduced CPU consumption on something like a Beaglebone Black which has a dedicated Ethernet interface. Just speculation as I don't have either device available to test on.

The point is that to get low power consumption you need efficient systems on small CPUs, hence we cant assume a laptop or PC etc. In that vein I agree with tkurki that the clients have more CPU than the server, so pushing out tasks to them is a good move.

Assuming we're powering and charging the clients on the boat, the net power consumption is probably about the same regardless of where the conversion is happening. Whether device A or device B is burning CPU cycles to do unit conversion doesn't really matter if they're both using a common power source of the boat's house batteries.

[T]he problem is tracking who wants what in which units, and routing/converting it an efficient way. The complexity it adds to implementations and the protocol is substantial.

You're right, it is a complex problem. A better approach to the units question may be to let the user set their preferences globally. After all, we're probably going to be doing some conversion because the n2k data (while pretty good) isn't always consistent or in a format that most people understand (radians for instance) and 0183 is all over the map. Internally, we need to pick a format and stick with it for calculations and storage, but as a last step before broadcasting, we could perform the conversions that the user wanted. It's probably better than doing a different conversion for each client and from an end user perspective, it's better than having to configure my unit preferences on every device.

Clients could obviously still do whatever conversions they want (which would be useful if you have a guest who prefers fathoms over meters or wants to know your fuel consumption in rods/hogshead).

As far as routing goes, topic-based subscriptions are also an option. See the Topics Tutorial from RabbitMQ for a primer (unless enterprise messaging is your métier too, in which case we may have more in common than open source boat bits). Depending on the implementation, this can be done client side or server side and it's usually up to the messaging stack how that's handled, ZeroMQ does it at the Publisher and RabbitMQ (and probably most other brokers) handle it at the broker.

In terms of interoperability with other projects a simple protocol is a big advantage.

I agree completely. However, one of the things that I don't like about n2k and the upcoming OneNet from NMEA is that every consumer of the data available has to already know how that data is formatted, what all the lookup fields mean, what units are being used, etc. A very real consequence of this is that in order for my Garmin chart plotter to be able to display audio track information from my Fusion stereo, Garmin has to spend engineering hours getting a spec from Fusion, implementing and testing said spec and then they have to release a firmware update which I have to download at home, put on an SD card, drive to the boat, stick in my chart plotter and then wait 30 minutes or so while it updates. This is dumb (plus Garmin has chosen not to do this for my particular model, so I'm screwed). The plotter already knows how to display audio track information on the screen in a useful manner because it can get that from a Garmin XM receiver. It should be able to query my stereo and find out that a) it provides audio track information and b) how that information is formatted. Now, I as the end user might be required to provide some help viz. mapping the specific Fusion-fields to their respective Garmin-fields, but provided each field has a human-parseable description, this is easy (and would only need to be done once).

And that is why I am proposing serving up metadata. Because it means that new players can produce messages which everyone can make use of without them having to go through a review process and become a "standard" message or be relegated to "manufacturer specific" messages that only that manufacturer and their friends can read. And most importantly, it get's closer to a real plug'n'play environment.

We focus on the how of the formatting, but not the what. We don't need canonical message IDs (PGNs), and if IDs exist at all, they can be defined by the server(s) on the network. We tell the client up-front what units we're using and then never transmit that information again unless they ask for it.

Another benefit of metadata is the ability for the end-user to globally configure boat-specific things. For instance, let's say I have an engine on my boat that tops out at 4000 RPM, and I know my fuel consumption goes all to hell above 3000 RPM, idle is 500 RPM and that the sweet-spot performance-wise is about 2200 RPM. I might want to configure my server with that information so that when a client requests RPM info, it knows automatically that it should display a gauge from 0 - 4000 RPM, that it should color 0-500 red, 2000-2400 green, and 3000-4000 red.

rob42 commented 10 years ago

@timmatthews ...unless enterprise messaging is your métier too, in which case we may have more in common... I am sitting at a desk right now building soa composites, messing with endless xml transforms, reading convoluted corporate specs, and wishing for something simple... :-( Rob

timmathews commented 10 years ago

@rob42 I wish I were so lucky. These days I'm working in healthcare, which means everything is EDI, so I get to work with fun things like EDI 834 and EDI 999. At least XML has structure, EDI has asterisks.

Anyway, my metadata API works for all PGNs now: http://pyxis.openseasproject.org/api/v1/messages, so all of those "@Details" links work.

tkurki commented 10 years ago

At this point I'd like to thank everybody for great discussion! Lots of food for though and this has broadened my horizons a lot. I am thinking of putting up a wiki page for the accumulating use cases. For example the RPM gauge colored sectors example is a clear use case for shared global configuration type metadata.

Also the mental model of a separate N2KView-like application bootstrapping itself is pretty good. It also underlines the current problem with NMEA vs N2K over wifi: all the decent navigation applications support NMEA over tcp and udp, but I am not aware of anything non proprietary supporting N2K. BTW there are many small nav apps in Google Play store, would be really good to have some app authors involved here.

As for using PGN's and N2K spec I fail to see a huge problem. The Fusion/Garmin use case underlines N2K's standardisation motivation, as if the PGN for this had been established both vendors could have used the same stuff instead of going for vendor-specific extensions. Your point about never getting the firmware upgrade is one motivational factor for me: with closed source there is no way to fix these things.

My main point here is that CANboat already provides a de facto standard with open source implementation. N2kd also provides a json streaming server - is there something more needed? Should I just bite the bullet and translate all my NMEA data to N2K-like json and modify my UI code to use that? One benefit of Node stack is that all code can be used on server and/or client at will.

Re:units - knots isn't universal, I for one like to see wind speed in m/s, but I understand I am in minority globally. The good thing about json is that there is no formal schema: if knots exist then use them, otherwise convert from m/s, as long as it is always clear what the units are so we don't miss our destination, be it a planet or just a buoy.

Also thumbs up for @timmathews for putting up the demo - working software is worth more than specs!

timmathews commented 10 years ago

I am thinking of putting up a wiki page for the accumulating use cases

Please do. Or if you're not opposed to the OpenSeas Project name: http://wiki.openseasproject.org

Should I just bite the bullet and translate all my NMEA data to N2K-like json and modify my UI code to use that?

Yes, probably. Everything that I am proposing can be built on top of what CANboat provides, so building clients for it will not be wasted effort. Since @canboat has mentioned that he would be willing to make some changes to the format, I believe that all of the field names should be converted to camelCase or snake_case instead of proper names. That would let JavaScript consumers do something like this:

var conn = new WebSocket("ws://pyxis.openseasproject.org/ws/v1/data")
conn.onmessage = function(evt) {
  var data = JSON.parse(evt.data);
  var ref = data.fields.reference;
  var windSpeed = data.fields.windSpeed;

  if(ref === 'apparent') {
    aws_kts = 1.9439 * windSpeed;
  } else if(ref === 'trueBoatReferenced') {
    tws_kts = 1.9439 * windSpeed;
  }
}

otherwise the same code looks like:

var conn = new WebSocket("ws://pyxis.openseasproject.org/ws/v1/data")
conn.onmessage = function(evt) {
  var data = JSON.parse(evt.data);
  var ref = data.['fields'].['Reference'];
  var windSpeed = data.['fields'].['Wind Speed'];

  if(ref === 'Apparent') {
    aws_kts = 1.9439 * windSpeed;
  } else if(ref === 'True (boat referenced)') {
    tws_kts = 1.9439 * windSpeed;
  }
}

I find the first version easier to read, but that's just personal preference.

[I]f knots exist then use them, otherwise convert from m/s, as long as it is always clear what the units are so we don't miss our destination, be it a planet or just a buoy.

FWIW, most (all?) n2k PGNs which deal with speed values report in m/s already.

tkurki commented 10 years ago

+1 for CamelCase. Should probably start with CANboat, so that I would have a reference. Speaking of which, see also https://github.com/canboat/canboat/issues/1

rob42 commented 10 years ago

I would do it like this:

if(data.wind){
   if(data.wind.apparent.speed) {
      aws_kts = 1.9439 * wind.apparent.speed;
    } 
  if(data.wind.referenced.speed) {
      tws_kts = 1.9439 * data.wind.referenced.speed;
    }
}

Thats why I would use this kind of format, plus its extendible, I can add my own custom attributes (accumulatedRevolutions) and it wont break your UI

 wind:{ 
    apparent:{
      angle:54,
      speed:6.32,
      accumulatedRevolutions:10000000
    },
    referenced: { 
      speed:5.172514148866067,
      angle:81.29917903663596
    } 
    },

Rob

ktuukkan commented 10 years ago

Hi all, @rob42 pointed me in here. Very interesting discussion as it happens to be so that also I have been thinking about harmonizing output, units of measurement etc. in my project. It's a basic NMEA 0183 parser for Java and could as well support JSON formats. I can also try to give some insight to 0183 if needed.

tkurki commented 10 years ago

I think I came up with a good measure/goal for interoperability: we should be able to point our respective applications at a (web)socket serving json of any other application and things should Just Work (TM).

timmathews commented 10 years ago

Hi @ktuukkan! I'm glad @rob42 invited you over, marine-api is very cool. Perhaps the most comprehensive open source NMEA 0183 library I've seen. Do you have plans to support more sentences? You're more than halfway to parity with the Actisense NGW-1 gateway.

ktuukkan commented 10 years ago

@tkurki I think that's an excellent definition :) It doesn't have to perfect on first shot, just get going and see how it works, then iterate again as the learning happens.

Thanks @timmathews, I'm glad to hear you like it. I've been adding new sentences quite slowly because users can also add them as needed, but yes I'm also thinking to add more. Any suggestions which sentences should I add next?

tkurki commented 10 years ago

Started out with changing CANboat json to use CamelCase: https://github.com/canboat/canboat/pull/13

timmathews commented 10 years ago

@ktuukkan These are the messages which the NGW-1 supports which marine-api doesn't yet, sorted by IMO the most important

Those would be the sentences I would focus on, but maybe @rob42 has a better suggested list?

rob42 commented 10 years ago

@tkurki - Ive been messing with formats too, came up with a similar result but camelCase rather than CamelCase - I'm a java geek :-)

But I feel we are missing something somewhere, its not that any of the above is not workable, its just that it doesnt feel 'right'.

So after some pondering I decided that we are talking about two formats - 'data' and 'metadata'. Data just moves the information about, metadata describes it. So that tidies the discussion about registering for data, discovering services etc. We should create a 'data' format that by 'default' uses SI units, and a metadata interface to discover, subscribe, and adjust data formats if the device supports it.

The second realization was that all current protocols are about sending snippets of data between devices. If you look at the formats they all reflect that concept - they are flat, single level, have flags for variants (true,magnetic,etc), and have no data relation to other messages.

In freeboardPLC (the arduino code) I had several processes that need to share data. Using the simple message keys was hopeless, forever passing them around, update one, and lots of other things need recalc etc.

So I implemented a model which held the data in a hierarchical form. This was much better, incoming data and sensors, just update the model, and the output is calculated from the models current data whenever needed. And the bonus was I can just dump the model (or any part) out to other devices, which can easily update from it in a consistent way.

I suggest we need to define the model first, then the data keys will simply be the full or relative path eg

model{
   environment{
       wind{...}
       sea{...}
       ais{..}
   vessel{
       nav{
             heading{...}
             speed{...}
             ....
      systems{
            autopilot{...}
            engine{...}
            ....
            toaster{..} :-)
      }
   }
}

This then opens up a whole different way to think about our software - it allows for a series of processes that collect and insert data, and others that monitor data and update displays or take various actions. A device can subscribe to all or parts of the model tree as producer or observer.

So my arduino would only care about the wind branch, the browser would subscribe to the whole tree, and get an instant full copy. Likewise peer2peer is simple, the tree could even be distributed over different devices with a bit of thought. And its still backwards compatible with current NMEA messaging, since we can easily map (using metadata) and send any key at any time.

Reading and sending over websockets can be a simple model update process, and instrument redraws run at predictable intervals. And monitoring/reporting processes (logging, engine temp, fuel, anchor alarm, etc) can run at lower priority.

Rob

tkurki commented 10 years ago

You make a good point Rob. If you think about where all this data is coming from the tendency towards simple one shot messages is understandable: sensors send out readings. I suppose there hasn't been a real, concrete need for something like what you describe - up until now.

I've done sort of the same thing: first put up one set of gauges with my own boat's data. Then I got a capture from another boat and had to deal with that. Then a third, and it called for a better way to extract the boat-specific configuration from all over the place. A single file that knows which individual data streams should be used for the different data items that the UI widget wants and how to derive the items that are not already present in the streams.

But how to move forward? One way I can think of is to share both data and processing:

rob42 commented 10 years ago

Well the data is naturally web friendly, so its not hard to expose our own servers and data streams. Been rethinking the model structure...

We should start by building a 'global' tree. Root is 'data', immediate child is a 'vessels' collection, one of which is your own vessel (and AIS populates the other nearby vessels). There may be other collections, maybe 'official', like forecasts,etc. Then within a vessel we have environment, motion, and systems. And so on...a bit like creating a db data model.

If we think 'distributed' from the start then when two suitably equipped boats come within wireless range, or stream data via mobile internet, then they can connect and merge their trees AIS style, so you can see their data too! That would be cool - seeing your mates, or race competitors on your chart and instruments - and totally beyond the current commercial 'state of the art'.

In the meantime if we each stream our models, we should be able to run clients against any one of them. Mine would be 'data.vessels.motu'...hmm, need to think about name collisions.

I think the next step is then to collect all the current keys, which we can extract from NMEA0183, and NMEA2000. Plus the ones we know from common sense. Then we arbitrarily assign to collections and sub-collections. Fairly soon we will have a tree, which we will probably rehash a couple of times.

If @timmathews , @canboat and @ktuukkan can modify their libraries to fit, we would soon have pretty powerful base data to build against.

For the immediate future we modify our code to provide and read everything by default, and then develop a server to provide meta-data, handle subscriptions and config requests, store history, etc.

Id like to hear others opinions - is this model the right direction forward? Have we missed anything? Who is in?

Rob

keesverruijt commented 10 years ago

I like this tree model stuff. It neatly suggests a query language as well: REST.

So if <uri> returns

   model{
       vessels[
           "244060807": { name: "Merrimac" } 
       ]
    }

then <uri>/model/vessels/244060807 would return the model for just one vessel. <uri>/data/vessels/244060807 would return current data for just that vessel.

@rob42 the vessel name collision could be solved by using MMSI instead of name.