search

SignalK / specification

Signal K is a JSON-based format for storing and sharing marine data from different sources (e.g. nmea 0183, 2000, seatalk, etc)
Other
91 stars 68 forks source link

Magnetic Deviation #242

Open tkurki opened 7 years ago

tkurki commented 7 years ago

For example HDG NMEA 0183 has heading, I believe we are missing it.

A bit tricky, as it is associated with the sensor and current heading, so where should it go?

timmathews commented 7 years ago

Deviation should be stored wherever we define static properties of a sensor. Its naturally a table of heading ranges and deviation angles. It should probably be added to the headingMagnetic and headingTrue values before they're sent out on the wire.

What is the NMEA convention here? In the HDG sentence, if deviation is present should it be added to the heading value or is purely informational and assumed to be already accounted for by the instrument?

A deviation table schema might look like:

{
  "deviation": [
    {
      "start": 0,
      "stop": 15.99,
      "deviation": 3
    },
    {
      "start": 16,
      "stop": 30.99,
      "deviation": -1
    },
    {
      "start": 31,
      "stop": 45.99,
      "deviation": 7
    },
    { ... }
  ]
}
sumps commented 7 years ago

@timmathews further to our discussions last night, I think we should have a dynamic deviation value in Signal K that we take from the NMEA 0183 HDG sentence and NMEA2000 PGN 127250 if present that is the current deviation value that is being used by the heading system. I am not sure how many systems output this data but even if they do not, if we have deviation defined in Signal K then sensors can use it or if we also support a Deviation table, a Signal K server or client could interpolate the current deviation and populate the current deviation key.

The deviation table should be a table with deviation values every 15degs, so 24 values which then allows for a very accurate table with all values entered or a less accurate table with values for every 30degs or even 45degs.

sumps commented 7 years ago

A server could also add value by working out magnetic variation using this code from NOAA...

https://www.ngdc.noaa.gov/geomag/WMM/soft.shtml

rob42 commented 7 years ago

This has more to do with correcting data than the real value of the heading or any other key. If I understand correctly the deviation we are talking about here is the compass error on the N2K compass for the given heading? There is no point in storing both the inaccurate result and the correction, we should correct the data, then store the accurate value in heading.

This same adjustment is made by various sensors internally (calibration) and needs to be applied to some data by the server, eg wind vane offset.

There was discussion on this recently (slack?) and I proposed a 'correctIncomingSensorData' functionality, but I think its an implementation choice. Not sure we can have a std format for the data we might need to store. Definitely a bad idea to start sending data plus corrections (that the receiver must apply anyway).

Easiest is to add "local adjustments to correct sensor data MUST(SHOULD?) be applied before data is updated in signalk. How this is done is implementation specific" for v1

Then we can do more after v1?

tkurki commented 7 years ago

Changed to after v1.

My point in this is that if we want this as part of the spec it needs to be in the data model, and I agree with Tim's point about this being part of the sensor.

Server correcting the incoming heading with this is beyond SK spec and an implementation detail.

sumps commented 7 years ago

Agree and I am dubious as to how many heading sensors actually output this data. They will send deviation corrected values out.

rob42 commented 7 years ago

Probably still a good idea to be clear about it even for V1 "Data in signalk is assumed to already be corrected for any known sensor adjustment"

sumps commented 5 years ago

During some work on iKommunicate, it became clear that we never did support magnetic deviation in the SIgnal K V1 spec. Think at the very least we should have the following key so that we have somewhere to store NMEA0183 and NMEA2000 Deviation values...

/vessels//navigation/magneticDeviation Units: rad (Radian)

Description: The magnetic deviation at the current heading that must be added to the magnetic heading to correct it for local magnetic deviation. Easterly deviations are positive and Westerly deviations are negative (in Radians).

rob42 commented 5 years ago

"Data in signalk is assumed to already be corrected for any known sensor adjustment" .

The gateway or sensor should apply the correction before sending the data.

The actual deviation varies with heading, its traditional to have a 'card' of values around 360deg so you can adjust any given heading for the correct amount. Its much simpler to fix this at the sensor or gateway than propagating the error around the network and having every subscriber compensate.

sumps commented 5 years ago

I agree @rob42 and if we were only dealing with new SIgnal K sensors then this would be fine, but we have to work with legacy, current and future 3rd party NMEA0183 and NMEA2000 based products and these output Deviation and at the moment there is nowhere to put this data in Signal K, so that is why I am proposing having a key for deviation and then developers can decide how they want to use it - No Key, No Option ;-)

tkurki commented 5 years ago

Even if data is adjusted so that deviation is not needed a user may want to display it, so having it in the schema is useful.

@sumps I think Rob's comment was more about your description of the value: that must be added to the magnetic heading to correct it. This is not in line with "all values being corrected beforehand" thinking.

Nor is local appropriate here, or talk about world magnetic model, as they relate to magnetic variation/declination - the diff between true & magnetic north in a certain location.

As I see it there are

  1. "current" deviation - related to the boat's current heading and the magnetic sensor that produces it
  2. Deviation table - again related to the magnetic sensor

(1) could be navigation/magneticDeviation/current and then the source is the key to associate that with the navigation/headingMagnetic that it is "current" for.

(2) could be a structure such as Tim's above, only where should that go? navigation/magneticDeviation/deviceId/ (where deviceId would be $sourceId?)

sumps commented 5 years ago

I don't think it is correct for developers and users to make assumptions about the data and whether it has been adjusted for variation and/or deviation.

Basically NMEA devices that transmit heading send the raw measured magnetic heading and if known the variation and deviation values for the "local" situation i.e. the local variation taken from the electronic chart or calculated from the world variation model and the local deviation table for that vessel.

Often a compass sensor will have been calibrated to remove the local deviation effects in which case its heading will include deviation and it will either output 0 deviation or "Null" data not available.

The heading, variation and deviation always have to be linked to the source of data, just like the Position, Datum, Satellite Status, etc. from a GPS but this is handled by the source metadata.

Variation can also change, albeit more slowly than deviation, so I think having some abstract concept of a current key is over complicating things.

Basically the majority of heading sensors with be communicating by NMEA and I just want a Signal K key in the schema to put this data in.

inspirity commented 5 years ago

I never saw a magnetic compass with the ability to add the vessels deviation to get a deviation corrected HDM as output. Since deviation is another vessel owned value over 360 degress like the polar performance.* keys. We should handle it the same way. I agree with Turki for the path navigation/magneticDeviation/current and have the deviation table for the vessel separate as the polar table would be. It is a fundamental decision having those correction/performance tables into the SK-tree or as external data sources.

sumps commented 5 years ago

I don't understand this concept of "current" by its very nature the deviation value will be the one applicable to the heading the compass is outputting.

You need another "static" deviation table stored in another part of the schema applicable to a particular compass sensor which an app can load at start up and apply to any values it receives from the applicable compass sensor.

As a general principle, I think we should always have a Signal K key for all NMEA data so that there is clear and accurate conversion of the data, with no further ambiguity or introduction of errors. Then if we think there is benefit in improving or expanding on that data, we extend the schema, create new derived values, implement new ways of doing things, etc.

rob42 commented 5 years ago

Just for clarity here - I was talking about compass deviation as compass inaccuracy eg most vessels (especially steel) have local magnetic fields that make the compass read wrongly in various directions.

Then we have the external magnetic deviation due to the earths magnetic poles not being aligned with geographic poles. We have navigation.magneticVariation for that, usually derived from world magnetic model based on location.

Which magnetic deviation are we talking about here?

@inspirity true, most current compasses dont, but in signalk we should fix sensor errors at the edge of the signalk network, not propagate the error info too. Incidentally thats different from navigation.magneticVariation, which is a valid data, since its not an error.

Early on we went to SI units throughout signalk, which dropped the need to send units and greatly simplifies apps and processing since we dont do conversions (except in the gui). To me errors are the same, we should fix the data rather than propagate an error.

inspirity commented 5 years ago

Magentic Variation or Declination is not the same than Magnetic Deviation. Variation or Declination is due to the difference between Magnetic and geographical Northpole Deviation is the ships own error.

I need the deviation AND the variation to navigate without GPS. So if my electronic compass fails and my gps fails i could still have an app calculating my compass course to navigate to the next waypoint. Without the deviation data i simply can't. It is the same if i want know my performance without polars. Its just a guessing. I absolutely agree and support calibrated sensors and not errors. Deviation is not more error than Variation since is not sensor related but boat related. Like the polars are not errors but boat related Information.

sumps commented 5 years ago

This theoretically perfect world where we have Signal K sensors that output perfect data is great but we still need to work with imperfect NMEA based sensors and displays and if we wish to receive and transmit imperfect data then we need to have keys to store them in.

rob42 commented 5 years ago

Any data can be wrong. Thats why eyeballs and brains are still the primary navigation instrument :-) But when we know errors we should correct then as near to source as possible.

That said I understand better what @inspirity is after now. If you have a compass that simply sends its heading and has no capacity for holding/correcting for deviation, then storing it and applying it elsewhere makes sense. I was thinking of @sumps example of an N2K packet that sent both heading and deviation, that should be corrected in the gateway.

There was some discussion elsewhere about applying corrections to signalk data but I cant find it. The general idea was that incoming data would be parsed for keys that were known to need correction and the corrections applied. It was a server implementation issue.

I think we need a generic way to add a correction factor into any signalk key. the meta structure seems to be the place, so meta.correction.* ?

The process would need to differentiate by source, and possibly a method to ensure it didnt correct the same value twice (eg on echo, or mirrored servers). Not simple.

inspirity commented 5 years ago

@rob42 i don't see deviation as an error but similar to the polars as a data table connected to the boat. Calibrations in meta.correction for a deviation table would have to be a function since the deviation depends from the headingMagnetic. We already have the headingTrue Key in the definitions wich should contain the Variation AND the deviation. The Deviation is usually a table. Question is do we store this data in SK or do we keep this external. Just to clarify what we are talking about: deviationtable

tkurki commented 5 years ago

If i want to build a display that shows the current deviation next to my analog compass I want that in the sk data model.

If I want show a deviation graph as in the previous post I want that in the sk data model.

Neither of these have anything to do with whether or not heading* values are corrected. I think that is a separete issue.

pe 14.12.2018 klo 23.25 Olivier Hendrikx notifications@github.com kirjoitti:

@rob42 https://github.com/rob42 i don't see deviation as an error but similar to the polars as a data table connected to the boat. Calibrations in meta.correction for a deviation table would have to be a function since the deviation depends from the headingMagnetic. We already have the headingTrue Key in the definitions wich should contain the Variation AND the deviation. The Deviation is usually a table. Question is do we store this data in SK or do we keep this external. Just to clarify what we are talking about: [image: deviationtable] https://user-images.githubusercontent.com/6239660/50028164-d2199480-ff92-11e8-9a4f-9ab7455aa8c6.jpg

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/SignalK/specification/issues/242#issuecomment-447483740, or mute the thread https://github.com/notifications/unsubscribe-auth/ABAETnWch3-x2zecg5AnkLTTinRmmMr3ks5u5Bc3gaJpZM4JsY5A .

inspirity commented 5 years ago

I support tkurki's point of view.

Maybe we should just clarify in the key's Description:

sumps commented 5 years ago

I also sense from this thread that no one knows whether any systems populate the deviation fields in NMEA 0183 and NMEA 2000 so having a key that iKommunicate, CANBoat, etc can populate will give us, for the first time, the visibility to start understanding what data is available and from which systems.

inspirity commented 5 years ago

@sumps Some Manufacturer let you drive slowly a circle in flat water with no current to populate their internal deviation table by storing the difference between COG and HDM + MagneticVariation.

sumps commented 5 years ago

The more we describe these marine values and terminologies, the more value we add to the schema and work that everyone is doing.

HeadingTrue used to be only generated by expensive gyro systems on commercial vessels and now with GPS Compass and MEMs sensors it will become a much more common data type.

In these circumstances having a Signal K Server that can efficiently and effectively manage the steering compasses deviation table is a desirable goal.

sumps commented 5 years ago

@inspirity yes Digital Yacht’s HSC100 is a good example and it is purely a magnetic calibration no use of COG is made.

rob42 commented 5 years ago

Taking the data above from @inspirity we can fulfil both @tkurki 's use cases: 1) show compass heading, and current deviation (match heading against table in meta.corrections) 2) I can draw a graph from the table.

That bits easy, but there are complex issues if we dont apply corrections as early as possible: a) When the autohelm subscribes for headingMagnetic, does it expect the correction has been applied, or does it check for a correction and apply it to every value it receives? b) if the correction is optionally applied, how do you find out if you have a corrected value or not? c) if you pass the value onto another signalk device should you correct or not? d) how do you stop a correction being applied twice when it passes through a chain of servers?

That whole mess is avoided by applying any corrections as early as possible, eg at the first 'smart' signalk device, be it gateway or server. Then we have the general rule that signalk data is as accurate as possible (but no guarantee its actually accurate)

By all means we should store the correction data, but when you subscribe to a value it should already be corrected. So if you want the uncorrected data (the less likely use case), you get corrected value, and the correction data and remove it to find the raw value. Or we store the raw value in meta too?

inspirity commented 5 years ago

@rob42 If you have a mechanical compass on the boat you know the headingMagnetic. You can compare it with the values you get from your electronic devices and you know if and which correction is applied. I can tell on my boat since i have some with the variation correction, some without and none of them with the deviation.

inspirity commented 5 years ago

@rob42 I dont think it is too complicated since the path with the description is clearly defined

To your questions:

a) When the autohelm subscribes for headingMagnetic, does it expect the correction has been applied, or does it check for a correction and apply it to every value it receives?

headingMagnetic has no correction headingTrue has it. So if the autohelm wants the corrected value then it should subscribe to headingTrue. If your Autohelm shows the course on the Autohelm, you can compare it with your gps (when subscription to headingTrue) if there is a small difference depending on which course you are sailing then you dont have the deviation. If your autohelm subscribes to headingMagnetic then autohelm and compas should show the same value.

b) if the correction is optionally applied, how do you find out if you have a corrected value or not?

See my previous post

c) if you pass the value onto another signalk device should you correct or not?

headingMagnetic has no correction headingTrue should have Variation and Deviation If your source is connected to headingTrue then no further correction

d) how do you stop a correction being applied twice when it passes through a chain of servers?

There should be no correction from server to server or if there is correction then the path will change. Same path no correction.

The important step is to figure out what the source delivers.

This is only an installation challenge.

This is an important discussion while most boats i know have a mess with instruments all showing different headings while not many people really care. SK could be very helpful with clear definitions.

rob42 commented 5 years ago

I have GPS, which shows heading true, but it becomes increasingly erratic as speed slows, since the GPS inaccuracy of +-10M min affects the heading of a boat at <2knts a lot. At anchor the GPS heading is useless.

I have an IMU, which is accurate to +-2deg, but very susceptible to any stray electric currents and ferrous/magnetic objects. So it constantly auto-calibrates, sometimes stupidly if you place the wrong object near it. Of course my dearest throws my tools onto the helm station, which is where the IMU also lives.....

I have an old school magnetic compass on the helm which was always great until I put a Samsung tablet on the helm. Tablets have powerful magnets to hold covers and keyboards, and if its close enough to read it affects the compass to some degree...The autohelm also has an internal compass, its near the helm too.

So I can actually steer my boat by waving my tablet around, or better yet arranging spanners across the nav table in holistic ways :-)

rob42 commented 5 years ago

My view is that the navigation.headingTrue.value is the heading of the vessel relative to N-S on the chart. navigation.headingMagnetic.value is the heading of the vessel relative to magnetic north. So navigation.headingMagnetic.value + declination = navigation.headingTrue.value (or maybe -declination, I forget.)

Both should be the best values we have onboard, errors should be corrected if we know them. headingMagnetic.value is not the value on the compass (which compass?), its the magnetic heading of the vessel.

Even those values should be corrected for known errors, eg deviation. This becomes even more important if you want a high reliability autonomous vessel. You read all the compasses and compare. Make sure a majority of sources agree within a certain error, and go with that. Flag the wayward sources as faulty.

sumps commented 5 years ago

I fear introduction of the term declination will just cause further confusion. Please let us keep to the two terms Variation and Deviation.

I'm sorry @rob42 but you cannot get Heading True from a single GPS sensor and if it is outputting heading true NMEA data then it is a poor product design.

You can get COG (True) from a single GPS but this is only accurate when you start to move, as you have seen. You need two or more GPS sensors, separated by a certain distance to calculate True Heading, such as the system developed by Hemisphere and badged/customised by various Marine Electronics companies.

A vessel's heading source, should be considered/handled in the same way as a vessel's position source with fully traceable data from each source. Mixing of data from different sources should be avoided and it should be the user/installer who selects the "best" source.

The reality is that all electronic compasses have a calibration routine that removes (as far as possible) the local magnetic influences on the boat that create the deviation at various headings. I do not know if all compasses then output a corrected heading.magnetic with a 0 deviation value or whether they output the measured heading.magnetic and the deviation value that was calculated during the calibration. I suspect the former but I have not tested/recorded every compass sensor on the market and that is why I am keen to be able to easily "see" deviation in the Signal K world.

Every helmsmen will then steer their boat to either a conventional compass or an electronic compass display and there is a requirement for that steering compass to have a deviation table in order to adjust the Course to Steer calculated by the Navigation system to match the steering compass i.e. the bearing to next waypoint is 123degs True, the MFD knows the Variation and adds/subtracts it creating a magnetic bearing to waypoint of say 125degs Magnetic and then if a Signal K app had a deviation table for the steering compass, it would know to add another 3degs to the Course to Steer on this heading to offset the deviation errors on that particular vessel's steering compass.

At the end of the day, every boat is different, every skipper sails their boat differently and every marine electronic system handles heading differently. All we can do is ensure that the schema can effectively store the various different values correctly and it will come down to the Server/App developers and the owners themselves to validate and use the data in the way they see fit.

However, we do all need to agree on these navigation terms and principles, otherwise we will never reach a good solution.

inspirity commented 5 years ago

We should keep different things apart to get results.

Can we agree to handle those topics separately or do I misunderstood or miss some of those aspects?

Thank you for the great job to all of you... Very constructive

sumps commented 5 years ago

No we can and should keep different things apart but you forgot one topic - the original question that this issue asked - where does Deviation go and I still believe this should be...

/vessels//navigation/magneticDeviation Units: rad (Radian)

Description: The magnetic deviation at the current heading that must be added to the magnetic heading to correct it for local magnetic deviation. Easterly deviations are positive and Westerly deviations are negative (in Radians).

sumps commented 5 years ago

It matches the way we handle variation, they can go next to each other in the list of keys, so people understand all aspects of the magnetic heading data and I think it should also be covered in a "Signal K Navigation Principles" White Paper which I am keen to write if I ever get a spare moment.

rob42 commented 5 years ago

@sumps - yes i am fully aware that GPS heading is not that useful. Thats why I am author of both signalk IMU projects, and I spent literally weeks full-time building calibration code for the first - the most difficult maths Ive ever done.

I also added magneticVariation to signalk, but on consensus it was changed to magneticDeclination, so I used that term now.

I have no objection to navigation/magneticDeviation, and storing the table in meta.corrections makes sense.

But I feel strongly that signalk data should be corrected for known errors, eg magnetic deviation - the compass error. So

/vessels//navigation/magneticDeviation Units: rad (Radian)

Description: The magnetic deviation of the compass at the current heading that was added to the compass heading to correct it for local magnetic deviation. Easterly deviations are positive and Westerly deviations are negative (in Radians).

You cant expect every subscriber to a data point to consider sensor error and check adjust for it. In a complex system it will cause endless problems, more complex coding, more bugs, and unforseen conundrums.

I so often seem to be the one with the contrary view, it comes from 30+ years working on many of NZ's most complex enterprise IT systems, across generations of change, often troubleshooting. Im not arguing because I'm argumentative, Im arguing because Ive seen so many problems repeated across systems, and I dont want them in signalk.

I know you all bring your own experience, and I respect that, but I have unique experience of mistakes in complex distributed systems, and propagating errors is one of them.

inspirity commented 5 years ago

@rob42 I understand absolutely what you talking about. This also means that most of the sources on boats wouldn't be able to deliver a corrected headingMagnetic value for SK. I'm navigating and sailing since more than 40 years and since 10 years full time around the globe. I never met a boat with a deviation calibrated electronic compass. I guess i have to investigate and get one.

This is no big issue, we could define headingMagnetic as the 'error free' value and let the user or source responsable to get the correct value. This is the actual situation on most boats anyway since most captains use gps's and don't care about deviation. SK doesn't need to do any special calculation. It is just a definition aspect what headingMagnetic in SK means, with or without deviation correction. That all a developer needs to handle it from there. Sk is not responsable for the quality of the sources, as long the definitions are clear. Till now i didn't know which value headingMagnetic expected.

Do I understand your proposition is:

I would totally agree with this solution.

sumps commented 5 years ago

In an ideal system you would make sure that all data was corrected/adjusted at source but I fear that on your average boat this is not the case or at least you have no way of knowing if this is the case or not.

This is where a Signal K Server could add value providing extra functionality, beyond what your average MFD or Instrument system does, to apply corrections to the received data or derive new values from the data before passing it on to other SK devices/apps.

It would be worth considering having a metadata field that says if the value is raw, calibrated or derived and even perhaps a quality rating so that different sources can be rated by the user/installer.

I am a bit worried about this comment you made...

“I also added magneticVariation to signalk, but on consensus it was changed to magneticDeclination, so I used that term now.“

I can find no Issue or PR for this change from navigation.Variation to navigationDerivation and this would break V1 backward compatibility. Can you clarify what you meant?

rob42 commented 5 years ago

@sumps you are quite right the spec has magneticVariation not magneticDeclination. More than happy to go back to using 'variation', never liked 'declination' anyway. Odd, I remember the discussion, however it was in the very early work so lost in time now.

rob42 commented 5 years ago

@inspirity - exactly, headingMagnetic as the 'error free' value, where error-free means its correct as best we know.

We can never have error free sensors ( https://en.wikipedia.org/wiki/Observer_effect_(physics) ), and never really know how precise they are. But if we have a compass we know is out 4deg at E, then we should fix that value in signalk (if we can).

From a consumers point of view we know that the value may not be correct, but we also know its the best we will get.

@sumps additional metadata on data quality is an interesting idea but I fear it will create a quagmire of quality definitions and estimation methods. In the end you get a data value (as a consumer) and you choose to trust it or not. If I wanted to fake data, then I would also fake the quality flag, so as a consumer you are no better off. Something to leave for V2+

So are we in agreement:

Rob

sumps commented 5 years ago

So how does a Signal K Compass Sensor communicate its data to the server, would we have to define headingMagneticRaw ?

I don't quite see how the fake data argument has made its way in to this issue. This is a subject all of its own and I assume is being tackled under Signal K Security.

Having a field that says whether data is Raw, Calibrated, Derived, etc. seems to make a lot of sense to me, allowing you to have one key and then a standard way for any data key to be categorised. It does not break V1 and could be added tomorrow as V1.x

Perhaps the use of a data Quality flag is a step too far, put a Priority flag in metadata could be used across all Data types as a way for the user/installer to select what they know to be the best or most reliable data of that type. The Priority field could also be auto-populated in NMEA2000 systems based on the CAN address - lowest being highest priority. Again a Priority flag could be added without breaking V1.

inspirity commented 5 years ago

@sumps without need to dig into metadata, flags etc. It is possible to add a key headingMagenticRaw which would hold a value without Deviation and Variation. We would have all the possibilities covered.

There are only Deviation and Variation between Magnetic and True. everything else is an sensor error and as @rob42 explained (even if rob42 sees the deviation as an error), this should be corrected at the source or where ever.

Once we have the definitions set we can figure out if, where and how the correction are made. This would be a different thread and applicable to all the errors and corrections.

The same applies to priority lists - defining which source is the best and in which order.

rob42 commented 5 years ago

Fake data was only demonstrating that any sort of data quality measurement provided by the server is just as suspect as the data, so of dubious value.

Raw, Calibrated, Derived, etc. could be added but I dont see the use case for it.

We already have a definition of the 'default' data value, its key.value, other sources for the data are in key.values.*. The artemis server actually uses a primary key internally to flag the preferred source, but a priority key would be better.

sumps commented 5 years ago

My suggestion of headingMagneticRaw was not meant to be taken literally, as there are many other data sets that have raw and calibrated data and I do not want to increase the number of keys in this way.

The key thing here is that there is no way to automatically tell from an NMEA compass sensor whether it is calibrated for deviation, whether it is a derived value say from a GPS Compass and calculated variation or any other combination.

We are dealing with an imperfect situation, as you and Rob have both already stated/agreed and so I would stick to the keys that we can populate automatically and accept/document that they may not be accurate and then consider a way through metadata to allow users/installers to categorise and prioritise the data.

inspirity commented 5 years ago

@sumps I see. The question is also how to implement a correction/calibration handling in SK.

Metadata is one possibility but since some corrections like deviation are not static but changing with the heading I dont see how to do this in the metadata object.

I mention once in slack about having a delta value and a factor value in the source meta data to be able to do some basic corrections but as @rob42 illustrated this could become a nightmare to handle on a installation with several servers.

We would need something like a values preprocessor once at the Input of the SK Server (similar to the InputHandler in the NodeRed plugin) and forget about errors in the data tree. We just need to know where which data goes. This could be an external addition and would keep any SK Server clean.

@sumps the problem with the quality of the data source is not only compass related and should be handled in a general approach.

sumps commented 5 years ago

@inspirity the value of deviation is data that is why I am proposing it has a key navigation.headingDeviation. The value of magnetic heading is also data and it will have a key navigation.headingMagnetic but it could potentially also have some metatdata fields to tell whether the value is raw, calibrated, derived, etc. and what priority it has in the system.

I already said that these metadata fields could be applied to any data object and that they could be introduced without breaking V1.

I think we need more opinions on this, otherwise we will start to repeat ourselves and never agree a solution.

inspirity commented 5 years ago

@sumps I understand your point. This would imply that I would have to check the metadata of every path or key to know with what kind of data I deal with before I can use it. Why then not just have heading as path and in the meta data define if it's true, magnetic raw, high priority etc? Where should we draw the line? I'm just trying to understand the practical advantages of this kind of metadata. We could end up with a headingMagnetic value the same than headingTrue just becaus they would both be raw uncalibraded etc.

One thing is to know the quality of the data. To start calculate corrections and calculating source choices is a completely different decision. So I agree to have the information about the quality like raw, uncalibraded, reliable or what ever... If this implies calculations then we could end up where @rob42 warned us.

I'm not arguing this is a very important matter and I'm thankful for the dialogue. To other readers, should this still be part of this thread or do we need a new title?

joabakk commented 5 years ago

From my theory days, once you have corrected magnetic heading for variance and deviation, you have true heading. Same way a gyro heading is corrected to get true heading. Some sensors or systems push out a current deviation, but I would not trust these, as I do not understand how they are made or trust the auto creation of internal deviation tables without a true reference. The prudent approach is to add a deviation table. But I don't understand the need for such a long discussion to decide adding the deviation path.

JonRowe commented 5 years ago

You have compass heading, magnetic heading and true heading. Auto deviation tables are to correct compass heading to magnetic heading. You still need a ~deviation~ variation correction table to reach true I believe, based on the position you are in.

joabakk commented 5 years ago

You are right, except that the earth's magnetic field correction is called variation. The NMEA sources that have been used in defining SK paths have used magnetic heading instead of compass heading. It is basically an uncorrected magnetic heading using a compass as source. See http://www.catb.org/gpsd/NMEA.html#_hdg_heading_deviation_amp_variation

sumps commented 5 years ago

Anyone interested in this subject should read this article...

https://en.wikipedia.org/wiki/Magnetic_declination

Aside from the fact that they use the term declination more than variation (they are the same), it covers the subject pretty well. The reason for why Signal K are using variation, is that this term is more commonly used in the marine world and throughout the NMEA specifications.

It is important that we are all "singing from the same hymn sheet" if we are to resolve this issue.