VMG target speed table broken

[jpilet]

when running "generateDevDB.sh" on master, I get a VMG target for Irene looking like this:

but it should look somewhat like this (upwind)

and downwind:

[jonasseglare]

So this is what it looks like when I run it on my laptop with the current state of the master branch:

[jonasseglare]

Running generateDebDB only on Irene, with --debug activated, outputs some plots too:

[jonasseglare]

Will investigate what they mean...

[jonasseglare]

Maybe a good approach for debugging would be to plot the raw points to which the curve is fitted.

[jonasseglare]

So with the current state of jp-gen-dev-db, I get this curve for Alinghi:

It looked different from what Julien got. But both curves look ugly. Could it be that we use uninitalized memory somewhere? Maybe I should clean the db and rerun the algorithm to see if I get the same curve again.

[jonasseglare]

I reran the algorithm, and the output looks pretty much the same:

[jonasseglare]

One possibility: Could it be that we load both the .log files and the .csv data from the AlinghiGC32 subdirectory? Let's investigate that, because the CSV-data is time shifted by 2 hours w.r.t. GMT, meaning that things might get bad.

An easy way to test that is to remove the .csv file temporarily, in order to see if the curves look different.

[jonasseglare]

With the removed CSV file we get the same plot:

So probably, that is not the problem.

[jonasseglare]

This is what running the processor on Alinghi with --debug flag produces:

so it is probably something in the computational backend that is not OK.

[jonasseglare]

I tried the --vmg:blind option, but it doesn't improve anything, really:

[jonasseglare]

This is what the raw data points look like for upwind (left) and downwind (right):

[jonasseglare]

I forced the calibration parameters to their default, but that didn't change much:

[jpilet]

what are the axis/units on those last graphs?

[jonasseglare]

X-axis: true wind speed Y-axis: VMG

unit: Knots

[jonasseglare]

So now I am at [jo-debug-target-speed-dev 0c66270].

If I first run the boatlog processor on Alinghi GC32 with that commit, and then from within Octave run the script called src/server/nautical/tgtspeed/visualize2, I get this graph:

Quite interesting... There is something fishy with the Simulated VMG, which is also the VMG used to produce the target speed curve.

[jonasseglare]

As a reminder, the VMG is computed in DispatcherTrueWindEstimator.cpp, using this code:

  Velocity<> boatSpeed = _filter.gpsSpeed();

  Velocity<> vmg = cos(twa) * boatSpeed;
  _dispatcher->publishValue(VMG, srcName, vmg);

So what are TWA and GPS Speed? First, we can plot the channels that are in the dispatcher, but it would also make sense to actually plot the filtered values using inside DispatcherTrueWindEstimator to understand what is going on.

[jonasseglare]

So here are those plots:

[jonasseglare]

For the sake of completeness, here is the GPS bearing:

[jonasseglare]

The above snapshot was taken at 2016-06-21T12-03-32 2016-06-22T15:10:51, with a margin of +- 2 minutes (a total of 4 minutes or 240 seconds).

[jonasseglare]

What about TWDIR, AWA, and AWS?

[jonasseglare]

So it seems like there are still problems in our true wind estimation. And those problems lead to more problems in the target speed estimation.

Remember that our true wind estimate depends on:

• gps bearing
• gps speed
• awa
• aws

[jonasseglare]

By the way, the AWA plot above is interesting: There seems to be two anemometers, with different calibration... Which one are we using?

[jonasseglare]

What happens if we take the Alinghi dataset and run twice, in either case disabling one of the Anemometers?

[jonasseglare]

The choice stands between:

NMEA2000/0 NMEA2000/c078be002fb00000 NMEA2000/c0aa82002fb04000

[jonasseglare]

With the commit [jo-debug-target-speed-dev 4cea5c7], where I only include apparent wind source NMEA2000/c078be002fb00000, I obtain these plots:

The upper part of the screen is downwind, the lower part is upwind.

[jonasseglare]

And these are the plots that I get if I only use NMEA2000/c0aa82002fb04000 as apparent wind source:

I think the first set of plots looked better, with the source NMEA2000/c078be002fb00000

[jonasseglare]

Here is a summary of the entire loaded datasets before the apparent wind channels have been removed:

NavDataset summary:
Merged channels:
  * Channel AWA (apparent wind angle) with 380009 values
  * Channel AWS (apparent wind speed) with 380009 values
  * Channel TWA (true wind angle) with 541968 values
  * Channel TWS (true wind speed) with 687212 values
  * Channel TWDIR (true wind direction) with 541986 values
  * Channel GPS_SPEED (GPS speed) with 296973 values
  * Channel GPS_BEARING (GPS bearing) with 170702 values
  * Channel MAG_HEADING (magnetic heading) with 131975 values
  * Channel GPS_POS (GPS position) with 310579 values
  * Channel DATE_TIME (GPS date and time (UTC)) with 40278 values
  * Channel TARGET_VMG (Target VMG) with 404828 values
  * Channel VMG (VMG) with 404828 values
  * Channel ORIENT (Absolute anemobox orientation) with 344173 values
NavDataset internal dispatcher: 
Dispatcher:
  Channel of type awa named NMEA2000/0 (prio: -10) with 386 samples with 0 listeners ()
  Channel of type awa named NMEA2000/c078be002fb00000 (prio: 0) with 145292 samples with 0 listeners ()
  Channel of type awa named NMEA2000/c0aa82002fb04000 (prio: 0) with 234717 samples with 1 listeners (0x3eacf60 )
  Channel of type aws named NMEA2000/0 (prio: -10) with 386 samples with 0 listeners ()
  Channel of type aws named NMEA2000/c078be002fb00000 (prio: 0) with 145292 samples with 0 listeners ()
  Channel of type aws named NMEA2000/c0aa82002fb04000 (prio: 0) with 234717 samples with 1 listeners (0x29fdcf0 )
  Channel of type twa named Anemomind estimator (prio: 0) with 396742 samples with 0 listeners ()
  Channel of type twa named NMEA2000/c078be002fb00000 (prio: 0) with 145226 samples with 1 listeners (0x2785af0 )
  Channel of type tws named Anemomind estimator (prio: 0) with 396742 samples with 0 listeners ()
  Channel of type tws named NMEA2000/c078be002fb00000 (prio: 0) with 290470 samples with 1 listeners (0x2edf360 )
  Channel of type twdir named Anemomind estimator (prio: 0) with 396742 samples with 0 listeners ()
  Channel of type twdir named NMEA2000/c078be002fb00000 (prio: 0) with 145244 samples with 1 listeners (0x2945980 )
  Channel of type gpsSpeed named Internal GPS (prio: -2) with 46980 samples with 0 listeners ()
  Channel of type gpsSpeed named NMEA2000/c07891002fb3645a (prio: 0) with 296941 samples with 1 listeners (0x476e440 )
  Channel of type gpsBearing named Internal GPS (prio: -2) with 46980 samples with 0 listeners ()
  Channel of type gpsBearing named NMEA2000/c07891002fb3645a (prio: 0) with 139461 samples with 1 listeners (0x283a250 )
  Channel of type magHdg named NMEA2000/c050a0012fb3245a (prio: 0) with 131975 samples with 1 listeners (0x408a3f0 )
  Channel of type pos named Internal GPS (prio: -2) with 24856 samples with 0 listeners ()
  Channel of type pos named NMEA2000/c07891002fb3645a (prio: 0) with 310579 samples with 1 listeners (0x2c50a10 )
  Channel of type dateTime named Internal GPS (prio: -2) with 16533 samples with 0 listeners ()
  Channel of type dateTime named NMEA2000/c07891002fb3645a (prio: 0) with 40278 samples with 1 listeners (0x312fef0 )
  Channel of type targetVmg named Anemomind estimator (prio: 0) with 404828 samples with 1 listeners (0x459a030 )
  Channel of type vmg named Anemomind estimator (prio: 0) with 404828 samples with 1 listeners (0x424d010 )
  Channel of type orient named IMU (prio: 0) with 344173 samples with 1 listeners (0x2b8ce40 )

  * The following channels are not part of this dataset: WAT_SPEED WAT_DIST RUDDER_ANGLE 

[jonasseglare]

Note that NMEA2000/c078be002fb00000 and NMEA2000/c0aa82002fb04000 both have priority 0, so the choice is sort of undefined. But one of them is not as good as the other.

[jonasseglare]

(removed wrong plots)

[jonasseglare]

Sorry, I plotted at the wrong timestamp... let me redo the plots at the right timestamp.

[jonasseglare]

These plots look better:

[jonasseglare]

The target speed plots in https://github.com/jpilet/anemomind/issues/772#issuecomment-231350241 where we manually selected one apparent wind source, looks better.

Conclusion: It seems like the Alinghi data has two different apparent wind sources with the same priority, among which one of them has a poorly calibrated angle. Our algorithm doesn't know which one to use, so it just uses one of them. But somehow it doesn't succeed in correcting for the bad angle and the bad AWA value is propagated into the true wind computation and then the VMG computation.

[jpilet]

Velocissima has only 1 navigation and it shows a upwind VMG of almost 0 instead of showing N/A.

[jonasseglare]

It is likely that the issues with the GPS filter that we solved may have improved things. Let's continue working on this issue. So what does the data look like before and after calibration?

[jonasseglare]

To help investigate this issue, I implemented two approaches for exporting a dispatcher to JSON:

jo-export-json2

that exports it in a compact format.

and

jo-export-json2-bak

that exports it in a less compact format.

[jonasseglare]

Started new fresh branch jo-issue-772

[jonasseglare]

This is what I suspect is the problem: We never output the filtered GPS speed/bearing.

[jonasseglare]

Well, now when I think about it, the calibration takes place before GPS filtering. Calibration, in turn, uses TrueWindEstimator to compute wind before and after a manoeuver. And if GPS speed data is missing, I would expect the that the TrueWindEstimator will use the last known outdated GPS speed values, meaning that the computed true wind will be wrong. And consequently, the calibration will be wrong. And consequently, target speed estimation will be wrong.

That is what I think. I will dig a bit more to see if this explanation is reasonable to what is going on.

[jonasseglare]

Also, for clarification, I verified that the calibration is wrong. Here is a snapshot of the calibrated datasets, just after simulation:

There is something fishy going on...

[jonasseglare]

As pointed out earlier, https://github.com/jpilet/anemomind/issues/772#issuecomment-231345321, we can get somewhat better results by cheating. But currently I don't see a "quick fix" to this issue.

[jpilet]

This is fixed I believe