Improvements to imosTiltVelocitySetQC & verification of Tilts - non-profiling Aquadopp

[BecCowley]

I don't think that this test should be applied to the aquadopp and aquadopp DW instruments which are current meters, not profiling ADCPs.

The latest version of Nortek's "The Comprehensive Manual for ADCPs" has a data processing section (1.1.9), which mostly is talking about profiling instrument data handling. I don't think it applies to the point current meters.

As for the tilt levels for the profiling ADCPs, they are different from the 20 and 30 degree cutoffs we have in the imos test:

The first step is to make sure that the instrument was oriented as intended. This is done by checking the time history of the tilt (Roll and Pitch) and heading sensor data. -If the tilt is large (greater than 10 degrees) then a post-processing option such as “map to vertical” is encouraged. -If the tilt is excessive (greater than 30 degrees), it may indicate current profiles are suspect to error. -If the tilt or heading measurements change greatly from profile to profile (buoy mounted), it is wise to ensure the instrument was configured as a buoy mount or with a suitable Compass update rate (check the deployment file or the header file).

The QC test might need some updating.

[ocehugo]

@BecCowley thanks for the heads up.

IMO, I think the test is still useful for the current meters because of possible knock-off events. I think removing them would do only harm since the current check is quite straight - anything above some pre-defined angle is marked. As always, the QC flags can be changed manually too.

As for the tilts updates, this needs wider discussion. At the moment we are calling good anything less than 15deg and until 50deg probably good. Hence, there is a potential that more data would be marked as bad, historical data range would be reduced, regressions, etc.

Lastly, I didn't fully understand the latter case - is profile to profile means a time interval measurement, ie time rate of change of the tilt!? If so, should be ok to do it as a new QC test, but we also need to discuss the acceptable range and check/implement the respective parser metadata information.

[BecCowley]

@ocehugo, the 'profile to profile' part is in bold to show that the text is referring to a profiling ADCP, not a current meter (ie, the Aquadopp and Aquadopp DW). This is why I don't think the tilt test should apply to these instrument types. For now, I will adjust the tilt cutoffs or unflag automatically flagged data from this test.

Happy to discuss this more in the future.

[petejan]

The aquadopp point current meter spec sheet has Maximum tilt: 30°, so for a aquadopp the limit should be 30 not 15 as for a RDI ADCP.

Is this a bit of a structural issue for the toolbox, the limits are different for different instruments measuring the same parameter?

From: Hugo Oliveira notifications@github.com Sent: Monday, 13 January 2020 1:30 PM To: aodn/imos-toolbox imos-toolbox@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: Re: [aodn/imos-toolbox] imosTiltVelocitySetQC should not apply to non-profiling Aquadopp (#635)

@BecCowleyhttps://github.com/BecCowley thanks for the heads up.

IMO, I think the test is still useful for the current meters because of possible knock-off events. I think removing them would do only harm since the current check is quite straight - anything above some pre-defined angle is marked. As always, the QC flags can be changed manually too.

As for the tilts updates, this needs wider discussion. At the moment we are calling good anything less than 15deg and until 50deg probably good. Hence, there is a potential that more data would be marked as bad, historical data range would be reduced, regressions, etc.

Lastly, I didn't fully understand the latter case - is profile to profile means a time interval measurement, ie time rate of change of the tilt!? If so, should be ok to do it as a new QC test, but we also need to discuss the acceptable range and check/implement the respective parser metadata information.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/aodn/imos-toolbox/issues/635?email_source=notifications&email_token=AAFQXTXHLAIMIQH5HL7V73TQ5PG3FA5CNFSM4KE7Y4WKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIXL2RY#issuecomment-573488455, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAFQXTQ6C44CB4PC32US223Q5PG3FANCNFSM4KE7Y4WA.

[ocehugo]

ATM, the limits are fixed for all nortek instruments at 20deg and 30deg (first/second thresholds). See automaticQC/imosTiltvelocitySetQC.txt.

I said 15/50 because that is the min/max tilt across all instruments.

By the above, we likely need to update the qc test to handle nortek particular models and update thresholds accordingly to the new docs.

[petejan]

Hmm, but Bec's point was point current meters need to be treated differently than profiling instruments.

Just another complication, we have RDIs where we have increased the range on them, so the limits need to be increased manually.

[BecCowley]

Here is a plot of the tilt from a current meter at approx 460m on the EAC500m mooring. If we cut off the tilt at 30 degrees, we will lose a lot of seemingly good data.

[ocehugo]

@BecCowley - great example. SO what should be the limit for these current meters then ? If any, how much the valid range tilt is depth-dependent? I imagine surface ones would attain higher variance/magnitude tilts.

I'm asking cause I wonder if I should implement a tilt check for CM as a completely different QC or just update the current QC to read instrument models and assign a higher value for CM/aquadoppdw/others.

I did a quick check, and I think the only ones using aquadoppdw is you guys, so I think we can decide over this channel easily.

[BecCowley]

@ocehugo, I will look at the other instruments and maybe we can come up with a range. It will be a few weeks before I can get to it.

[ocehugo]

@BecCowley , great - It would be a good thing to discuss at QC summit, iMO.

[BecCowley]

@ocehugo, I have reviewed the AQD tilts. The deeper the instrument, and further offshore, the less it tilts. The inshore ones (on the 500m mooring) tend to tilt up to 45degrees. I suggest that all this data is good and recommend that a 45degree cutoff would be the default.

[ocehugo]

OK - I will add a 45 default cutoff for aquadoppdw's. Should I also add a little dialogbox to set cutoffs when the test is selected? Just wondering if this is a kind of thing you would change frequently to flag less/more stuff.

[BecCowley]

I think it would be good to include a dialog box to allow us to adjust if needed. It would be nice to be able to plot the tilts for ADCP instruments from the Tools Menu so we can check it. Probably would need to be able to look at corresponding pressures and U/V values too. Maybe error velocities?

[ocehugo]

I think it would be good to include a dialog box to allow us to adjust if needed.

yeap, that was what I thought. IMO, if a parameter need change frequently, then a dialog (for this case) should be available. Frequent editing of config files to modify runtime behaviour is not optimal. Worse, it can lead to silent errors.

It would be nice to be able to plot the tilts for ADCP instruments from the Tools Menu so we can check it. Probably would need to be able to look at corresponding pressures and U/V values too. Maybe error velocities?

Let's not overflow the issue - May you create another with those !?

Requirements:

• [ ] Update default values for aquadoppdw.
• [ ] Test values are read correctly.
• [ ] Include dialog box for changing default values in imosTiltVelocitySetQC.
• [ ] Test batch mode using defaults (dialog box do not affect batch mode).

[BecCowley]

@ocehugo, Thanks. Let's get this implemented, then we can open another issue when testing.

[ocehugo]

Note for me: I'm leaving this open to track the issue of the dialog box and the testing.

I just created a new issue to only include the cutoff in the next release since it's a very small effort.

[ocehugo]

6ade4aabb58a3d2f823c470f1467f99de75c26d8 closed this issue.