search

HakaiInstitute / hakai-ctd-qc

Series of tests applied to the Hakai CTD profile data based on the QARTOD tests and other Hakai Specific ones.
0 stars 0 forks source link

Level 2 Flag convention to use within the CTD QC #6

Closed JessyBarrette closed 2 years ago

JessyBarrette commented 2 years ago

I just want to make sure that we're all on the same page regarding which convention should be use regarding the Level2 flags.

As of now, a level 1 flag column will be use and will follow the QARTOD convention (1,2,3,4,9)

The level 2 flag convention as of now follow (QARTOD: description, ";' delimited). ex:

3: density_inversion_test; 4: gross_range_test

The main question is should we use a QARTOD convention for the initial flag or use the equivalent within the Hakai convention.

fostermh commented 2 years ago

I was thinking we would use the equivalent in the hakai convention

raytula commented 2 years ago

Hey @JessyBarrette let's discuss when we meet a bit later today, as I'm not certain what you are asking. With the sensor network, the intention of the level 2 flag is to make it feasible for anyone using the data to gain a better understanding of why the value was assigned the current level 1 flag and, in some cases, see what values were clipped.

Here is an example, snapshot below, where the level 2 flag includes the reason for the flag (BR) the specific failure (WaterTemp60>12) and what the values were before they were clipped. It also includes 'Auto' to indicate that the QC results were determined automatically.

This is just how it works for sensor network data, I believe EIMS data does things a bit differently. I seem to recall that Axiom uses (or used) the level 2 flag to implement a type of 'bitmask' to indicate which QARTOD tests were run---but I may be wrong about that.

image

JessyBarrette commented 2 years ago

Ok after dicussion we're gonna map the level2 qartod flags to Hakai standard:

AV = GOOD (1)
NA = UNKNOWN (2)
SVC = SUSPECT (3)
SVD = FAIL (4)
MV = MISSING (9)

I'm not sure if I'll be able to include the thresholds values for each tests in there. I'll see what I can do.

raytula commented 2 years ago

That mapping makes sense to me.

JessyBarrette commented 2 years ago

I haven't added the thresholds within the level 2 flags. This would add a some complications and would potentially make the code more restrictive (maybe not, definitely slower). Since it clearly states which test failed it is fairly easy to know the threshold by referencing the QARTOD test configuration here: https://github.com/HakaiInstitute/hakai-profile-qaqc/blob/main/hakai_profile_qc/config/hakai_ctd_profile_qartod_test_config.json