search

HakaiInstitute / hakai-datasets

Hakai Datasets that are going into https://catalogue.hakai.org/erddap/
0 stars 0 forks source link

Update: Hakai Flag Convention Standard #38

Closed JessyBarrette closed 2 years ago

JessyBarrette commented 3 years ago

Hakai Flag Convention Standard

This is a more a broader issue to clearly identify the flagging convention used and accepted within both the sensor network and EIMS. Once agreed this work will get integrated within the Jupyter Notebook data QCing tools in development.

I have been told by @jdelbel that not all the terms listed below are accepted within EIMS. @fostermh can you confirm that?

Hakai Flag Convetion

Code Name Comments/description Sensor Network EIMS QARTOD Mapping
AV Accepted value Has been reviewed and looks good GOOD (1)
SVC Suspicious value - caution Value appears to be suspect, use with caution SUSPECT (3)
SVD Suspicious value - reject Value is clearly suspect, recommend discarding FAIL (4)
EV Estimated value Value has been estimated
NA Not available No value available MISSING (9)
MV Missing value No measured value available because of equipment failure, etc. UNKNOWN (2)
LB Low battery Sensor battery dropped below a threshold
CD Calibration due Sensor needs to be sent back to the manufacturer for calibration
CE Calibration expired Value was collected with a sensor that is past due for calibration
IC Invalid chronology One or more non‐sequential date/time values
PV Persistent value Repeated value for an extended period
AR Above range Value above a specified upper limit
BR Below range Value below a specified lower limit
SE Slope exceedance Value much greater than the previous value, resulting in an unrealistic slope
SI Spatial inconsistency Value greatly differed from values collected from nearby sensors
II Internal inconsistency Value was inconsistent with another related measurement
BDL Below detection limit Value was below the established detection limit of the sensor
ADL Above detection limit Value was above the established detection limit of the sensor

Source: https://docs.google.com/spreadsheets/d/1NZcwn7zPZ-98za4HpxQH705uw3tsEgKUwQhkPl0UiS8/edit?usp=sharing

jdelbel commented 3 years ago

That's helpful, thanks.

So ADL could be applied to the data that fell above the sensor specified max?

On Fri, 21 May 2021 at 13:45, Jessy Barrette @.***> wrote:

Hakai Flag Convention Standard

This is a more a broader issue to clearly identify the flagging convention used and accepted within both the sensor network and EIMS. Once agreed this work will get integrated within the Jupyter Notebook data QCing tools in development.

I have been told by @jdelbel https://github.com/jdelbel that not all the terms listed below are accepted within EIMS. @fostermh https://github.com/fostermh can you confirm that? Hakai Flag Convetion Code Name Comments/description Sensor Network EIMS QARTOD Mapping AV Accepted value Has been reviewed and looks good GOOD (1) SVC Suspicious value - caution Value appears to be suspect, use with caution SUSPECT (3) SVD Suspicious value - reject Value is clearly suspect, recommend discarding FAIL (4) EV Estimated value Value has been estimated NA Not available No value available MISSING (9) MV Missing value No measured value available because of equipment failure, etc. UNKNOWN (2) LB Low battery Sensor battery dropped below a threshold CD Calibration due Sensor needs to be sent back to the manufacturer for calibration CE Calibration expired Value was collected with a sensor that is past due for calibration IC Invalid chronology One or more non‐sequential date/time values PV Persistent value Repeated value for an extended period AR Above range Value above a specified upper limit BR Below range Value below a specified lower limit SE Slope exceedance Value much greater than the previous value, resulting in an unrealistic slope SI Spatial inconsistency Value greatly differed from values collected from nearby sensors II Internal inconsistency Value was inconsistent with another related measurement BDL Below detection limit Value was below the established detection limit of the sensor ADL Above detection limit Value was above the established detection limit of the sensor

Source: https://docs.google.com/spreadsheets/d/1NZcwn7zPZ-98za4HpxQH705uw3tsEgKUwQhkPl0UiS8/edit?usp=sharing

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/HakaiInstitute/hakai-datasets/issues/38, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJPW44Q2IJOXFXIZRHOGZ7LTO3A67ANCNFSM45J2WXIQ .

-- Justin Del Bel Belluz, MSc. Research Technician - Bio-Optical Oceanography Hakai Institute 100 - 1002 Wharf Street Victoria, BC Canada V8W 1T4 www.hakai.org

JessyBarrette commented 3 years ago

@jdelbel If it is an acceptable flag within EIMS yes! This wouldn't show up within #32 since we're omitting the flags. We can however very easily filter out that data or keep depending on what you consider good.