search

WHOIGit / nes-lter-ims

NES-LTER information system components
MIT License
5 stars 0 forks source link

first pass QC of underway data #29

Open ecrockford opened 3 years ago

ecrockford commented 3 years ago

Establish some simple filters to pass over underway data for rough QC prior to uploading to the API.

Output NA for skipped data. When looking at max/min values use some sort of rolling average or rolling median? If 1 parameter is bad, do we mask out all parameters? Do we group TSG together and any others separate?

Possible filters:

joefutrelle commented 3 years ago

Locate column header for flow meter for different cruises.

sbeaulieu commented 3 years ago

Column header for salinity: SBE45S. Column header for flow meter: FLOW. A challenge with flow meter is that the average value (in ml/s) differs between cruises (e.g., ~50, ~80, ~130 depending on cruise). I saw zero values in several, and NAN values in some. Let's discuss with Taylor re: thresholds for minimum salinity and flow meter values.

ecrockford commented 3 years ago

Yes thanks for pointing that out Stace. A while ago Joe and I discussed this. We thought maybe a running average with significant deviation from that average could work? Something along those lines. I've heard in passing from the ship's techs that the flow meter isn't super great so we should take values with a grain of salt. Basically, looking for deviations from consistency.

sbeaulieu commented 3 years ago

Noting that in today's Zoom we discussed providing a quality flag column for each of these: salinity, flow meter, and fluorometer. We also discussed providing a comments column that would be auto-populated to alert the end user that a flag(s) was applied.

joefutrelle commented 3 years ago

Endeavor data does not use the column headers specified here, so the API will need to skip adding quality flags for that data.

Once column names are regularized, this will not require per-cruise or per-vessel configurations (although regularization will).

joefutrelle commented 3 years ago

Related to #30

sbeaulieu commented 3 years ago

develop QA/QC checks using Armstrong data and evaluate with Taylor

joefutrelle commented 1 year ago

From the original description:

Establish some simple filters to pass over underway data for rough QC prior to uploading to the API.

If these tools are not part of the REST API then we should consider tracking this work elsewhere. I'm leaving the issue open for now but addressing it may not touch this codebase.