New Task: ATN DAC QC protocols and modified Argo QARTOD code, trajectory and profile are documented and shared via GitHub

ioos / marine_life_data_network

Planning efforts relevant to IOOS Marine Life and IOOS DMAC.

https://ioos.github.io/marine_life_data_network/

Other

1 stars 2 forks source link

New Task: ATN DAC QC protocols and modified Argo QARTOD code, trajectory and profile are documented and shared via GitHub #14

Open MathewBiddle opened 5 months ago

MathewBiddle commented 5 months ago

Who is requesting this?

@ioos/marine-life

What is being requested?

ATN DAC QC protocols and modified Argo QARTOD code, trajectory and profile are documented and shared via GitHub.

If this is something that builds on ioos_qc (https://github.com/ioos/ioos_qc), enhancements should be proposed to that package.

What is the requested deadline and why?

2024-07-01

What is the current status quo (i.e., what happens if this does not get done)?

The Marine Life team and animal telemetry community will not fully understand what the QC protocols are for ATN data.

Provide a description or any other important information.

What indicates this is done?

Multiple actions could indicate this activity is done. The options are listed below but depend on what path the effort takes. These can be "either/or" but also "and" statements.

[ ] A pull request is submitted to ioos_qc with the enhancements for modified Argo QARTOD
[ ] The https://github.com/ioos/ioos-atn-data/ repository is updated to include a markdown describing QA/QC and modified Argo QARTOD
[ ] The source code for modifications are published on GitHub at https://github.com/ioos/ioos-atn-data/, or other appropriate repository.

mmckinzie commented 5 months ago

Consider updating task title to: Document and share ATN DAC QC protocols - satellite telemetry data What is being requested?

(Axiom/ML) Documentation about ATN DAC QC protocols, similar to: Argo_CTD&TRAJ_qcmanual_version3.3.docx (noaa.gov)
List of QARTOD tests run for both profile (CTD/TD) and trajectory data
Include test parameters, ranges and limits, particularly if they deviate from the Argo manual and/or are specific to the ATN (satellite telemetry data); include description of [flags] (QARTOD Manual for Real-Time Oceanographic Data Quality Control Flags (noaa.gov))
Include translation of CTD QARTOD flags to GTSPP Flags for insertion into BUFR messages; GTS flags: doc_num.php (wmo.int) (implementing new BUFR/GTS QC protocols and embedding flags into messages will need to be a separate issue/task- see issue #17)

Deadline - Consider updating to the end of ML YR2 SOW - November 30, 2024

MathewBiddle commented 3 months ago

@mmckinzie it looks like ioos_qc already had an argo test. See:

Please coordinate the ATN team with the ioos_qc package and submit changes upstream to ioos_qc. If you need help, please reach out.

mmckinzie commented 2 months ago

@MathewBiddle Yes, the Argo tests/code are well documented but modifications that were made to this base code and implemented by Axiom for the ATN DAC to make it applicable for animal borne platforms are not clearly documented anywhere. i.e., list of specific tests run for trajectory vs profile data, applied thresholds (different from those used for Argo floats), and modifications to base tests, e.g., date test includes not only impossible date_test but also whether or not location is within PI reported deployment window.

mmckinzie commented 2 months ago

Ian Jonsen is working on developing/updating IMOS near r/t QC protocols for animal borne profile data and corresponding trajectory files. This work is part of a GOOS Bio-Eco panel pilot project with AniBOS to serve behavioral data to OBIS/GBIF. Jon Pye will also be helping with this effort. Work to begin week of July 15th, 2024. This work will likely utilize elements of the remora package and OTN DwC templates/OBIS thematic node. This effort will be discussed at our upcoming AniBOS data committee meeting later this month.

I'll be curious to see how these IMOS/OTN protocols and QC procedures differ from what the ATN DAC is presently doing using QARTOD.

These new r/t QC tests will be distinct and differ from existing MEOP protocols used to develop delay-mode data products, available here: https://osmc.noaa.gov/erddap/tabledap/MEOP_profiles.html

MathewBiddle commented 2 months ago

Thanks. What I read from that is there are three tasks for this:

Document the ATN modified Argo tests publicly
Compare the ATN modified Argo tests with the IMOS r/t QC protocols.
Do something with that comparison? Either adjust ATN's implementation or document the differences, I assume.

mmckinzie commented 2 months ago

Yes, I think that covers it.

Issue #47 also mentions creating a jupyter notebook with documentation and making it publicly available. Not sure if we want to merge those two issues or keep as separate although related tasks.

mmckinzie commented 1 month ago

IMOS best practices manual for SMRU CTD data including instrument calibration, near real-time and delayed mode data QA/QC is now available.

MathewBiddle commented 1 month ago

Wooohoo!

Do you know if it will be published in Ocean Best Practices? I would prefer to link to a DOI instead of a Google drive file.

mmckinzie commented 1 month ago

Yes, looks like it is already published on Ocean Best Practices - here.

DOI: 10.26198/ev75-0j83

Jonsen, I., McMahon, C. and Harcourt, R. (2024) Best Practice Manual for SMRU CTD Satellite Relay Data Loggers: Instrument Calibration, Near Real-Time and Delayed Mode Data QA/QC. Version 1.0. Hobart, Australia, Integrated Marine Observing System, 24pp. DOI: https://www.doi.org/10.26198/ev75-0j83).

edit: edited to fix urls

MathewBiddle commented 3 weeks ago

How is what ATN is doing different (or the same) as https://github.com/ianjonsen/ArgosQC ?

mmckinzie commented 3 weeks ago

we only run the diag and ctd files through QARTOD

MathewBiddle commented 3 weeks ago

Has anyone started a repo for this effort? Should we be contributing back to the ArgosQC package?