New Task: Release v1.0 of ATN DAC netCDF trajectory template

[MathewBiddle]

Who is requesting this?

@ioos/marine-life and NCEI

What is being requested?

ATN establishes a netCDF template for trajectory data. This includes:

• [ ] Draft the netCDF template following the contributing guidelines published from https://github.com/ioos/ioos-atn-data/issues/79
• [ ] Present to IOOS DMAC for review and approval.
• [ ] Present netCDF template to NCEI for approval (maybe needs NCEI Dataset Readiness Review (DSRR) board approval?)

What is the requested deadline and why?

2024-12-31

What is the current status quo (i.e., what happens if this does not get done)?

NCEI will not archive data that aren't fully documented. The telemetry community will splinter into multiple file templates for similar observing methods.

What indicates this is done (i.e., how do we know this is complete)?

• [ ] A successfully merged pull request to the jekyll base webpage at https://ioos.github.io/ioos-atn-data/ to document v1.0 iteration of the profile.

Provide a description or any other important information.

xref:

• https://github.com/ioos/marine_life_data_network/issues/41

[mmckinzie]

Considering splitting into two separate issues? – satellite telemetry templates: trajectory and profile or possibly create overarching standardized NCEI netCDF submission milestone or similar, as there will eventually be additional satellite telemetry file types beyond the location (trajectory) and the CTD-F/CTD/TD (profile) files that will need to get formalized as well as generation of templates/files for other telemetry types, e.g., acoustic, multisensor, archival, which will eventually need to get linked to the NCEI ATN collections.

NOTE: Multisensor data effort, focused on DTags is included under a supplemental ML SOW – hope to resume work/effort on integrating these data into the ATN DAC by FY25.

What is being requested?

(ML) ATN DAC trajectory template:

• Develop template - done
• Ensure template is approved by NCEI Dataset Readiness Review (DSRR) board - done
• Publish finalized template and gold standard example and related documentation on ioos website, integrate into ioos compliance checker, link to atn data github – mostly done, needs some revision

(ML) ATN DAC profile template:

• Develop template - in progress
• Ensure template is approved by NCEI Dataset Readiness Review (DSRR) board
• Publish finalized template and gold standard example and related documentation on ioos website, integrate into ioos compliance checker, link to atn data github

(Axiom) Operationalize and automate pipeline from ATN DAC to NCEI WAF, production folder – for both trajectory and profile (finalize trajectory first):

• Successful generation of standardized satellite telemetry trajectory netCDF file - in progress
• Successful generation of standardized satellite telemetry profile netCDF file
• Ability to create these files for all auto-ingested satellite telemetry data from Argos based platforms (w/ or w/o Fastloc or equivalent GPS quality positions) regardless of tag manufacturer and/or tag type
• Automate file generation, from ADR to WAF, push newly flagged deployments to WAF production folder daily
• Integrate ability to list/track NCEI accession numbers in the ADR and mechanism for ATN DC to trigger file generation/release to WAF - done

(Axiom) other related tasks:

• Develop ability to convert manually uploaded data to appropriate netCDF template, link file to ADR
• Link archived files to project’s dataset pages in the ATN portal, embed access link to NCEI using DOI/accession number(s)
• Release/publish and link code used to covert auto-ingested raw satellite telemetry data into ATNs standardized netCDF profiles and/or trajectory file

(ML) other related tasks:

• Finalize ATN collection-level metadata record – see related issue ticket #4

Deadlines: finalize satellite telemetry trajectory template, update IOOS website content and ensure files are successfully generated by Axiom and able to be pushed to WAF production folder – July 1, 2024; Operationalize/Automate NCEI pipeline for trajectory files – September 1, 2024; Release/publish trajectory template code – September 1, 2024; Link NCEI files to ATN portal – November 30, 2024; Profile template development and IOOS website documentation – November 30, 2024; Successfully generate profiles files and serve to NCEI – add to YR3 ML SOW

[MathewBiddle]

@mmckinzie I agree this should be split into two issues. Thank you for the additional details.

I think some of your comments can stay here and some can be moved to the pipeline development tickets that have been xref'ed above.

From your comments it sounds like we need to adjust this ticket and create a new one so we have two tracking mechanisms (documentation of profile and trajectory templates). The thinking is these tickets are strictly for documenting and publishing the netCDF template. Not necessarily for developing the archival pipeline or cyberinfrastructure to implement said pipeline. So, I propose we adjust the title of this issue and create a second issue as follows:

• [ ] Publish ATN DAC trajectory netCDF template
• [ ] Publish ATN DAC profile netCDF template

Both of those will be intrinsically linked to the "develop the archive pipeline" in #41 and #42. The first step in those two issues is to define the netCDF structure, so we can make this piece a dependency.

As new formats/pipelines come up (e.g., acoustic, multisensor, archival) we will create issues to manage the development of the template and the establishment of a pipeline to NCEI. For now, let's focus on the trajectory and profile parts.

Do you mind creating a new issue for this?

NOTE: Multisensor data effort, focused on DTags is included under a supplemental ML SOW – hope to resume work/effort on integrating these data into the ATN DAC by FY25.

The issue should be concise and explicitly about the task at hand. @laurabrenskelle wrote a contributing guide to help frame the scope of a new issue. If you have questions don't hesitate to reach out!

[MathewBiddle]

Title

Publish ATN DAC trajectory netCDF template

Who is requesting this?

@ioos/marine-life

What is being requested?

ATN DAC trajectory and profile standardized templates/NCEI netCDF file format documentation and code

• [ ] Draft the netCDF template following the contributing guidelines published from https://github.com/ioos/ioos-atn-data/issues/79
• [ ] Present netCDF template to NCEI for approval (maybe needs NCEI Dataset Readiness Review (DSRR) board approval?)

What is the requested deadline and why?

2024-07-01

What is the current status quo (i.e., what happens if this does not get done)?

NCEI will not archive data that aren't fully documented. The satellite telemetry community will splinter into multiple file templates for similar observing methods.

What indicates this is done?

• [ ] A successfully merged pull request to the jekyll base webpage at https://ioos.github.io/ioos-atn-data/ to document the current standard.

Provide a description or any other important information.

[MathewBiddle]

I am updating the description of this issue to adjust:

• That this topic should be two issues. One for the trajectory template and one for profile template.
• Clearly articulate what is being done.
• This issue will now be for tracking progress on the trajectory template development.

[MathewBiddle]

@mmckinzie @laurabrenskelle I'm wondering if it would be cleaner if we closed this issue and opened a new one without all the changes?

[MathewBiddle]

Website needs to be updated to align with changes to the template recently.

Website should be updated to reference to a production file.

@mmckinzie to send final file once complete