New Task: For raw ATN profiling datasets, finalize and implement automated NCEI data archive pathway

MathewBiddle commented 6 months ago

Who is requesting this?

@ioos/marine-life, and the ATN community

What is being requested?

[ ] Establish netCDF template in coordination with ATN DC (@mmckinzie), IOOS National Glider Data Assembly Center (POC: @kbailey-noaa) and IOOS DMAC (POC: @mwengren ). See https://github.com/ioos/marine_life_data_network/issues/57
[ ] finalize agreement of nc metadata attributes with NCEI
[ ] create metadata and format file template for NCEI review
[ ] test load template to NCEI ftp site/ WAF
[ ] begin operationally submitting non-embargoed deployments to NCEI
[ ] create automated service within ATN DAC Reg App for grouping deployments into synthetic 'projects' and connecting metadata returned from NCEI for access by PIs
[ ] Ensure archived data, dataset citations and DOIs are integrated into ATN DAC portal catalog
[ ] Update documentation at https://ioos.github.io/ioos-atn-data to reflect current netCDF implementation.
[ ] Document issues and proposals for future versions at https://github.com/ioos/ioos-atn-data/issues

What is the requested deadline and why?

Properly archive ATN profiling data for public access

What is the current status quo (i.e., what happens if this does not get done)?

ATN profiling data are not archived at NCEI and PARR requirements are not met.

What indicates this is done (i.e., how do we know this is complete)?

[ ] NCEI automatically pulls data from ATN on identified frequency
[ ] IOOS publishes a news story about it.
[ ] Add “gold standard” example files to https://github.com/ioos/erddap-gold-standard/tree/main/datasets and make them accessible via https://standards.sensors.ioos.us/erddap/index.html

Provide a description or any other important information.

MathewBiddle commented 6 months ago

What does "and archive administrative tools" mean in the title?

mmckinzie commented 6 months ago

I am also not sure what 'archive administrative tools' means...maybe remove from title?

MathewBiddle commented 6 months ago

😆 okay if we both don't know what it means, let's remove it.

conniekot commented 6 days ago

@iwensu0313 Please update this issue with progress and any documentation or links to relevant code - thanks!

iwensu0313 commented 2 days ago

Creating a NCEI archival template for profile data will start in March 2025 in the new contract. Active NCEI archival pipeline work is captured here https://github.com/ioos/marine_life_data_network/issues/41 and related to trajectory data - we will be able to continue archiving trajectory data to NCEI as they come in. I marked this ticket status as TO DO but let me know if BLOCKED is better

ioos / marine_life_data_network