NCEI-archiving
Code, documentation and issue tracking for NANOOS NCEI archiving
NCEI Disclaimer
The recommendations provided by NCEI staff on the issues and documents in this repository are primarily for the NANOOS archival process at NCEI. While some of the information might be useful and applicable to other data sets, these are not blanket statements for all of NCEI's archival procedures.
NANOOS OHSU CMOP NCEI Archiving
- Monthly netcdf files for archiving (pulled by NCEI): http://data.nanoos.org/ncei/ohsucmop/
- NCEI Submission Agreement (SA). The SA has extensive technical details about the archiving procedures. Some relevant fragments from the SA, with small additions: "Submission Information Packages (SIP) will be organized into 'bags'. Each 'bag' will contain data, metadata, and manifest files which fully document the files intended to be submitted. The 'bags' will be folders on http://data.nanoos.org/ncei/ohsucmop/ which correspond to the name of the platform. E.g. abpoa/, riverrad/, saturn01/, etc. Within the station folder (or 'bag') there will be four standard files with the following names: bag-into.txt, bagit.txt, manifest-sha256.txt, and tagmanifest-sha256.txt as well as a data/ directory which will contain folders for all of the netCDF files to be submitted. Each of the folders within the data/ directory represent an instrument/instrument deployment.... NCEI will organize the Archival Information Packages (AIP) by station. Each time a new station arrives, a new AIP will be generated."
- Sample archived OHSU CMOP packages on NCEI servers
Other NANOOS or NANOOS-related data archived on NCEI
In addition to the formal data archiving process we are setting up, which will follow the IOOS-NCEI "gold standard" and will be done in coordination with NANOOS DMAC, other data packages may have been archived already at NCEI with a keyword linkage or acknowledgment to NANOOS. This "NANOOS" search to the NCEI Geoportal returns all current NCEI records matching a NANOOS string.
- Currently (2016-12-7), there is one record there: Dissolved inorganic carbon, total alkalinity, phosphate, silicate, and other variables collected from profile and discrete sample observations using CTD, Niskin bottle and other instruments in the northwest coast of the United States near the Cha Ba mooring off La Push, Washington from 2011-05-22 to 2014-10-24 (NCEI Accession 0145160). It looks like it was submitted March 2016, but may have been updated in early Nov 2016. Data files are in Excel and csv formats ("Distribution Formats: Originator data format").
- Note that Glider DAC data are not returned! Neither CMOP gliders, nor the CeNCOOS-NANOOS glider, both of which have been submitted to the Glider DAC. (Emilio has discussed this issue with IOOS, NCEI and Glider DAC teams since June 2016; Glider DAC technical options/solutions were discussed at https://github.com/ioos/ioosngdac/issues/105, but it's not clear a solution has been implemented as of 12/9/2016).
- However, I did find the CMOP Glider DAC records by searching for "CMOP": the search returned 16 records; here's one of the records. But none of the records have any description words or keywords listing NANOOS. I'm sure we had such affiliation somewhere in the ACDD or other information we provided to the Glider DAC (hence the ability to find NANOOS-affiliated gliders in the Glider DAC ERDDAP), so I assume the affiliation was lost in the transfer from the DAC to NCEI.
- I also found one of the Trinidad Head glider records, which appears to have been submitted by OSU directly (not the Glider DAC); here's the record obtained by searching on
"trinidad" AND "glider"; a "glider" and geographical search didn't return any other records. This Trinidad record (Accession# 0125046) is the same one Anatoli told me about in May 2015.
- The complete list of NANOOS-affiliated glider datasets (deployments) submitted to the Glider DAC are listed on the DAC ERDDAP using "NANOOS" as a search string. Currently there are 16 CMOP ("corie") and 5 OSU / Trinidad ("mbari") deployments. At the Glider DAC v2 site (not ERDDAP), we can get at these listings by "provider" for corie and mbari