Extended Attributes at variable level: `processing_level`, `comment`, `creator_name`, `project`, `date_modified`, `date_metadata_modified`

[DocOtak]

netCDF allows for a lot more information than exists in exchange files, with the CCHDO documentation metadata extraction project going, eventually we will need a place to put that metadata. For a while now, I have wanted to store the information contained in the "Bob Headers" in a more structured way. The following ACDD attributes, when pushed down from the global to variable level, should enable the creation of "Bob Headers": processing_level, comment, creator_name, project, date_modified, date_metadata_modified. Further examination of each of these:

• processing_level In ACDD the processing level is a freeform string. We should use this to indicate the following status that very roughly correspond to the satellite communities L0 though L4 processing levels :
  • collected - water was taken but not received
  • raw - used for CTD but not discrete
  • preliminary - data in the file that maybe has not had final calibration applied
  • final - data that is not expecting any more updates
  • product - we probably won't use this, but included since that is what L4 tends to be A controlled vocabulary of these should be searched for.
• comment Free text notes, usually these are very short for each parameter. This is the "notes" part of the Bob Headers
• creator_name This is the PI for the parameter in question, we should use array of strings for multiple PIs in our at rest data files. This is the "who" part of the Bob Headers. There is also a creator_url attribute that we might consider storing ORCiDs in.
• project We need a way to tie multiple variables with the same PI/status together, e.g. nutrients are usually 3~5 variables. In the ACDD docs, a program (GO-SHIP) is made up of multiple projects (Total Carbon, pH, Nutrients, CTD, etc..). Variables that have the same project value would be grouped into the "includes" list in the Bob Headers, the comment and creator_name would need to be the same to avoid ambiguity. There probably is not a single controlled vocabulary for these project names, they would also likely benefit from some coordination with GO-SHIP.
• date_modified If the data itself is changed, this would be updated to be the date it was changed in the data file. The merge_fq accessor already updates this.
• date_metadata_modified If only the metadata were modified, this attribute would be updated to the date the change was done. The merge_fq accessor already updates this if the print format is different.

The only non standard ACDD usage of the above are being at the variable level rather than global, and the possible use of arrays of strings. We could define combining rules to put all this information in the global attributes that fully conform to ACDD, but this would likely be one way (update the globals from variables, not the other way around). For example: the global date_modified would be set to the most recent date seen from all the variables that also have date_modified.

Things this might make possible:

• Getting a list of updated files since (or even between/before) a certain date could be done at a per variable level by examining the date_modified attribute. We can even exclude simple metadata updates that didn't change the values used in science.
• Find all the preliminary data or exclude preliminary data from a result set.
• Know who has not turned in their data yet by examining the processing_level attribute for "collected" and the creator_name attribute. This can also be done for bottle data with flag 1.

[DocOtak]

Some resources for processing_level: https://wiki.pangaea.de/wiki/Processing_levels only the Argo data status and GCMD "Levels" are in NVS.

[DocOtak]

More resources, we should absolutely support ORCiDs in the creator URL, we could then populate or validate against the ORCiD record using content negotiation to get a json-ld schema.org/Person record: see https://github.com/ORCID/ORCID-Source/blob/main/CONTENT_NEGOTIATION.md

[cberys]

We could define combining rules to put all this information in the global attributes that fully conform to ACDD, but this would likely be one way (update the globals from variables, not the other way around). For example: the global date_modified would be set to the most recent date seen from all the variables that also have date_modified.

Minor comment that the date_metadata_modified might be modified later in time at the global level than at the variable level - ie that one might not be just one-way variable to global.

[Savannah-Lewis]

Processing level: I love having this, since this is a question I’ve had to field several times already. Assuming our submission doesn’t change, would we automatically assume everything is preliminary unless otherwise stated? Any reason not to implement the PANGAEA (or other existing processing levels such as EOSDIS) schema rather than retaining our own? Creator_url: I would definitely be in favor of adding this attribute to store ORCiDs. Anecdotally, I noticed that BCO-DMO requires data submitters to login via ORCid. Project: This seems really important, but also already a cluster in GO-SHIP i.e. discussed with Alison today our definition of noble gases and how that changes based on what the PI measures i.e. sometimes only He and Tr, other times includes more including Ar, Ne, etc. Would be a good idea to get started on piecing these things apart now and I think it will be helpful in a multitude of ways, not just our metadata model! The date_modified and date_metadata_modified seem super useful. I think adding the ‘lineage statement’ might be helpful as well as a way to store any doi’s that are submitted with the dataset.