Provision of netCDF files for the netCDF header examples shown in the Conventions?

[atmodatcode]

Title

Requesting access to the netCDF files that are shown as netCDF header examples in the Conventions document

Moderator

@user

Moderator Status Review [last updated: YYYY-MM-DD]

Brief comment on current status, update periodically

Requirement Summary

When creating CF-compliant netCDF files, users frequently not only need an idea how the netCDF header should look like, but also how the data variables need to be stored. So, a GitHub folder with some small sample netCDF files would be extremely beneficial.

Technical Proposal Summary

Brief proposal overview

Benefits

Users trying to create CF-compliant netCDF files will have good netCDF example files for guidance.

Status Quo

Discussion of the current state CF and other standards.

Associated pull request

Link to associated pull request, if present.

Detailed Proposal

As a regular reader and user of the CF Conventions, I frequently need example netCDF files that demonstrate the implementation of specific CF Convention aspects. As of now, the CF Conventions shows example netCDF headers, but to my knowledge, the corresponding CF-compliant netCDF files are not provided. With CF-compliant, I mean netCDF files that pass state-of-the-art CF-checkers such as IOOS Compliance Checker. Also for workshops that cover aspects of the CF conventions, the availability of sample netCDF files that relate to the examples shown in the CF conventions document would be beneficial. Using them, users can e.g. try to read them in, modify them and check if they are still CF compliant. I would very much appreciate if a folder with sample netCDF files could be made accessible. Thanks a lot.

[JonathanGregory]

Dear @atmodatcode

Thanks for your proposal. In fact the rules for changing the conventions require a test file to be produced for each change, but that rule has never been followed, and no test files have ever been presented. As far as I know, the examples in the document exist only "on paper", and no netCDF files have been produced. Your request for the example files is reasonable and our rules anticipate it, but unfortunately we haven't got any to give you!

I suppose that one obstacle to producing files when changes are proposed is that the proposal typically affects only one feature, whereas an example file needs a lot more contents that aren't relevant to the change. A strict requirement to create such a file might delay or prevent changes being implemented, because of the extra work.

In any case, that wouldn't address the request for files to illustrate the existing conventions. It would be useful to know what people think about whether and how such files could be provided.

Best wishes

Jonathan

[MathewBiddle]

Both of these examples are using CF-1.6 with ACDD and other added flair:

• NOAA's IOOS examples
• NOAA's NCEI examples

I'm sure other groups have examples too.

It would be nice if there was an authoritative source from the CF community to reference.

[marqh]

i think that there are some practical aspects to this request that could be explored, which might make it feasible to deliver over time

• netCDF enables the building of netCDF files from partial CDL, in particular array payloads not specified in the CDL are encoded as missing data

this would enable valid netCDF files to be encoded as CDL in the source tree, as plain text files, with any unnecessary array data left blank

such CDL files can be encoded into netCDF using the ncgen tool, enabling them to be used as example files

asciidoc's include directive enables partial inclusion of lines from another plain text file https://docs.asciidoctor.org/asciidoc/latest/directives/include/

These facets could enable the construction of an example file management process where:

1. an example CDL plain text file is stored in the cf-conventions source tree
2. the relevant snippets of that file are then used to populate the precise section of the conventions document that is required (delivering like for like comparison with current doc)
3. an automated action builds the CDL into NC as a pre-commit Pull Request test - assuring validity of CDL
4. an automated action builds the CDL into NC then validates with respect to CF conventions as a pre-commit Pull Request - assuring CF compliance

this work flow could fit quite neatly with the management process in place on this repository

if this is of interest, then we could try to prepare an example, with work flow, for consideration

i think that each example in the conventions doc would require some analysis to produce a valid CF CDL full file to subset to maintain content integrity. So, we could investigate the work flow first with one example, then consider an ongoing activity to update (editorially) example snippets of CDL through the conventions document over time

hth marqh

[JonathanGregory]

In website issue #257 @jypeter has raised a similar point, as follows:

A quick question (but answering may take some time), following a coffee discussion I have with @oliviermarti this morning : does the CF convention site provide netCDF test/example files?

• There is no mention of test files in the Quick Links section on the home page
• I have searched test in the Latest release page of the documentation: there are 19 occurrences of test, but nothing about test files
• There are more occurrences of example and sample, but the few I have checked did not seem to be example files or sample files
• I have just noticed that there is no mention of CMIP or ESGF in the CF convention, in case someone answers here that people could just grab files from ESGF

The document does provide some example text dumps of netCDF files headers, but having easy access to actual test files could be useful for:

• end user who want to look at actual files following CF, and play with them
• developers who want to check if their tools can handle the CF convention (at least partly)

I have recently started another thread about test files

To avoid duplication, I suggest that the discussion of that issue continues here instead.

Jonathan

[taylor13]

I would note that (purportedly) CF-compliant sample files are available from CMIP6 at, for example, https://esgf-node.llnl.gov/search/cmip6/ . Not sure if that's what you had in mind though.

[atmodatcode]

….I was thinking of CF-compliant sample files for all kinds of more „fancy“ CF features. CMIP6 files are relatively „simple“ (regular grid, regular time series, no auxiliary coordinate variables (I think), no featureType etc). Having some correct CF-netCDF files for e.g. H.2.4. Contiguous ragged array representation of time series H.2.5. Indexed ragged array representation of time series H.3.2. Incomplete multidimensional array representation of profiles 5.12. British National Grid + Newlyn Datum in CRS WKT format would be extremely valuable as templates.

thanks Angelika

Am 10.09.2022 um 01:19 schrieb taylor13 @.***>:

I would note that (purportedly) CF-compliant sample files are available from CMIP6 at, for example, https://esgf-node.llnl.gov/search/cmip6/ https://esgf-node.llnl.gov/search/cmip6/ . Not sure if that's what you had in mind though.

— Reply to this email directly, view it on GitHub https://github.com/cf-convention/cf-conventions/issues/348#issuecomment-1242551139, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATBT566CMRSBMQZBKALTX2LV5PAYRANCNFSM5LTEO6AQ. You are receiving this because you were mentioned.

[taylor13]

Indeed, those features aren't needed for CMIP data (although we do have some data on grids other than latxlon, we have some parametric vertical coordinates, and we have some auxiliary scalar coordinates). I agree that files with the more complex advanced features you list would be especially helpful.

[jypeter]

Sample files can't hurt (even for non-fancy CF features).

As a CMIP data user, I know I can get some files from ESGF, but there is no mention (yet) of ESGF on the CF website (https://github.com/cf-convention/cf-convention.github.io/issues/256)

That's one of the reasons why referencing actual projects that use CF could be useful

[Armin-RS]

Similar requirements are included in this discussion from September 2024: https://github.com/orgs/cf-convention/discussions/355

[lhmarsden]

The other discussion https://github.com/orgs/cf-convention/discussions/355 will continue here.

To summarise what was discussed:

• An example dataset should include a small amout of data.
• An example dataset should be a synthetic example including no real data - though it could of course be inspired by a real dataset
• An example dataset should be fully CF compliant
• A collection of example datasets could be searchable based on keywords for example.
• An example dataset should be in CDL notation form

It seems like we have made reasonable progress on what we believe an example should be.

Should the roadmap also describe the mechanism for how example datasets will be proposed? Should we use github issues for example to propose these.

Where will they be hosted? How will this look? Should this be touched upon in the roadmap paper?

A note, since this topic had different origins, the roadmap paper will outline CFs intentions/desires for the next 5 years. It will cover a range of topics. The section on example datasets should be brief (a paragraph or two, or maybe even in bullet point form). So we don't need to decide on everything in the roadmap paper. Just how we plan/hope to get there.