Add CF web page listing software that supports CF

[ethanrd]

It would be good to add a page listing all software tools/packages/libraries that produce, read, manipulate, and display CF data (similar to the netCDF “Software for Manipulating or Displaying NetCDF Data” page). The CF website already has a page listing compliance checkers that could be folded into a broader list of software.

[JonathanGregory]

I strongly agree with this suggestion. We should have such a webpage, possibly not just a list of links, but a table with one line of description of the capabilities of each item. Thanks.

[JonathanGregory]

And well done on getting us to No 100.

[ngalbraith]

Good idea, thank you; this will be a nice addition. Will we need some basic rules for deciding what should be included?

And, I agree with Jonathan that a short description would be really helpful. The list on the NetCDF tools page is far too cryptic - unless you know the name of the tool you're looking for, you'd have to click on each one. It's also full of acronyms, with no explanations. Setting up some 'rules' for tool submissions at the start will be important.

[davidhassell]

Good ideas.

Simply "reads and/or writes netCDF files" is too broad a rule for this list, I presume - there has to be some CF knowledge involved.

[sadielbartholomew]

I also think this is a good idea.

Simply "reads and/or writes netCDF files" is too broad a rule for this list, I presume

Agreed, especially as Unidata already provide essentially this, see their 'Software for Manipulating or Displaying NetCDF Data' page. (Incidentally, cfdm & cf-python are not listed there, at least as far as I can see. Can we get them added?)

[sadielbartholomew]

Oh, just seen that list was referenced in the opening comment! My bad.

[sadielbartholomew]

possibly not just a list of links, but a table with one line of description of the capabilities of each item

I also agree this is important, as there is a large difference in "bolted on" support for older version(s) of CF, such as provided by Iris, & inherent support for any version, as in cfdm & cf-python. (Obviously I am biased, but for a general reader that would certainly be useful & relevant information.)

[ethanrd]

@ngalbraith - The netCDF software page lists all the software packages at the top, each item links to a section below with a descriptions of that software package. We could do that or a table or maybe a term/definition (collapsible?) list.

[ethanrd]

@sadielbartholomew

Agreed, especially as Unidata already provide essentially this, see their 'Software for Manipulating or Displaying NetCDF Data' page. (Incidentally, cfdm & cf-python are not listed there, at least as far as I can see. Can we get them added?)

Yes, to request their addition just send an email to support@unidata.ucar.edu with the name and some descriptive text (with links) for each.

[erget]

@ethanrd this looks like a good start. FWIW my personal preference is markdown vs HTML for readability. If nested lists are too cumbersome a table can do the trick too.

[DocOtak]

I think the usage of the <details> element without the "open" attribute should be avoided since it's basically saying "most people will never look at these". I'd probably use a simple markdown unordered list, if you really are going for html semantics, you could probably use a description list:

<dl>
  <dt>cfdm - Python CF Data Model Package</dt>
  <dd>A description of the <a href="https://ncas-cms.github.io/cfdm/">Python CF Data Model Package</a>.</dd>

  <dt>cf-python - CF Python Package</dt>
  <dd>A description of the <a href="https://ncas-cms.github.io/cf-python/">CF Python Package</a>.</dd>
</dl>

renders like (on github at least):

cfdm - Python CF Data Model Package

A description of the Python CF Data Model Package.

cf-python - CF Python Package

A description of the CF Python Package.

[ethanrd]

Like @erget, my preference would be to stick to Markdown rather than include HTML, just to keep it simple.

I'm leaning towards a third-level header for each software package with a paragraph (or two) description with links. It isn't very compact vertically (with the default style anyway) but it is simple and the Markdown is clean. All the other options seem a bit cumbersome, either as Markdown/HTML or as rendered. This may also make it easier to build a TOC if the list gets long enough to need one.

Thoughts or strong objections? To keep things moving, I'm going to update the PR along these lines. We can make changes later if we decide on a different approach.

If you are the maintainer of one of the packages already listed, please provide some text describing your package. Similar if you have other packages that should be added. Just comment here and someone will add it or, if you have write permission to this repo, you should be able to push to this PR branch.

[erget]

@ethanrd I like this approach - if we switch to another style at some point we'll probably be grateful that the MD is straightforward.

[ethanrd]

I just rebased this on master and the build now looks good. There seems to be general agreement on adding this page but we need some text for the currently listed tools (and others?) before we merge this page.

• @davidhassell - Could you send me some text describing cfdm?
• @sadielbartholomew - Could you send me some text describing cf-python?
• @RosalynHatcher - Could you send me some text describing the CF Compliance Checker(s)? Including some info on the difference between the two would be great.
• @erget - Should we add the flattener to the list? If so, could you send me some text?
• @mwengren - Can you (or someone) send me a brief description of the IOOS Compliance Checker for inclusion in this page?

Thanks all!

[davidhassell]

@ethanrd Some cfdm text. Let me know if it's too much/too little.

The cfdm package (https://ncas-cms.github.io/cfdm) implements the CF data model (https://doi.org/10.5194/gmd-10-4619-2017) for its internal data structures and so is able to process any CF-compliant dataset. It is not strict about CF-compliance, however, so that partially conformant datasets may be ingested from existing datasets and written to new datasets. This is so that datasets which are partially conformant may nonetheless be modified in memory.

The central element defined by the CF data model is the field construct, which corresponds to a CF-netCDF data variable with all of its metadata.

The cfdm package can:

• read field constructs from netCDF datasets,
• create new field constructs in memory,
• inspect field constructs,
• test whether two field constructs are the same,
• modify field construct metadata and data,
• create subspaces of field constructs,
• write field constructs to netCDF datasets on disk,
• incorporate, and create, metadata stored in external files,
• read, write, and create data that have been compressed by convention (i.e. ragged or gathered arrays), whilst presenting a view of the data in its uncompressed form, and
• read, write, and create coordinates defined by geometry cells.

Note that cfdm enables the creation of CF field constructs, but it’s up to the user to use them in a CF-compliant way.

The cfdm package has, with few exceptions, only the functionality required to read and write datasets, and to create, modify and inspect field constructs in memory.

[ethanrd]

@davidhassell - I was thinking we would want to keep it quite short, maybe just your first paragraph. Otherwise, we will quickly need a TOC of some sort. What do others think?

Also, we might want to mention that this is a Python package. Maybe just add "Python" so it begins: "The cfdm Python package ..."?

[mwengren]

@ethanrd Happy to be invited to the software party! cc @Bobfrat In case you think there's anything here that I missed or should be changed.

Here's a short description of the IOOS Compliance Checker:

The IOOS Compliance Checker is a python based tool for data providers to check for completeness and community standard compliance of local or remote netCDF files against CF, ACDD, and IOOS Metadata Profile file standards. Compliance Checker can be used as a command-line tool or as a library that can be integrated into other software.

The Compliance Checker also includes a web-based version that enables a broader audience and improve accessibility for the checker. With the web version, providers can simply provide a link or upload their datasets and get the full suite of capabilities that Compliance Checker offers.

It also includes a plug-in framework that can be implemented by groups interested in developing their own test suites. Several plug-ins have been developed by IOOS to test compliance with specifications for preparing data to be submitted to different IOOS and NOAA data assembly centers. A list of IOOS-managed Compliance Checker plugins can be found on GitHub here.

Compliance Checker currently supports the following sources and standards:

Standard	Source	.nc/OPeNDAP/.cdl	SOS
ACDD (1.1, 1.3)	Built-in	X	-
CF (1.7)	Built-in	X	-
CF (1.6)	Built-in	X	-
IOOS SOS	Built-in	-	GetCapabilities, DescribeSensor
IOOS (1.1)	Built-in	X	-
IOOS (1.2)	Built-in	X	-
Glider DAC	ioos/cc-plugin-glider	X	-
NCEI (1.1, 2.0)	ioos/cc-plugin-ncei	X	-

Full usage instructions and examples can be found on the Compliance Checker GitHub repository.

[mwengren]

@ethanrd Just saw your earlier comment, the first two or three paragraphs could be used for brevity purposes.

[sadielbartholomew]

@ethanrd, sure, here's a summary of cf-python. I've put roughly the same level of detail as others pending a decision on the amount of text to provide.

@davidhassell, let me know if I have missed anything here, or if you'd like to make changes. I based this on the README text & docs introduction text (which are as you know essentially the same), but cut a little to avoid too much overlap with the cfdm text you supplied which should cover it by reference. I also added a sentence mentioning its relationship to cf-plot.

@ajheaps would you like cf-plot to be listed here? I think it would be good to list it as it displays CF data so fits the criteria.

---

The Python cf package, "cf-python", is an Earth Science data analysis library. It is built on cfdm [link to cfdm section of the listing] and implements the CF data model for its internal data structures so is able to process any CF-compliant dataset. Field constructs from cf can also be visualised in a powerful and flexible way with the cf-plot package [link to cf-plot section of listing if present].

The cf package can:

• read field constructs from netCDF, CDL, PP and UM datasets,
• create new field constructs in memory,
• inspect field constructs,
• test whether two field constructs are the same,
• modify field construct metadata and data,
• create subspaces of field constructs,
• write field constructs to netCDF datasets on disk,
• incorporate, and create, metadata stored in external files,
• read, write, and create data that have been compressed by convention (i.e. ragged or gathered arrays), whilst presenting a view of the data in its uncompressed form,
• read, write, and create coordinates defined by geometry cells,
• combine field constructs arithmetically,
• manipulate field construct data by arithmetical and trigonometrical operations,
• perform statistical collapses on field constructs,
• perform histogram, percentile and binning operations on field constructs,
• regrid field constructs with (multi-)linear, nearest neighbour, first- and second-order conservative and higher order patch recovery methods,
• apply convolution filters to field constructs,
• create moving means from field constructs,
• calculate derivatives of field constructs,
• create field constructs to create derived quantities (such as vorticity).

All of the above use LAMA functionality, which allows multiple fields larger than the available memory to exist and be manipulated.

[sadielbartholomew]

I was thinking we would want to keep it quite short, maybe just your first paragraph. Otherwise, we will quickly need a TOC of some sort. What do others think?

My thoughts on this are that a little more detail might be better, so that readers won't have to follow various links to compare and contrast the various tools. However, brevity could also be good.

If it wouldn't be too much hassle (you may want to keep it simple which would be fair enough), there could be short (summary) listing and a long (more verbose) listing, or keep a single listing but have something like an expand/collapse button to open up further detail?

[davidhassell]

@sadielbartholomew, @ajheaps yes, I think cf-plot should be included in the list

[erget]

• @erget - Should we add the flattener to the list? If so, could you send me some text?

Yes! I'll try adding it to your PR when I get a minute, hopefully later today.

I agree with @ethanrd that a shorter text is better - that way we don't duplicate information in too many places, where things might be subject to change.

[davidhassell]

@sadielbartholomew, @ajheaps yes, I think cf-plot should be included in the list

@ethanrd I'm happy with the shorter version. That was my first thought, too, but I thought I'd chuck in more detail that's easy to take away.

[ajheaps]

@sadielbartholomew https://github.com/sadielbartholomew, @davidhassell https://github.com/davidhassell Yes, I think cf-plot should be included here.  Do you need me to cook up some text other than that at the start of the cf-plot user guide?

cf-plot is a set of Python functions for making common contour, vector and line plots that climate researchers use. cf-plot generally usescf-python https://cfpython.bitbucket.io/to present the data and CF attributes for plotting. It can also use numpy arrays of data as the input fields making for flexible plotting of data. cf-plot uses the Python numpy, matplotlib and scipy libraries.

On 03/06/2020 21:34, David Hassell wrote:

@sadielbartholomew https://github.com/sadielbartholomew, @ajheaps https://github.com/ajheaps yes, I think cf-plot should be included in the list

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cf-convention/cf-convention.github.io/issues/100#issuecomment-638446913, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAV5ONY767AOPUQEJLBPPCTRU2XWHANCNFSM4M25U5AQ.

--

Andy Heaps
National Centre for Atmospheric Science (NCAS)
Room 118
Harry Pitt Building,
Reading University,
Earley Gate,
PO Box 243,
Reading RG6 6BB,
U.K.

For SatNav please use RG6 7BE

tel: 0118 378 6421 fax: 0118 378 8316 e-mail: andy.heaps@ncas.ac.uk

[davidhassell]

@ajheaps Thanks - I think that is fine. (just need to update the cf-python link to the Python3 version: https://ncas-cms.github.io/cf-python.)

[sadielbartholomew]

@ethanrd I am happy to go with the general consensus for shorter descriptions. In that case, please allow me to amend my earlier submission slightly to summarise the subsequent listing of functionality in a sentence (in an overarching way that will not become out-of-date), and to add a link to cf-python which in hindsight was missing but is quite crucial! The updated description:

---

The Python cf package, "cf-python" (https://ncas-cms.github.io/cf-python/), is an Earth Science data analysis library. It is built on cfdm [link to cfdm section of the listing] and implements the CF data model for its internal data structures so is able to process any CF-compliant dataset. It can read, write and inspect field constructs and manipulate the data and metadata therein by means of statistical operations, collapsing, subspacing, regridding and more. Field constructs from cf can also be visualised with the cf-plot package [link to cf-plot section of listing].

[RosalynHatcher]

@ethanrd - I'll send along information on the NCAS cf-checker as soon as (Reading/CEDA checkers are one and the same).

[ethanrd]

Hi all - I just pushed an update to the PR (#104) with, I believe, all the descriptions that I've received so far. Please take a look and see if it looks OK.

I plan to merge this in by the end of the day today (Friday) to get it in before the CF Workshop next week.

@RosalynHatcher - I left a place holder for the compliance checkers. We can get that added later after this PR is merged if necessary.

Have a good weekend, all.

[RosalynHatcher]

@ethanrd Here's a short description for the cf-checker at Reading/CEDA.

The cf-checker is a python tool to check compliance of netCDF files against the CF Convention. It can be run as a command-line tool, via a web interface (available at NCAS and CEDA) or imported as a python library. The cf-checker verifies conformance according to the requirements and recommendations laid out in the CF Conformance Document. It is possible to check conformance against any CF version.

[ethanrd]

Thanks @RosalynHatcher !