SciTools / iris

A powerful, format-agnostic, and community-driven Python package for analysing and visualising Earth science data
https://scitools-iris.readthedocs.io/en/stable/
BSD 3-Clause "New" or "Revised" License
635 stars 283 forks source link

Technical overview of netcdf interface #5598

Open pp-mo opened 11 months ago

pp-mo commented 11 months ago

Summary (ideally in detail) of netcdf interfacing (i.e. save + load), analagous to the one on PP/FF

@trexfeathers suggested this can be a thin skeleton to start with -- that gives us a place to put the awkward detail items (see below) without needing to provide a fully-fledged + rounded account from the first

See also this comment and subsequent ones ...

another case for the NetCDF I/O technical paper

I reckon we've now clocked up (at least) 4 "special detail issues" relating to netcdf, that would really like a place like that to live :


The tech papers probably shouldn't stay in the Developer's Guide really

Totally, I think they should all be combined in the further topics UserGuide area

pp-mo commented 11 months ago

See also https://github.com/SciTools/iris/pull/5597, now ongoing

pp-mo commented 11 months ago

We also really need to make it clear what Iris means by "user" attributes.
Roughly, this means non-managed ones which can exist in an ".attributes"dictionary -- unlike those which are "recognised + controlled/interpreted" by Iris, e.g. _FillValue, units, scale, conventions, axis. However ... there are are CF attributes which are not managed by Iris, at least not at present : e.g. "references", "external_variables", "coordinate_interpolation". And then there also CF attributes which are "recognised" by Iris but can still appear in the "user" attributes dictionary : e.g. "comment", "history".

trexfeathers commented 11 months ago

Additional note : the split-attributes (cube attributes) description could also be be better linked into the existing documentation. Notably, here

If you look at the latest version of this page, the specialism of CubeAttrsDict is highlighted in 2 separate places.

pp-mo commented 11 months ago

Additional note : the split-attributes (cube attributes) description could also be be better linked into the existing documentation. Notably, here

If you look at the latest version of this page, the specialism of CubeAttrsDict is highlighted in 2 separate places.

OK I think I failed to look in the latest version ! Removed that note

pp-mo commented 7 months ago

For the split-attrs notes, an account written for an internal support conversation.

"" Broadly, like any FUTURE flag, we think it is the better way + you should adopt it unless you need to preserve some specific results from legacy code.

My quick summary : In the "old" style, some specific cube attributes (defined by CF) are always "local", i.e. attached to a data variable. Ones not defined by CF are saved as netcdf "global" attributes if they have the same value for all saved cubes, otherwise local. In the "new" version, attributes can be recorded on the cube as either 'global' or 'local', most notably by the netcdf load. This is (mostly) respected on save, except that global settings with different values in different output cubes are "demoted" to local ones.

You see, it really isn't that simple to explain concisely ... In fact, a complete explanation of the legacy behaviour is in itself very complicated ! ""