Open trexfeathers opened 1 year ago
2023-02-02
, 2023-02-14
, 2023-03-22
Note this issue is not intended as a debate, hence why it is not posted as a discussion. The below conversations took place in real time, with a group deliberately sized to aid decision making.
When loading NetCDF files, Iris will load all CF-compliant elements. A container of non-compliant variables and attributes will be attached to the Cube
(s).
Encourage users:
If this causes you problems, please reach out to us to see if we can collaborate on a solution.
Cube
s or isolated in own list?FUTURE
flag?ncedit
or NetCDF4, but this can be challenging, and editing a copy may be unrealistic.Cube
, to help with fixing?Cube
and _DimensionalMetadata
.Cube
s from scratch.Determined via voting.
Oooh just discovered this issue via DragonTaming board @trexfeathers.
Sounds like you've got a fair bit of input from working group already; please shout though if useful to have more, as this is a particularly painful area for space weather - and we've got a good amount of requirements (ionosphere and lower) in the iris-o-sphere of traditional geographic lat/lon coords!
My personal proposal, after some loose discussion with @bjlittle and @pp-mo:
CFVariableMixin
gets a new member called something like loading_problems
- this is either a str
or list
of str
.LOADING_PROBLEMS
. This is a dict
of list
.Cube
, AuxCoord
, whatever) get wrapped in a try
-except
block. In the except
block:
Cube
with minimal parsing - the array (if present) goes into Cube.data
and everything else goes into Cube.attributes
.Cube.loading_problems
.LOADING_PROBLEMS
- the dict
key is the file nameCube
during the loading process get wrapped in a try
-except
block. In the except
block:
.loading_problems
member of this objectLOADING_PROBLEMS
- the dict
key is the file nameLOADING_PROBLEMS
is checked to see if it has been populated (or if it has grown, depending whether we clear it out between load calls), and a single warning is issued suggesting that the user check the contents.Cube
s to other classes. We already have the from_metadata()
method. Maybe all we need to add is decent documentation examples, but maybe we need new conveniences too?This should serve to allow loading to continue under as many circumstances as possible, and providing the user with recourse to fix up problem objects post-loading. Should be reasonably simple to scour through the loading code to find likely places for try-except
. This feature need not be limited to CF-parsing in NetCDF, although that is presumably the source of most of the problems.
From @SciTools/peloton : consider an option to fail/warn fast.
Iris needs a public statement on how it handles NetCDF files that deviate from the CF conventions. This will serve multiple benefits:
Writing this statement will involve making some difficult decisions. A working group is tackling this now: @tkknight, @bjlittle, @lbdreyer, @pp-mo, @trexfeathers, @stephenworsley, @ESadek-MO, @scottrobinson02, @HGWright
Factors at play
Items affected
(please edit if you know of others)
5119
5126
5068
5067
5003
4495
1801
5171
4453
5257