pp-mo
commented
5 months ago Thanks @ScottWales for raising this. I certainly do see what you mean, and I know what it sounds like, but in fact I think this error is more of a mis-speak than a total mistake. What it should really say is "is not a valid NetCDF -CF variable name". I.E. it is applying the rules given here
So, Iris generally follows CF conventions, and in many cases requires them. We consider "CF strictness" as generally quite an important feature of Iris, though we do also have quite a few workarounds for common violations (like invalid "standard_name" and "units" attributes, and dimension variables in "coordinates" attributes). The appropriate level of this has also been a subject of some considerable discussion in recent times.
Unfortunately, we don't have a workaround for this problem.
I think it might even be a bit tricky to devise a solid automatic scheme for replacing non-compliant variable names, since the CF rules are rather highly fussy, not allowing "%" for instance.
Obviously we can fix this error message to give the proper context, but it remains a fact that Iris is rather strict about this + it is hard to work around. If practical, you might consider using "ncdata" to fix this, which at least avoids a need to copy or modify the original file. It's very much a kind of problem which that was designed for. Something like this should work ...
from ncdata.netcdf4 import from_nc4
from ncdata.iris import to_iris
ncds = from_nc4("sample.nc")
ncds.variables.rename("a(x)", "a_x")
cubes = to_iris(ncds)
print(cubes)I tested that on your minimal example + it does work for me.
Of course, in an actual case it may not be so simple, as you may need to correct references to the variable elsewhere, like e.g. data::coordinates = "a(x) b(y)" -- though I suppose that wouldn't be good CF either, so perhaps it's not a likely case.
Does this help at all @ScottWales ?
🐛 Bug Report
Iris is incorrectly validating netcdf variable names, rejecting opening valid files
How To Reproduce
Steps to reproduce the behaviour:
Iris fails in iris/common/mixin.py", line 177 with error
ValueError: 'a(x)' is not a valid NetCDF variable name.Expected behaviour
Iris should safely load the variable with a valid NetCDF name.
Iris matches names against a regex at https://github.com/SciTools/iris/blob/main/lib/iris/common/metadata.py#L41-L43, referencing an outdated link to netCDF documentation. If validation is performed on reading NetCDF files it should follow the current format rules:
Alternately validation could be performed only when writing out files, or left to the NetCDF library.
Unidata give this guidance on netcdf names in the file format documentation:
The netCDF4 library will give an error if you try to create a variable with an invalid name, e.g. one that ends with whitespace:
Environment
Additional context
Full stack trace
``` $ python sample_iris.py 2>&1 | sed 's:".*/python:".../python:' Traceback (most recent call last): File "/scratch/hc46/saw562/tmp/pete/sample_iris.py", line 13, in