Closed lewismc closed 7 years ago
The issue is related to https://github.com/Reading-eScience-Centre/pycovjson/blob/master/pycovjson/read_netcdf.py#L347 I am not sure exactly what the standard_name is however I'm debugging now and will find out.
There are at least two bugs in the following function
def get_axes(self):
axes_dict = OrderedDict()
x_list = ['lon', 'longitude', 'LONGITUDE', 'Longitude', 'x', 'X']
y_list = ['lat', 'latitude', 'LATITUDE', 'Latitude', 'y', 'Y']
t_list = ['time', 'TIME', 't', 'T']
z_list = ['depth', 'DEPTH']
for coord in self.dataset.coords:
try:
if self.dataset[coord].axis == 'T':
axes_dict['t'] = coord
if self.dataset[coord].axis == 'Z':
axes_dict['z'] = coord
except:
pass
try:
if self.dataset[coord].units == 'degrees_north':
axes_dict['y'] = coord
if self.dataset[coord].units == 'degrees_east':
axes_dict['x'] = coord
except:
pass
try:
if self.dataset[coord].positive in ['up', 'down']:
axes_dict['z'] = coord
except:
pass
if coord in t_list or self.dataset[coord].name in t_list:
axes_dict['t'] = coord
if coord in z_list or self.dataset[coord].name in z_list:
axes_dict['z'] = coord
return axes_dict
If for example I execute the following
>>> self.dataset[coord].axis
Traceback (most recent call last):
File "/Users/lmcgibbn/.p2/pool/plugins/org.python.pydev_5.2.0.201608171824/pysrc/_pydevd_bundle/pydevd_exec2.py", line 3, in Exec
exec(exp, global_vars, local_vars)
File "<console>", line 1, in <module>
File "/Users/lmcgibbn/miniconda3/lib/python3.5/site-packages/xarray-0.8.2-py3.5.egg/xarray/core/common.py", line 194, in __getattr__
(type(self).__name__, name))
AttributeError: 'DataArray' object has no attribute 'axis'
If I execute the following
>>> self.dataset[coord].positive
Traceback (most recent call last):
File "/Users/lmcgibbn/.p2/pool/plugins/org.python.pydev_5.2.0.201608171824/pysrc/_pydevd_bundle/pydevd_exec2.py", line 3, in Exec
exec(exp, global_vars, local_vars)
File "<console>", line 1, in <module>
File "/Users/lmcgibbn/miniconda3/lib/python3.5/site-packages/xarray-0.8.2-py3.5.egg/xarray/core/common.py", line 194, in __getattr__
(type(self).__name__, name))
AttributeError: 'DataArray' object has no attribute 'positive'
I think all of these are symptoms of the fact that the code does not yet fully handle all possibilities that NetCDF files can offer. It makes some simplifying assumptions (e.g. it assumes all coordinate systems are lat-lon-time-depth), which is of course not always the case. It also seems to assume that the standard_name is always present, which is also not always true. (The standard_name is a term from the Climate and Forecast vocabulary - it is recommended to use a standard name if an appropriate one exists, but sometimes it doesn't. So we need to handle the case where the standard name is not present.
It would be quite a bit of work to fully support all of the CF conventions, so we may have to take a pragmatic approach and fix issues as they arise. (It might be a good idea to look at other Python libraries like Iris or cf-python, which may help us to support CF more fully.)
Have pushed a new version of read_netcdf which I think will adress some of these issues. @jonblower I agree. I have tried to support as many cases as possible, for example any netcdf file with a 'positive' attribute, is assumed to be the Z axis. Which I believe is part of the CF conventions for files which contain a 'z' axis.
@jonblower @RileyWilliams yes I agree. I wonder if we were just to use the netCDF Python API instead of xarray... would this solve the issue. AFAICT it is xarray which does not follow CF conventions. Right now read_netcdf does not use any netCDF functions.
I think netcdf-python supports some of CF (e.g. missing values) but only a small part of it. Not sure what xarray does. Usually you either have to code these in yourself or use a higher-level library. @RileyWilliams I could put you in touch with the cf-python developer if you like, although did we find out that this is not cross-platform? Also @RileyWilliams we could talk through your get_axes() function as there may be some ways we could improve it and squash a couple of bugs.
@lewismc @jonblower xarray uses the netcdf4 library in the backend, it should be able to read every file that the netcdf4 library can. Xarray stores netcdf files in a DataArray, the issues occur when mapping variables to their respective axis.
ACK OK, lets have a look at read_netcdf then. Riley, you said you had pushed some sort of update... is that correct?
Yes I just pushed a quick fix for the issue you described, however I think that it does need some more work to make it more robust.
Additionally, I have also experienced an isolated issue with a netcdf file not being read by xarray correctly, but the netcdf library can read it fine. So there may be more going on behind the scenes. I have been going over the xarray code to work out how they go about decoding netcdf.
I am running into the following issue when attempting to convert a particular data product (link below available in both netCDF3 and netCDF4) to covjson. http://podaac-opendap.jpl.nasa.gov/opendap/allData/aquarius/L3/mapped/V4/7day_running/SCI/2015/158/Q20151582015164.L3m_R7_SCI_V4.0_SSS_1deg.bz2.html If I use pycovjson to read the file all is well
When I attempt to covert the product to covjson I get the following
I've not tried to debug this yet however it seems to be a bug. I've seen error messages similar to this before and I am going to start logging each occurrence here from now on so we can even write tests to try and verify if bugs are present.