Closed lewismc closed 7 years ago
lewismc
commented
7 years ago The issue is related to https://github.com/Reading-eScience-Centre/pycovjson/blob/master/pycovjson/read_netcdf.py#L347 I am not sure exactly what the standard_name is however I'm debugging now and will find out.
lewismc
commented
7 years ago There are at least two bugs in the following function
def get_axes(self):
axes_dict = OrderedDict()
x_list = ['lon', 'longitude', 'LONGITUDE', 'Longitude', 'x', 'X']
y_list = ['lat', 'latitude', 'LATITUDE', 'Latitude', 'y', 'Y']
t_list = ['time', 'TIME', 't', 'T']
z_list = ['depth', 'DEPTH']
for coord in self.dataset.coords:
try:
if self.dataset[coord].axis == 'T':
axes_dict['t'] = coord
if self.dataset[coord].axis == 'Z':
axes_dict['z'] = coord
except:
pass
try:
if self.dataset[coord].units == 'degrees_north':
axes_dict['y'] = coord
if self.dataset[coord].units == 'degrees_east':
axes_dict['x'] = coord
except:
pass
try:
if self.dataset[coord].positive in ['up', 'down']:
axes_dict['z'] = coord
except:
pass
if coord in t_list or self.dataset[coord].name in t_list:
axes_dict['t'] = coord
if coord in z_list or self.dataset[coord].name in z_list:
axes_dict['z'] = coord
return axes_dictIf for example I execute the following
>>> self.dataset[coord].axis
Traceback (most recent call last):
File "/Users/lmcgibbn/.p2/pool/plugins/org.python.pydev_5.2.0.201608171824/pysrc/_pydevd_bundle/pydevd_exec2.py", line 3, in Exec
exec(exp, global_vars, local_vars)
File "<console>", line 1, in <module>
File "/Users/lmcgibbn/miniconda3/lib/python3.5/site-packages/xarray-0.8.2-py3.5.egg/xarray/core/common.py", line 194, in __getattr__
(type(self).__name__, name))
AttributeError: 'DataArray' object has no attribute 'axis'If I execute the following
>>> self.dataset[coord].positive
Traceback (most recent call last):
File "/Users/lmcgibbn/.p2/pool/plugins/org.python.pydev_5.2.0.201608171824/pysrc/_pydevd_bundle/pydevd_exec2.py", line 3, in Exec
exec(exp, global_vars, local_vars)
File "<console>", line 1, in <module>
File "/Users/lmcgibbn/miniconda3/lib/python3.5/site-packages/xarray-0.8.2-py3.5.egg/xarray/core/common.py", line 194, in __getattr__
(type(self).__name__, name))
AttributeError: 'DataArray' object has no attribute 'positive'
jonblower
commented
7 years ago I think all of these are symptoms of the fact that the code does not yet fully handle all possibilities that NetCDF files can offer. It makes some simplifying assumptions (e.g. it assumes all coordinate systems are lat-lon-time-depth), which is of course not always the case. It also seems to assume that the standard_name is always present, which is also not always true. (The standard_name is a term from the Climate and Forecast vocabulary - it is recommended to use a standard name if an appropriate one exists, but sometimes it doesn't. So we need to handle the case where the standard name is not present.
It would be quite a bit of work to fully support all of the CF conventions, so we may have to take a pragmatic approach and fix issues as they arise. (It might be a good idea to look at other Python libraries like Iris or cf-python, which may help us to support CF more fully.)
RileyWilliams
commented
7 years ago Have pushed a new version of read_netcdf which I think will adress some of these issues. @jonblower I agree. I have tried to support as many cases as possible, for example any netcdf file with a 'positive' attribute, is assumed to be the Z axis. Which I believe is part of the CF conventions for files which contain a 'z' axis.
lewismc
commented
7 years ago @jonblower @RileyWilliams yes I agree. I wonder if we were just to use the netCDF Python API instead of xarray... would this solve the issue. AFAICT it is xarray which does not follow CF conventions. Right now read_netcdf does not use any netCDF functions.
jonblower
commented
7 years ago I think netcdf-python supports some of CF (e.g. missing values) but only a small part of it. Not sure what xarray does. Usually you either have to code these in yourself or use a higher-level library. @RileyWilliams I could put you in touch with the cf-python developer if you like, although did we find out that this is not cross-platform? Also @RileyWilliams we could talk through your get_axes() function as there may be some ways we could improve it and squash a couple of bugs.
RileyWilliams
commented
7 years ago @lewismc @jonblower xarray uses the netcdf4 library in the backend, it should be able to read every file that the netcdf4 library can. Xarray stores netcdf files in a DataArray, the issues occur when mapping variables to their respective axis.
lewismc
commented
7 years ago ACK OK, lets have a look at read_netcdf then. Riley, you said you had pushed some sort of update... is that correct?
RileyWilliams
commented
7 years ago Yes I just pushed a quick fix for the issue you described, however I think that it does need some more work to make it more robust.
RileyWilliams
commented
7 years ago Additionally, I have also experienced an isolated issue with a netcdf file not being read by xarray correctly, but the netcdf library can read it fine. So there may be more going on behind the scenes. I have been going over the xarray code to work out how they go about decoding netcdf.
I am running into the following issue when attempting to convert a particular data product (link below available in both netCDF3 and netCDF4) to covjson. http://podaac-opendap.jpl.nasa.gov/opendap/allData/aquarius/L3/mapped/V4/7day_running/SCI/2015/158/Q20151582015164.L3m_R7_SCI_V4.0_SSS_1deg.bz2.html If I use pycovjson to read the file all is well
When I attempt to covert the product to covjson I get the following
I've not tried to debug this yet however it seems to be a bug. I've seen error messages similar to this before and I am going to start logging each occurrence here from now on so we can even write tests to try and verify if bugs are present.