NetCDF: Attempting netcdf-4 operation on netcdf-3 file

[mickaellalande]

I'm trying to open MODIS .hdf files, but I get the error : NetCDF: Attempting netcdf-4 operation on netcdf-3 file. Does anyone knows how to open that files? (https://nsidc.org/data/MOD10C1)

import xarray as xr
xr.open_dataset('MOD10C1.A2000055.061.2020037182124.hdf')

RuntimeError: NetCDF: Attempting netcdf-4 operation on netcdf-3 file

I already opened hdf files from another product without any issue... (https://nsidc.org/data/MOD10CM)

Here are two examples, with one that works and the other one that causes the issue: MODIS.zip

Thanks in advance for your help!

Output of xr.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 3.8.5 | packaged by conda-forge | (default, Jul 24 2020, 01:25:15) [GCC 7.5.0] python-bits: 64 OS: Linux OS-release: 4.19.0-16-amd64 machine: x86_64 processor: byteorder: little LC_ALL: None LANG: fr_FR.UTF-8 LOCALE: fr_FR.UTF-8 libhdf5: 1.10.6 libnetcdf: 4.7.4 xarray: 0.16.0 pandas: 1.1.0 numpy: 1.19.1 scipy: 1.5.2 netCDF4: 1.5.4 pydap: None h5netcdf: None h5py: None Nio: None zarr: 2.4.0 cftime: 1.2.1 nc_time_axis: 1.2.0 PseudoNetCDF: None rasterio: 1.1.5 cfgrib: 0.9.8.5 iris: None bottleneck: None dask: 2.21.0 distributed: 2.21.0 matplotlib: 3.2.0 cartopy: 0.17.0 seaborn: None numbagg: None pint: None setuptools: 49.2.0.post20200712 pip: 20.2 conda: None pytest: 6.0.0 IPython: 7.16.1 sphinx: None

[shoyer]

could you kindly share the full error traceback?

[mickaellalande]

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-4-db5c6cefdcdc> in <module>
----> 1 xr.open_dataset(root_data_path+'/MODIS/MOD10C1_Terra/MOD10C1.A2000055.061.2020037182124.hdf')

~/miniconda3/envs/phd_v3/lib/python3.8/site-packages/xarray/backends/api.py in open_dataset(filename_or_obj, group, decode_cf, mask_and_scale, decode_times, autoclose, concat_characters, decode_coords, engine, chunks, lock, cache, drop_variables, backend_kwargs, use_cftime, decode_timedelta)
    543 
    544     with close_on_error(store):
--> 545         ds = maybe_decode_store(store)
    546 
    547     # Ensure source filename always stored in dataset object (GH issue #2550)

~/miniconda3/envs/phd_v3/lib/python3.8/site-packages/xarray/backends/api.py in maybe_decode_store(store, lock)
    449 
    450     def maybe_decode_store(store, lock=False):
--> 451         ds = conventions.decode_cf(
    452             store,
    453             mask_and_scale=mask_and_scale,

~/miniconda3/envs/phd_v3/lib/python3.8/site-packages/xarray/conventions.py in decode_cf(obj, concat_characters, mask_and_scale, decode_times, decode_coords, drop_variables, use_cftime, decode_timedelta)
    578         encoding = obj.encoding
    579     elif isinstance(obj, AbstractDataStore):
--> 580         vars, attrs = obj.load()
    581         extra_coords = set()
    582         file_obj = obj

~/miniconda3/envs/phd_v3/lib/python3.8/site-packages/xarray/backends/common.py in load(self)
    121         """
    122         variables = FrozenDict(
--> 123             (_decode_variable_name(k), v) for k, v in self.get_variables().items()
    124         )
    125         attributes = FrozenDict(self.get_attrs())

~/miniconda3/envs/phd_v3/lib/python3.8/site-packages/xarray/backends/netCDF4_.py in get_variables(self)
    397 
    398     def get_variables(self):
--> 399         dsvars = FrozenDict(
    400             (k, self.open_store_variable(k, v)) for k, v in self.ds.variables.items()
    401         )

~/miniconda3/envs/phd_v3/lib/python3.8/site-packages/xarray/core/utils.py in FrozenDict(*args, **kwargs)
    436 
    437 def FrozenDict(*args, **kwargs) -> Frozen:
--> 438     return Frozen(dict(*args, **kwargs))
    439 
    440 

~/miniconda3/envs/phd_v3/lib/python3.8/site-packages/xarray/backends/netCDF4_.py in <genexpr>(.0)
    398     def get_variables(self):
    399         dsvars = FrozenDict(
--> 400             (k, self.open_store_variable(k, v)) for k, v in self.ds.variables.items()
    401         )
    402         return dsvars

~/miniconda3/envs/phd_v3/lib/python3.8/site-packages/xarray/backends/netCDF4_.py in open_store_variable(self, name, var)
    374         # netCDF4 specific encoding; save _FillValue for later
    375         encoding = {}
--> 376         filters = var.filters()
    377         if filters is not None:
    378             encoding.update(filters)

netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable.filters()

netCDF4/_netCDF4.pyx in netCDF4._netCDF4._ensure_nc_success()

RuntimeError: NetCDF: Attempting netcdf-4 operation on netcdf-3 file

[shoyer]

OK, so the (misleading) error message here is ultimately coming from the underlying netCDF library, not Xarray.

Inspecting these files, it looks like they are HDF4 files, not netCDF3. We haven't really tested HDF4 files much with Xarray, but it would be nice to support them.

[kmuehlbauer]

Late to the party. If this is still an issue for you @mickaellalande, MODIS data can also be read using engine="rasterio" (needs rioxarray).

In [1]: import xarray as xr
   ...: with xr.open_dataset('source/MODIS/MOD10C1.A2000055.061.2020037182124.hdf', engine="rasterio") as ds:
   ...:     display(ds)
   ...: 
<xarray.Dataset>
Dimensions:                 (band: 1, x: 7200, y: 3600)
Coordinates:
  * band                    (band) int64 1
  * x                       (x) float64 -180.0 -179.9 -179.9 ... 179.9 180.0
  * y                       (y) float64 89.97 89.92 89.88 ... -89.93 -89.98
    spatial_ref             int64 ...
Data variables:
    Day_CMG_Snow_Cover      (band, y, x) float32 ...
    Day_CMG_Clear_Index     (band, y, x) float32 ...
    Day_CMG_Cloud_Obscured  (band, y, x) float32 ...
    Snow_Spatial_QA         (band, y, x) float32 ...
Attributes: (12/50)
    ALGORITHMPACKAGEACCEPTANCEDATE:     09-2013
    ALGORITHMPACKAGEMATURITYCODE:       Normal
    ALGORITHMPACKAGENAME:               MOD_PR10C1
    ALGORITHMPACKAGEVERSION:            6
    ASSOCIATEDINSTRUMENTSHORTNAME.1:    MODIS
    ASSOCIATEDPLATFORMSHORTNAME.1:      Terra
    ...                                 ...
    SNOWCOVERPERCENT:                   28
    SOUTHBOUNDINGCOORDINATE:            -90.0
    SPSOPARAMETERS:                     none
    VERSIONID:                          61
    WESTBOUNDINGCOORDINATE:             -180.0
    ZONEIDENTIFIER:                     Other Grid System