RuntimeError: NetCDF: HDF error

lwq-star commented 2 years ago

What happened?

I am reading the era5 nc data and extracting the values by latitude and longitude. My code worked fine for 8 hours but then suddenly it broke. Here is the error message. I'm guessing its an error related to the threads?

What did you expect to happen?

No response

Minimal Complete Verifiable Example

No response

MVCE confirmation

[ ] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
[ ] Complete example — the example is self-contained, including all data and the text of any traceback.
[ ] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
[ ] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

RemoteTraceback: 
"""
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "D:\project\assimilation\fun_tools.py", line 54, in cell_assimilation
    result = run_ensemble(observations, longitude, latitude)
  File "D:\project\assimilation\enkf.py", line 142, in run_ensemble
    enkf.setup_wofost()
  File "D:\project\assimilation\enkf.py", line 48, in setup_wofost
    self.weather_db = NCWeatherDataProvider(self.longitude, self.latitude)     
  File "D:\project\assimilation\ncweatherdataprovider.py", line 56, in __init__
    self._read_observations()
  File "D:\project\assimilation\ncweatherdataprovider.py", line 105, in _read_observations
    value = pnt[varname.replace("-","_")].values[0]
  File "C:\ProgramData\Anaconda3\lib\site-packages\xarray\core\dataarray.py", line 642, in values
    return self.variable.values
  File "C:\ProgramData\Anaconda3\lib\site-packages\xarray\core\variable.py", line 512, in values
    return _as_array_or_item(self._data)
  File "C:\ProgramData\Anaconda3\lib\site-packages\xarray\core\variable.py", line 252, in _as_array_or_item
    data = np.asarray(data)
  File "C:\ProgramData\Anaconda3\lib\site-packages\xarray\core\indexing.py", line 552, in __array__
    self._ensure_cached()
  File "C:\ProgramData\Anaconda3\lib\site-packages\xarray\core\indexing.py", line 549, in _ensure_cached
    self.array = NumpyIndexingAdapter(np.asarray(self.array))
  File "C:\ProgramData\Anaconda3\lib\site-packages\xarray\core\indexing.py", line 522, in __array__
    return np.asarray(self.array, dtype=dtype)
  File "C:\ProgramData\Anaconda3\lib\site-packages\xarray\core\indexing.py", line 423, in __array__
    return np.asarray(array[self.key], dtype=None)
  File "C:\ProgramData\Anaconda3\lib\site-packages\xarray\coding\variables.py", line 70, in __array__
    return self.func(self.array)
  File "C:\ProgramData\Anaconda3\lib\site-packages\xarray\coding\variables.py", line 137, in _apply_mask
    data = np.asarray(data, dtype=dtype)
  File "C:\ProgramData\Anaconda3\lib\site-packages\xarray\core\indexing.py", line 423, in __array__
    return np.asarray(array[self.key], dtype=None)
  File "C:\ProgramData\Anaconda3\lib\site-packages\xarray\backends\netCDF4_.py", line 93, in __getitem__
    return indexing.explicit_indexing_adapter(
  File "C:\ProgramData\Anaconda3\lib\site-packages\xarray\core\indexing.py", line 712, in explicit_indexing_adapter
    result = raw_indexing_method(raw_key.tuple)
  File "C:\ProgramData\Anaconda3\lib\site-packages\xarray\backends\netCDF4_.py", line 106, in _getitem
    array = getitem(original_array, key)
  File "src\netCDF4\_netCDF4.pyx", line 4406, in netCDF4._netCDF4.Variable.__getitem__
  File "src\netCDF4\_netCDF4.pyx", line 5350, in netCDF4._netCDF4.Variable._get
  File "src\netCDF4\_netCDF4.pyx", line 1927, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: HDF error
"""

Anything else we need to know?

My code is not easily simplified so it may not be reproducible, the last time I ran the code it was an error after 5 hours of normal work. first: 53%|█████▎ | 952/1803 [5:32:09<4:56:55, 20.93s/it] second: 78%|███████▊ | 1408/1803 [8:02:31<2:15:22, 20.56s/it] The error is the same

Environment

commit: None python: 3.8.12 (default, Oct 12 2021, 03:01:40) [MSC v.1916 64 bit (AMD64)] python-bits: 64 OS: Windows OS-release: 10 machine: AMD64 processor: Intel64 Family 6 Model 140 Stepping 1, GenuineIntel byteorder: little LC_ALL: None LANG: zh_CN.UTF8 LOCALE: ('Chinese (Simplified)_China', '936') libhdf5: 1.12.1 libnetcdf: 4.8.1 xarray: 2022.3.0 pandas: 1.3.5 numpy: 1.22.3 scipy: 1.7.3 netCDF4: 1.5.8 pydap: None h5netcdf: 1.0.1 h5py: 3.7.0 Nio: None zarr: None cftime: 1.5.1.1 nc_time_axis: None PseudoNetCDF: None rasterio: 1.2.10 cfgrib: None iris: None bottleneck: 1.3.4 dask: 2022.6.1 distributed: 2022.6.1 matplotlib: 3.5.1 cartopy: 0.20.1 seaborn: 0.11.2 numbagg: None fsspec: 2022.3.0 cupy: None pint: None sparse: None setuptools: 61.2.0 pip: 21.2.2 conda: 4.13.0 pytest: 7.1.2 IPython: 7.31.1 sphinx: 4.4.0

dcherian commented 2 years ago

Are you reading from a file on disk or on the network?

In general, HDF Error indicates a corrupt file or a network problem.

headtr1ck commented 1 year ago

Closing for now. Seems like an upstream issue. Feel free to reopen if it is still relevant.

pydata / xarray