search

mrschweizer / PyThat

This is a community package which helps reading .h5 files created by ThatecOS and converting them to xarray objects and netcdf files. This software is not maintained by and has no affiliations to THATec Innovation GmbH.
MIT License
4 stars 1 forks source link

Please introduce netcdf fallback #22

Open jullma opened 1 month ago

jullma commented 1 month ago

`Traceback (most recent call last): File "/.venv/lib/python3.12/site-packages/xarray/backends/api.py", line 1360, in to_netcdf dump_to_store( File "/.venv/lib/python3.12/site-packages/xarray/backends/api.py", line 1407, in dump_to_store store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims) File "/.venv/lib/python3.12/site-packages/xarray/backends/common.py", line 367, in store self.set_variables( File "/.venv/lib/python3.12/site-packages/xarray/backends/common.py", line 409, in set_variables writer.add(source, target) File "/.venv/lib/python3.12/site-packages/xarray/backends/common.py", line 254, in add target[...] = source


  File "/.venv/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 79, in __setitem__
    data[key] = value
    ~~~~^^^^^
  File "src/netCDF4/_netCDF4.pyx", line 5532, in netCDF4._netCDF4.Variable.__setitem__
  File "src/netCDF4/_netCDF4.pyx", line 5819, in netCDF4._netCDF4.Variable._put
  File "src/netCDF4/_netCDF4.pyx", line 2113, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: HDF error

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/TR-MOKE-Linescans-H6/red_h5.py", line 27, in <module>
    mt = MeasurementTree(file_path,override=True)#,index=None)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/PyThat/h5to_nc.py", line 63, in __init__
    self.save_netcdf()
  File "/.venv/lib/python3.12/site-packages/PyThat/h5to_nc.py", line 91, in save_netcdf
    self.save_file_from_string(self.dataset, self.save_path)
  File "/.venv/lib/python3.12/site-packages/PyThat/h5to_nc.py", line 73, in save_file_from_string
    obj.to_netcdf(path)#,engine="h5netcdf")
    ^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/xarray/core/dataset.py", line 2329, in to_netcdf
    return to_netcdf(  # type: ignore  # mypy cannot resolve the overloads:(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.venv/lib/python3.12/site-packages/xarray/backends/api.py", line 1376, in to_netcdf
    store.close()
  File "/.venv/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 577, in close
    self._manager.close(**kwargs)
  File "/.venv/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 234, in close
    file.close()
  File "src/netCDF4/_netCDF4.pyx", line 2618, in netCDF4._netCDF4.Dataset.close
  File "src/netCDF4/_netCDF4.pyx", line 2585, in netCDF4._netCDF4.Dataset._close
  File "src/netCDF4/_netCDF4.pyx", line 2113, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: HDF error`
mrschweizer commented 1 month ago

Since there is already a exception handler, it would make sense to catch the RuntimeError and load it with engine='scipy' as backup. While scipy is a little bit less efficient, it is a bit mor stable, in my experience.