search

eX-Mech / pymech

A Python software suite for Nek5000 and SIMSON
https://pymech.readthedocs.io/en/stable
GNU General Public License v3.0
24 stars 24 forks source link

Xarray's open_mfdataset function does not work without dask: `ValueError: unrecognized chunk manager dask - must be one of: []` #122

Open ashwinvis opened 2 weeks ago

ashwinvis commented 2 weeks ago

Workaround: pip install dask

ashwinvis commented 2 weeks ago

For example see this test case tests/test_mfdataset.py::test_xr_open_mfdataset

ashwinvis commented 2 weeks ago 
>>> import pymech as pm
>>> pm.open_dataset("/tmp/channel3D_0.f00001")
<xarray.Dataset> Size: 15MB
Dimensions:   (z: 64, y: 64, x: 64)
Coordinates:
  * x         (x) float64 512B 0.0 0.05037 0.1603 0.3105 ... 6.123 6.233 6.283
  * y         (y) float64 512B -1.0 -0.9974 -0.9918 ... 0.9918 0.9974 1.0
  * z         (z) float64 512B 0.0 0.02518 0.08017 0.1553 ... 3.061 3.116 3.142
    time      float64 8B 0.2
Data variables:
    xmesh     (z, y, x) float64 2MB 0.0 0.05037 0.1603 ... 6.123 6.233 6.283
    ymesh     (z, y, x) float64 2MB -1.0 -1.0 -1.0 -1.0 -1.0 ... 1.0 1.0 1.0 1.0
    zmesh     (z, y, x) float64 2MB 0.0 0.0 0.0 0.0 ... 3.142 3.142 3.142 3.142
    ux        (z, y, x) float64 2MB 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0
    uy        (z, y, x) float64 2MB 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0
    uz        (z, y, x) float64 2MB 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0
    pressure  (z, y, x) float64 2MB 0.004913 0.0467 0.0195 ... 0.03768 0.0761
>>> ds = pm.open_dataset("/tmp/channel3D_0.f00001")
>>> import xarray as xr
>>> xr.open_dataset("/tmp/channel3D_0.f00001", engine="pymech")
<xarray.Dataset> Size: 15MB
Dimensions:   (z: 64, y: 64, x: 64)
Coordinates:
  * x         (x) float64 512B 0.0 0.05037 0.1603 0.3105 ... 6.123 6.233 6.283
  * y         (y) float64 512B -1.0 -0.9974 -0.9918 ... 0.9918 0.9974 1.0
  * z         (z) float64 512B 0.0 0.02518 0.08017 0.1553 ... 3.061 3.116 3.142
    time      float64 8B ...
Data variables:
    xmesh     (z, y, x) float64 2MB ...
    ymesh     (z, y, x) float64 2MB ...
    zmesh     (z, y, x) float64 2MB ...
    ux        (z, y, x) float64 2MB ...
    uy        (z, y, x) float64 2MB ...
    uz        (z, y, x) float64 2MB ...
    pressure  (z, y, x) float64 2MB ...
>>> xr.open_mfdataset("/tmp/channel3D_0.f00001", engine="pymech")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/avmo/Sources/snek5000/pymech/.nox/release-tests-dist_type-only-binary/lib/python3.9/site-packages/xarray/backends/api.py", line 1077, in open_mfdataset
    datasets = [open_(p, **open_kwargs) for p in paths]
  File "/home/avmo/Sources/snek5000/pymech/.nox/release-tests-dist_type-only-binary/lib/python3.9/site-packages/xarray/backends/api.py", line 1077, in <listcomp>
    datasets = [open_(p, **open_kwargs) for p in paths]
  File "/home/avmo/Sources/snek5000/pymech/.nox/release-tests-dist_type-only-binary/lib/python3.9/site-packages/xarray/backends/api.py", line 594, in open_dataset
    ds = _dataset_from_backend_dataset(
  File "/home/avmo/Sources/snek5000/pymech/.nox/release-tests-dist_type-only-binary/lib/python3.9/site-packages/xarray/backends/api.py", line 370, in _dataset_from_backend_dataset
    ds = _chunk_ds(
  File "/home/avmo/Sources/snek5000/pymech/.nox/release-tests-dist_type-only-binary/lib/python3.9/site-packages/xarray/backends/api.py", line 318, in _chunk_ds
    chunkmanager = guess_chunkmanager(chunked_array_type)
  File "/home/avmo/Sources/snek5000/pymech/.nox/release-tests-dist_type-only-binary/lib/python3.9/site-packages/xarray/namedarray/parallelcompat.py", line 117, in guess_chunkmanager
    raise ValueError(
ValueError: unrecognized chunk manager dask - must be one of: []
ashwinvis commented 2 weeks ago

Further reading:

We need to check if this is an upstream issue or the way we implement pymech as a xarray backend.