Can we make dask an optional dependency of rechunker?
IIUC conceptually rechunker doesn't explicitly require dask - rechunking plans can be executed with a variety of executors.
This has been discussed in #85 and #87 but there still seem to be hard imports of both dask and dask.array.
Context: In https://github.com/pydata/xarray/pull/7019 I'm working on generalizing xarray to wrap other chunked parallel arrays, in particular cubed. We would like to be able to test xarray's handling of chunked arrays via cubed without ever importing dask. @tomwhite was able to remove the explicit dependency of cubed on dask, but cubed fundamentally depends on rechunker, which currently depends explicitly on dask. I think cubed only imports the parts of rechunker that don't explicitly depend on dask (i.e. the executors, see #87), but at the moment dask will still be installed when rechunker is installed.
Looking at the code, it seems that the dask dependence is already mostly compartmentalized into executors/dask.py. Dask is also imported in api.py, but it's mostly for isinstance() checks against dask.array.Array. We could handle those in a similar way to how xarray handles checking array types against optionally imported modules.
The only imports I'm not immediately sure how to handle are:
Can we make dask an optional dependency of rechunker?
IIUC conceptually rechunker doesn't explicitly require dask - rechunking plans can be executed with a variety of executors.
This has been discussed in #85 and #87 but there still seem to be hard imports of both
daskanddask.array.Context: In https://github.com/pydata/xarray/pull/7019 I'm working on generalizing xarray to wrap other chunked parallel arrays, in particular cubed. We would like to be able to test xarray's handling of chunked arrays via cubed without ever importing dask. @tomwhite was able to remove the explicit dependency of cubed on dask, but cubed fundamentally depends on rechunker, which currently depends explicitly on dask. I think cubed only imports the parts of rechunker that don't explicitly depend on dask (i.e. the executors, see #87), but at the moment dask will still be installed when rechunker is installed.
Looking at the code, it seems that the dask dependence is already mostly compartmentalized into
executors/dask.py. Dask is also imported inapi.py, but it's mostly forisinstance()checks againstdask.array.Array. We could handle those in a similar way to how xarray handles checking array types against optionally imported modules.The only imports I'm not immediately sure how to handle are:
dask.array.asarraywhen an xarray dataset is detected, which seems like an unnecessarily hard-coding to me now? https://github.com/pangeo-data/rechunker/blob/d625b6db8255bbf144d22f8aafd4cb8e17d5fced/rechunker/api.py#L446dask.utils.parse_bytes, but that could be vendored like Tom vendored it within cubed https://github.com/pangeo-data/rechunker/blob/d625b6db8255bbf144d22f8aafd4cb8e17d5fced/rechunker/api.py#L547cc @rabernat