search

pangeo-data / rechunker

Disk-to-disk chunk transformation for chunked arrays.
https://rechunker.readthedocs.io/
MIT License
163 stars 25 forks source link

Use dask checkpoint function in dask executor #81

Open rabernat opened 3 years ago

rabernat commented 3 years ago

Dask recently released some advanced graph manipulation tools, which includes a checkpoint function.

We could use this instead of our low-level hack to make a checkpoint here:

https://github.com/pangeo-data/rechunker/blob/6cc0f26720bfecb1eba00579a13d9b7c8004f652/rechunker/executors/dask.py#L51-L53