P2P Rechunking - Graph submission time reduction - HLG Layer for P2P rechunking

dask / distributed

A distributed task scheduler for Dask

https://distributed.dask.org

BSD 3-Clause "New" or "Revised" License

1.57k stars 718 forks source link

P2P Rechunking - Graph submission time reduction - HLG Layer for P2P rechunking #8676

Closed fjetter closed 2 months ago

fjetter commented 3 months ago

The P2P rechunking is still using a MaterializedLayer which can quickly blow up to substantial graph sizes due to the rather heavy nature of its individual tasks (see for some discussion on https://github.com/dask/dask/issues/11162)

We should write a proper HLG layer for this