search

dask-contrib / dask-histogram

Histograms with task scheduling.
https://dask-histogram.readthedocs.io
BSD 3-Clause "New" or "Revised" License
22 stars 4 forks source link

feat: use a tree reduce for staged fills instead of pairwise adds #120

Closed lgray closed 7 months ago

lgray commented 7 months ago

This results in a task graph with better parallelism since now the leaves of the tree reduce do not wait on each other outside of split_every sized blocks. Better fault recovery too (fewer recomputed subgraphs if there's a failure).

martindurant commented 7 months ago

This results in a task graph with better parallelism

For posterity, can we have images of before and after?

lgray commented 7 months ago

Here's before: image

Here's after: image