Open alexander-held opened 1 year ago
An intermediate solution that might also help is the possibility to run histogram production on a subset of all templates via some filter, allowing to parallelize manually by calling the same function with different filters.
With
dask-awkward
and integration withhist
, it seems conceptually possible to make the template histogram creation fully Dask-driven. This could result in two significant improvements:The latter requires
dask-awkward
anddask-histogram
and needs to build the full task graph before callingcompute
. This is a much bigger change than providing an interface that allows distribution of template construction (which acts as a black box function) via e.g. Dask.