scikit-hep / cabinetry

design and steer profile likelihood fits
https://cabinetry.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
26 stars 21 forks source link

Dask for template histogram construction #401

Open alexander-held opened 1 year ago

alexander-held commented 1 year ago

With dask-awkward and integration with hist, it seems conceptually possible to make the template histogram creation fully Dask-driven. This could result in two significant improvements:

The latter requires dask-awkward and dask-histogram and needs to build the full task graph before calling compute. This is a much bigger change than providing an interface that allows distribution of template construction (which acts as a black box function) via e.g. Dask.

alexander-held commented 1 year ago

An intermediate solution that might also help is the possibility to run histogram production on a subset of all templates via some filter, allowing to parallelize manually by calling the same function with different filters.