search

HSF / PyHEP.dev-workshops

PyHEP Developer workshops
https://indico.cern.ch/e/PyHEP2023.dev
BSD 3-Clause "New" or "Revised" License
9 stars 1 forks source link

Using custom schedulers with dask #23

Closed btovar closed 7 months ago

btovar commented 1 year ago

Along the same lines of #18, it would be interesting to discuss the place of custom schedulers for Dask. In particular for Parsl and TaskVine, these execution engines have been developed to target resources commonly available to HEP analysis workflows, such as HPCs and university clusters. For example, Parsl has very nice profiles that target specific HPC installations out-of-the-box, and TaskVine adjust the resources (such as cores and memory) allocated to each function invocation to maximize throughput.

lgray commented 1 year ago

Ah on #18 lemme be clear I'm meaning more the concept of the dask task graph - which is engine independent... But seeing if we can hook whatever execution engine we want into all these steps would be great!