search

dask / distributed

A distributed task scheduler for Dask
https://distributed.dask.org
BSD 3-Clause "New" or "Revised" License
1.55k stars 712 forks source link

Ability to specify cost value for computations #8654

Open BitTheByte opened 1 month ago

BitTheByte commented 1 month ago

Some functions may require more resources than others so the ability to set the function cost is useful to avoid overloading workers.

Imagine function we have 1 worker with 4 threads and F(x), G(x) while 4 instances of G(x) could be computed easily on the worker, 2 or more instances of F(x) will overload the worker CPU thus a possible API could be.

f.submit(x=123, compute_cost=4)

This will allow the function to fill up the 4 threads that the worker has without submitting more tasks to it before completing this function.

Of course, the API requires considerable design choices but this issue serves as a discussion starter.

BitTheByte commented 1 month ago

I'm fully aware that https://distributed.dask.org/en/latest/resources.html exists however it doesn't trigger autoscaling.