search

dask / distributed

A distributed task scheduler for Dask
https://distributed.dask.org
BSD 3-Clause "New" or "Revised" License
1.55k stars 712 forks source link

Warn user if we encounter too many tasks groups on the scheduler #8678

Closed hendrikmakait closed 5 days ago

hendrikmakait commented 2 weeks ago

Problem

As described in https://github.com/dask/distributed/issues/8677, having too many task groups can become prohibitively expensive, in particular for the TaskProgress dashboard.

Solution

We should warn the user if there are too many tasks groups on the scheduler. This might either be a simple threshold as mentioned in https://github.com/dask/distributed/issues/8677 or something that also involves a ratio of tasks to task groups.

fjetter commented 2 weeks ago

With https://github.com/dask/distributed/pull/8681 I hope that we won't see any more issues popping up. Having "many" task groups is still something unusual but I'm not sure how bad it really is