dask / distributed

A distributed task scheduler for Dask
https://distributed.dask.org
BSD 3-Clause "New" or "Revised" License
1.57k stars 718 forks source link

Improve graph submission time for P2P rechunking by avoiding unpack recursion into indices #8672

Closed fjetter closed 3 months ago

fjetter commented 3 months ago

Closes https://github.com/dask/dask/issues/11162

This is a bit annoying but it is very effective. It reduces the time to unpack the graph tasks from about 8min down to 2s

cc @dcherian @hendrikmakait

dcherian commented 3 months ago

hehe cool

github-actions[bot] commented 3 months ago

Unit Test Results

_See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests._

    29 files  ± 0      29 suites  ±0   11h 10m 42s :stopwatch: + 10m 29s  4 057 tests + 1   3 953 :white_check_mark:  -  3     97 :zzz: ±0  7 :x: +4  55 883 runs  +14  53 712 :white_check_mark: +10  2 163 :zzz: ±0  8 :x: +4 

For more details on these failures, see this check.

Results for commit a571fc4e. ± Comparison against base commit 7cbfc4d6.

hendrikmakait commented 3 months ago

Thanks, @fjetter!