dask / distributed

A distributed task scheduler for Dask
https://distributed.dask.org
BSD 3-Clause "New" or "Revised" License
1.55k stars 712 forks source link

split frames for large graphs #8684

Open fjetter opened 2 weeks ago

fjetter commented 2 weeks ago

There's been a report about HLG serialization to be broken for very large graphs, see https://github.com/dask/distributed/issues/5581#issuecomment-2025702355 due to a msgpack limitation

I'm not entirely sure what's happening but this serialize/serialize_and_aplit thing may be related. We've seen slow transmissions of large grpahs in the past and I will have to review this closely because I don't want our transmission to be slower than it already is. This would break up the frames into 64MB chunks by default which might not be what we want.

github-actions[bot] commented 2 weeks ago

Unit Test Results

_See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests._

    29 files  ±0      29 suites  ±0   11h 4m 35s :stopwatch: + 2m 3s  4 058 tests ±0   3 957 :white_check_mark: ±0     97 :zzz: ±0  4 :x: ±0  55 897 runs  ±0  53 729 :white_check_mark:  - 1  2 162 :zzz: ±0  6 :x: +1 

For more details on these failures, see this check.

Results for commit 9ce84ab9. ± Comparison against base commit 9672121c.