search

quasiben / dask-scheduler-performance

BSD 3-Clause "New" or "Revised" License
2 stars 5 forks source link

Big Shuffle Performance Plots #88

Open quasiben opened 3 years ago

quasiben commented 3 years ago

Not Optimized

Duration: 1355.80 s
Tasks Information
number of tasks: 100000
compute time: 10.16 s
transfer time: 5273.35 s
Scheduler Information
Address: tcp://10.33.12.31:8786
Workers: 40
Threads: 40
Memory: 270.47 GB
Screen Shot 2021-01-13 at 2 39 59 PM

PySpy Not Optimized

Optimized

Duration: 953.72 s
Tasks Information
number of tasks: 100000
compute time: 11.79 s
transfer time: 4690.80 s
Scheduler Information
Address: tcp://10.33.12.37:8786
Workers: 40
Threads: 40
Memory: 270.47 GB
Dask Version: 2020.12.0+34.g4f4d2950
Dask.Distributed Version: 2020.12.0+22.g74e8dc64
Screen Shot 2021-01-13 at 2 41 36 PM

PySpy Optimized

sched-graph