search

JuliaParallel / DTables.jl

Distributed table structures and data manipulation operations built on top of Dagger.jl
MIT License
73 stars 4 forks source link

[DTable] Negative groupby scaling with threads #8

Open krynju opened 2 years ago

krynju commented 2 years ago

Noticed this in my benchmarks Table shows results for 2, 4, 8 and 16 threads and their comparisons against times from 2-thread config Other ops mostly scale well

Either the groupby algo is messed up or the scheduler doesn't handle it well

image

krynju commented 2 years ago

Scaling works fine with processes though each worker has 4 threads in this setup

4 workers time -> 696.21 8 workers time -> 234.21 speedup -> 2.97