codspeed-hq[bot]
commented
3 weeks ago CodSpeed Performance Report
Merging #3191 will degrade performances by 33.51%
Comparing colin/preshufflemerge (ab364ac) with main (5115c32)
Summary
❌ 2 regressions
✅ 15 untouched benchmarks
:warning: Please fix the performance issues or acknowledge them on CodSpeed.
Benchmarks breakdown
| Benchmark | main |
colin/preshufflemerge |
Change | |
|---|---|---|---|---|
| ❌ | test_iter_rows_first_row[100 Small Files] |
260.7 ms | 392 ms | -33.51% |
| ❌ | test_show[100 Small Files] |
42.4 ms | 50.7 ms | -16.45% |
codecov[bot]
Enables an experimental pre-shuffle merge strategy for shuffles.
Background
Our traditional shuffle strategy is a fully materializing map-reduce. Each of the
Mmap partition tasks ends in a fanout, producingNoutput partitions, and thenNreduce partition tasks are ran to mergeMoutputs each.Algorithm
The pre-shuffle merge strategy works by merging
Pmap partition tasks, before doing a fanout. TheNreduce partition tasks will now only have to mergeM/Poutputs. The benefit here is that we can reduce the total number of intermediate objects by a factor ofP, i.e. fromM * NtoM / P * N.Pcan be decided dynamically or statically. I chose to go with the dynamic route, together with a memory threshold of1gbto cap the maximum merging size. Essentially, the pre-shuffle merge algorithm in this PR attempts to greedily merge map outputs as they are ready, as long as the combined size of the merged partitions is less than1gb.Tests
1000 x 1000 shuffle of 100mb partitions,
2000 x 2000 shuffle of 100mb partitions
Todos