gunnarmorling / 1brc

1️⃣🐝🏎️ The One Billion Row Challenge -- A fun exploration of how quickly 1B rows from a text file can be aggregated with Java
https://www.morling.dev/blog/one-billion-row-challenge/
Apache License 2.0
6k stars 1.8k forks source link

Try more chunks than threads, and of different sizes #644

Closed tivrfoa closed 7 months ago

tivrfoa commented 7 months ago
/**
 * Solution based on thomaswue solution, commit:
 * commit d0a28599c293d3afe3291fc3cf169a7b25ae9ae6
 * Author: Thomas Wuerthinger
 * Date:   Sun Jan 21 20:13:48 2024 +0100
 *
 * The goal here was to try to improve the runtime of his 10k
 * solution of: 00:04.516
 *
 * With Thomas latest changes, his time is probably much better
 * already, and maybe even 1st place for the 10k too.
 * See: https://github.com/gunnarmorling/1brc/pull/606
 *
 * But as I was already coding something, I'll submit just to
 * see if it will be faster than his *previous* 10k time of
 * 00:04.516
 *
 * Changes:
 *   It's a similar idea of my previous solution, that if you split
 * the chunks evenly, some threads might finish much faster and
 * stay idle, so:
 *   1) Create more chunks than threads, so the ones that finish first
 * can do something;
 *   2) Decrease chunk sizes as we get closer to the end of the file.
 */

Check List:

Times for 220 million rows for the 10k, but the times on my machine are not good indicators ...

time ./target/CalculateAverage_thomaswue_image > thomas.out

real    0m5,239s
user    0m0,000s
sys 0m0,013s

time ./target/CalculateAverage_tivrfoa_image > me.out

real    0m4,810s
user    0m0,000s
sys 0m0,021s
gunnarmorling commented 7 months ago

00:02.205 now, nice!