gunnarmorling / 1brc

1️⃣🐝🏎️ The One Billion Row Challenge -- A fun exploration of how quickly 1B rows from a text file can be aggregated with Java
https://www.morling.dev/blog/one-billion-row-challenge/
Apache License 2.0
6.23k stars 1.87k forks source link

Use LinkedBlockingQueue to process results - based on thomaswue #603

Closed tivrfoa closed 9 months ago

tivrfoa commented 9 months ago

/**

Check List:

Time for 200 million rows 10k dataset:

time java --enable-preview --class-path target/average-1.0.0-SNAPSHOT.jar dev.morling.onebrc.CalculateAverage_tivrfoa > me.out

real    0m4,288s
user    0m0,380s
sys 0m0,175s

time java --enable-preview --class-path target/average-1.0.0-SNAPSHOT.jar dev.morling.onebrc.CalculateAverage_thomaswue > thomas.out

real    0m9,242s
user    0m0,341s
sys 0m0,183s
gunnarmorling commented 9 months ago

Output differs from the expected one for the 1B file.

Scratch that, error on my end.

gunnarmorling commented 9 months ago

00:04.255, nice.

tivrfoa commented 9 months ago

00:04.255, nice.

Thank you! Do you have the time for the 10k, please?

gunnarmorling commented 9 months ago

Yepp, here you go:

Validating calculate_average_tivrfoa.sh -- measurements_10K_1B.txt

+ rm -f measurements.txt
+ ln -s measurements_10K_1B.txt measurements.txt
Benchmark 1: timeout -v 300 ./calculate_average_tivrfoa.sh 2>&1
  Time (mean ± σ):      9.300 s ±  0.031 s    [User: 0.002 s, System: 0.004 s]
  Range (min … max):    9.266 s …  9.338 s    5 runs

Summary
  tivrfoa: trimmed mean 9.29848737266, raw times 9.294628507659999,9.27442990966,9.32640370066,9.266159457659999,9.33784911966

Leaderboard

| # | Result (m:s.ms) | Implementation     | JDK | Submitter     | Notes     |
|---|-----------------|--------------------|-----|---------------|-----------|
|   | 00:09.298 | [link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_tivrfoa.java)| 21.0.2-graal | [tivrfoa](https://github.com/tivrfoa) | GraalVM native binary, uses Unsafe |
tivrfoa commented 9 months ago

Yepp, here you go:

Validating calculate_average_tivrfoa.sh -- measurements_10K_1B.txt

+ rm -f measurements.txt
+ ln -s measurements_10K_1B.txt measurements.txt
Benchmark 1: timeout -v 300 ./calculate_average_tivrfoa.sh 2>&1
  Time (mean ± σ):      9.300 s ±  0.031 s    [User: 0.002 s, System: 0.004 s]
  Range (min … max):    9.266 s …  9.338 s    5 runs

Summary
  tivrfoa: trimmed mean 9.29848737266, raw times 9.294628507659999,9.27442990966,9.32640370066,9.266159457659999,9.33784911966

Leaderboard

| # | Result (m:s.ms) | Implementation     | JDK | Submitter     | Notes     |
|---|-----------------|--------------------|-----|---------------|-----------|
|   | 00:09.298 | [link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_tivrfoa.java)| 21.0.2-graal | [tivrfoa](https://github.com/tivrfoa) | GraalVM native binary, uses Unsafe |

Thank you!

My idea completely failed. Trying to optimize using a different HW does not work ... :sweat_smile:

gunnarmorling commented 9 months ago

Haha, yeah, at the top it's a bit like walking in the dark if you're on a different ISA.

Message ID: @.***>