Use LinkedBlockingQueue to process results - based on thomaswue

[tivrfoa]

/**

• Solution based on thomaswue @thomaswue solution, commit:
• commit d0a28599c293d3afe3291fc3cf169a7b25ae9ae6
• Author: Thomas Wuerthinger thomas.wuerthinger@oracle.com
• Date: Sun Jan 21 20:13:48 2024 +0100
• Changes:
• 1) Use LinkedBlockingQueue to store partial results, that
• will then be merged into the final map later.
• As different chunks finish at different times, this allows
• to process them as they finish, instead of joining the
• threads sequentially.
• This change seems more useful for the 10k dataset, as the
• runtime difference of each chunk is greater.
• 2) Use only 4 threads if the file is >= 14GB.
• This showed much better results on my local test, but I only
• run with 200 million rows (because of limited RAM), and I have
• no idea how it will perform on the 1brc HW. */

Check List:

• [x] Tests pass (./test.sh <username> shows no differences between expected and actual outputs)

• [x] All formatting changes by the build are committed

• [x] Your launch script is named calculate_average_<username>.sh (make sure to match casing of your GH user name) and is executable

• [x] Output matches that of calculate_average_baseline.sh

• [x] For new entries, or after substantial changes: When implementing custom hash structures, please point to where you deal with hash collisions (line number): same as thomaswue existing solution

• Execution time:

• Execution time of reference implementation:

Time for 200 million rows 10k dataset:

time java --enable-preview --class-path target/average-1.0.0-SNAPSHOT.jar dev.morling.onebrc.CalculateAverage_tivrfoa > me.out

real    0m4,288s
user    0m0,380s
sys 0m0,175s

time java --enable-preview --class-path target/average-1.0.0-SNAPSHOT.jar dev.morling.onebrc.CalculateAverage_thomaswue > thomas.out

real    0m9,242s
user    0m0,341s
sys 0m0,183s

[gunnarmorling]

Output differs from the expected one for the 1B file.

Scratch that, error on my end.

[gunnarmorling]

00:04.255, nice.

[tivrfoa]

00:04.255, nice.

Thank you! Do you have the time for the 10k, please?

[gunnarmorling]

Yepp, here you go:

Validating calculate_average_tivrfoa.sh -- measurements_10K_1B.txt

+ rm -f measurements.txt
+ ln -s measurements_10K_1B.txt measurements.txt
Benchmark 1: timeout -v 300 ./calculate_average_tivrfoa.sh 2>&1
  Time (mean ± σ):      9.300 s ±  0.031 s    [User: 0.002 s, System: 0.004 s]
  Range (min … max):    9.266 s …  9.338 s    5 runs

Summary
  tivrfoa: trimmed mean 9.29848737266, raw times 9.294628507659999,9.27442990966,9.32640370066,9.266159457659999,9.33784911966

Leaderboard

| # | Result (m:s.ms) | Implementation     | JDK | Submitter     | Notes     |
|---|-----------------|--------------------|-----|---------------|-----------|
|   | 00:09.298 | [link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_tivrfoa.java)| 21.0.2-graal | [tivrfoa](https://github.com/tivrfoa) | GraalVM native binary, uses Unsafe |

[tivrfoa]

Yepp, here you go:

Validating calculate_average_tivrfoa.sh -- measurements_10K_1B.txt

+ rm -f measurements.txt
+ ln -s measurements_10K_1B.txt measurements.txt
Benchmark 1: timeout -v 300 ./calculate_average_tivrfoa.sh 2>&1
  Time (mean ± σ):      9.300 s ±  0.031 s    [User: 0.002 s, System: 0.004 s]
  Range (min … max):    9.266 s …  9.338 s    5 runs

Summary
  tivrfoa: trimmed mean 9.29848737266, raw times 9.294628507659999,9.27442990966,9.32640370066,9.266159457659999,9.33784911966

Leaderboard

| # | Result (m:s.ms) | Implementation     | JDK | Submitter     | Notes     |
|---|-----------------|--------------------|-----|---------------|-----------|
|   | 00:09.298 | [link](https://github.com/gunnarmorling/1brc/blob/main/src/main/java/dev/morling/onebrc/CalculateAverage_tivrfoa.java)| 21.0.2-graal | [tivrfoa](https://github.com/tivrfoa) | GraalVM native binary, uses Unsafe |

Thank you!

My idea completely failed. Trying to optimize using a different HW does not work ... :sweat_smile:

[gunnarmorling]

Haha, yeah, at the top it's a bit like walking in the dark if you're on a different ISA.

Message ID: @.***>