gunnarmorling / 1brc

1️⃣🐝🏎️ The One Billion Row Challenge -- A fun exploration of how quickly 1B rows from a text file can be aggregated with Java
https://www.morling.dev/blog/one-billion-row-challenge/
Apache License 2.0
6.23k stars 1.87k forks source link

1brc using concurrent hash map #587

Closed vikasmb closed 9 months ago

vikasmb commented 9 months ago

Check List:

gunnarmorling commented 9 months ago

Tests don't pass (see CI for details).

vikasmb commented 9 months ago

@gunnarmorling Can you please review this again ? With my latest changes pushed to this PR, test_ci.sh passes in my machine:

./test_ci.sh vikasmb
+ source ./prepare_vikasmb.sh
+ gtimeout -v 300 ./test.sh vikasmb
Validating calculate_average_vikasmb.sh -- src/test/resources/samples/measurements-1.txt
Validating calculate_average_vikasmb.sh -- src/test/resources/samples/measurements-10.txt
Validating calculate_average_vikasmb.sh -- src/test/resources/samples/measurements-10000-unique-keys.txt
Validating calculate_average_vikasmb.sh -- src/test/resources/samples/measurements-2.txt
Validating calculate_average_vikasmb.sh -- src/test/resources/samples/measurements-20.txt
Validating calculate_average_vikasmb.sh -- src/test/resources/samples/measurements-3.txt
Validating calculate_average_vikasmb.sh -- src/test/resources/samples/measurements-boundaries.txt
Validating calculate_average_vikasmb.sh -- src/test/resources/samples/measurements-complex-utf8.txt
Validating calculate_average_vikasmb.sh -- src/test/resources/samples/measurements-dot.txt
Validating calculate_average_vikasmb.sh -- src/test/resources/samples/measurements-rounding.txt
Validating calculate_average_vikasmb.sh -- src/test/resources/samples/measurements-short.txt
Validating calculate_average_vikasmb.sh -- src/test/resources/samples/measurements-shortest.txt

echo $?
0
gunnarmorling commented 9 months ago

Fails when running on the eval machine (on 32 cores, I suppose this messes up the chunking):

+ timeout -v 300 ./test.sh vikasmb
Validating calculate_average_vikasmb.sh -- src/test/resources/samples/measurements-10000-unique-keys.txt
Validating calculate_average_vikasmb.sh -- src/test/resources/samples/measurements-10.txt
Validating calculate_average_vikasmb.sh -- src/test/resources/samples/measurements-1.txt
Validating calculate_average_vikasmb.sh -- src/test/resources/samples/measurements-20.txt
Exception in thread "main" java.lang.IndexOutOfBoundsException
    at java.base/java.nio.Buffer$1.apply(Buffer.java:757)
    at java.base/java.nio.Buffer$1.apply(Buffer.java:754)
    at java.base/jdk.internal.util.Preconditions$4.apply(Preconditions.java:213)
    at java.base/jdk.internal.util.Preconditions$4.apply(Preconditions.java:210)
    at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:98)
    at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:106)
    at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:302)
    at java.base/java.nio.Buffer.checkIndex(Buffer.java:768)
    at java.base/java.nio.DirectByteBuffer.get(DirectByteBuffer.java:358)
    at dev.morling.onebrc.CalculateAverage_vikasmb$FileChunker.adjustPos(CalculateAverage_vikasmb.java:123)
    at dev.morling.onebrc.CalculateAverage_vikasmb$FileChunker.initChunks(CalculateAverage_vikasmb.java:108)
    at dev.morling.onebrc.CalculateAverage_vikasmb$FileChunker.<init>(CalculateAverage_vikasmb.java:89)
    at dev.morling.onebrc.CalculateAverage_vikasmb.main(CalculateAverage_vikasmb.java:50)
gunnarmorling commented 9 months ago

Closing this one, as I haven't heard back and we're after the cut-off time. Thanks a lot for participating nevertheless, @vikasmb!