gunnarmorling / 1brc

1️⃣🐝🏎️ The One Billion Row Challenge -- A fun exploration of how quickly 1B rows from a text file can be aggregated with Java
https://www.morling.dev/blog/one-billion-row-challenge/
Apache License 2.0
5.69k stars 1.73k forks source link

My Probably last attempt to optimize performance #693

Closed PanagiotisDrakatos closed 5 months ago

PanagiotisDrakatos commented 5 months ago

Check List:

PanagiotisDrakatos commented 5 months ago

@gunnarmorling please check again i believe this is my last attempt to improve performance after that my capabilities are limited

gunnarmorling commented 5 months ago

Runs out of heap space for the tests on 32 cores:

Caused by: java.lang.OutOfMemoryError: Java heap space
    at dev.morling.onebrc.CalculateAverage_PanagiotisDrakatos$MeasurementRepository.<init>(CalculateAverage_PanagiotisDrakatos.java:274)
    at dev.morling.onebrc.CalculateAverage_PanagiotisDrakatos.MappingByteBufferToData(CalculateAverage_PanagiotisDrakatos.java:164)
    at dev.morling.onebrc.CalculateAverage_PanagiotisDrakatos$$Lambda/0x00007f6b43001860.apply(Unknown Source)
    at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
    at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
    at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708)
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
    at java.base/java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:960)
    at java.base/java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:934)
    at java.base/java.util.stream.AbstractTask.compute(AbstractTask.java:327)
    at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:754)
    at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
    at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
    at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
    at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
    at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)
PanagiotisDrakatos commented 5 months ago

@gunnarmorling i fixed VM options arguments and increased the xmx. Probably it would not failed now

gunnarmorling commented 5 months ago

Looking good now: 00:05.559. Passing for 10K keys, too.