search

gunnarmorling / 1brc

1️⃣🐝🏎️ The One Billion Row Challenge -- A fun exploration of how quickly 1B rows from a text file can be aggregated with Java
https://www.morling.dev/blog/one-billion-row-challenge/
Apache License 2.0
6.07k stars 1.83k forks source link

Suggestion for the next time #528

Open balayanv opened 7 months ago

balayanv commented 7 months ago

Reading some of the best submissions i think a lot of clever tricks people do never come to play (or even harm the numbers) because C2 will not optimise it the way they intended, imo for future submissions, top N competing solutions that get very close (instead of sharing 1st place among 3 people) should run their solutions with JMH and get more realistic and consistent numbers. Granted JMH will keep those hetzner instances sweating for a long time but it's only run for top say 5 submissions where margin of error makes it impossible to evaluate numbers with just 5 runs. Current eval method could simply act as filter and JMH invoked only on the best ones.

Love the challenge though, i've found lots of interesting crazy tricks i didn't know about.

gunnarmorling commented 7 months ago

Hey, thx for opening this one. Agreed that a JMH benchmark of the top 10 or so would be great, but I'm not sure whether I have the capacity to do so right now. If someone provided the actual benchmark harness/suite so that I can simply run it on the eval machine (or elsewhere), that would be amazing.

balayanv commented 7 months ago

If no one does before next weekend, i'll contribute one, should be trivial. Though i have not configured it for maven before.

gunnarmorling commented 7 months ago

That would be awesome, thanks! The challenge closes on Jan 31, i.e. it would be right on time then to get those JMH benchmarks for the top contenders.