search

duckdblabs / db-benchmark

reproducible benchmark of database-like ops
https://duckdblabs.github.io/db-benchmark/
Mozilla Public License 2.0
136 stars 27 forks source link

many solutions are missing from join 1e9 #57

Closed jangorecki closed 8 months ago

jangorecki commented 8 months ago

It is a bit confusing that join 1e9 has so few solutions. There are no notes below the benchplot explaining that. Ideally is to have all solutions included, and whenever something is not working, then exception defined in benchplot dictionary that refers to gh issues #num of particular project. You will find multiple examples in current benchplot dictionary because that was the way we handled failures and exceptions in different solutions. This way you are also allowing users to eventually fix the problems that are behind particular issues.

Tmonster commented 8 months ago

Thinking about it over the weekend you are right. I had been working on updating the benchmark/using the new machine/new PR process etc. and overlooked a few things. I had just been noticing that many solutions could not finish the 50GB join and I did not want to have the new machine running for a long amount of time just to test solutions that would most likely fail or reach the timeout for the 50GB join queries. I thought I mentioned it in the blog post, but I think I eventually edited it out. Apologies for this.

However, this doesn't mean failures should not be visible. For this reason I will run all the of the solutions on the 50GB join. To avoid having the machine for a long time however, I will also lower the 50GB join timeout from 180 minutes to 90 minutes.

90 minutes was chosen since collapse was able to finish all 50GB join solutions in ~900 seconds or 15 minutes. I added an extra hour for a buffer and to accommodate the cooldown period. After that I rounded up to the nearest 30 minute interval

jangorecki commented 8 months ago

Great to hear that. When changing machine, there is a possibility that solutions that have failed before, will complete this time, so also make sense to ensure if it still fails.

Main run.sh script and its launcher underneath were designed to run only solutions that has need version (git commit), unless force_run=true, so you may want to run everything again, and only "new" ones will be actually started (logs.csv is tracking exactly that). You can as well use mockup=true, which will not run any benchmark but will print which scripts would be normally run (when mockup=false). Those flags can be useful to reduce maintenance related to switching on and off different solutions, different tasks, different data sizes/cases.

I also noticed that join 1e9 has only one data case, while other join sizes has 3 cases (having NAs and pre-sorted one).

sl-solution commented 8 months ago

@Tmonster would you please make sure that my recent PR (#55 ) is merge before running the join task?

Tmonster commented 8 months ago

@sl-solution yes will do

Tmonster commented 8 months ago

Results have been updated. Nice work @sl-solution!