apache / datafusion

Apache DataFusion SQL Query Engine
https://datafusion.apache.org/
Apache License 2.0
5.92k stars 1.12k forks source link

Add H2O.ai Database-like Ops benchmark to `dfbench` #7209

Open alamb opened 1 year ago

alamb commented 1 year ago

Is your feature request related to a problem or challenge?

Follow on to #7052 There is an interesting database benchark called "H20.ai database like benchmark" that DuckDB seems to have revived (perhaps because the original went dormant with very old with very old/ slow duckdb results). More background here: https://duckdb.org/2023/04/14/h2oai.html#results

@Dandandan added a new solution for datafusion here: https://github.com/duckdblabs/db-benchmark/pull/18

However, there is no easy way to run the h2o benchmark within the datafusion repo. There is an old version of some of these benchmarks in the code: https://github.com/apache/arrow-datafusion/blob/main/benchmarks/src/bin/h2o.rs

Describe the solution you'd like

I would like someone to make it easy to run the h20.ai benchmark in the datafusion repo.

Ideally this would look like

# generate data
./benchmarks/bench.sh data h20.ai
# run 
./benchmarks/bench.sh run h20.ai

I would expect to be able to run the individual queries like this

cargo run  --bin dfbench -- h2o.ai --query=3

Some steps might be

  1. port the existing benchmark script to dfbench following the model in https://github.com/apache/arrow-datafusion/pull/7120
  2. update bench.sh, following the model of existing benchmarks
  3. Update the documentation

Describe alternatives you've considered

We could also simply remove the h20.ai benchmark script as it is not clear how important it will be long term

Additional context

I think this is a good first issue as the task is clear, and there are existing patterns in bench.sh, dfbench and in

palash25 commented 1 year ago

I would like to work on this

alamb commented 1 year ago

Thank you @palash25

palash25 commented 1 year ago

sorry for the inactivity on this. my RSI came back so i was taking a break from typing, i will try to submit the PR in a day or two.

alamb commented 1 year ago

No problem -- I hope you feel better soon

drewhayward commented 1 month ago

Is this something that's still wanted? I took a look at doing this but it looks like the data isn't hosted on the benchmark repo, just data gen scripts in R.

alamb commented 1 month ago

Is this something that's still wanted? I took a look at doing this but it looks like the data isn't hosted on the benchmark repo, just data gen scripts in R.

I think it would be useful. Thank you

I think figuring out how to generate the data locally would be super valuable -- perhaps we can use a docker like approach as we do for tpch:

https://github.com/apache/datafusion/blob/89677ae66ec2238ffd58de8c36fe6030fad458fa/benchmarks/bench.sh#L286-L292

So it would run like

./bench.sh data h2o

Which would leave data in datafusion/benchmarks/data/h2o

🤔