search

trinodb / benchto

Framework for running macro benchmarks in a clustered environment
Apache License 2.0
28 stars 29 forks source link

Add repeatLevel parameter to control what part of benchmark should be repeated #60

Closed radek-kondziolka closed 1 year ago

radek-kondziolka commented 1 year ago

The parameter repeatLevel adds possibility to choose whether benchmark or particular query should be repeated i.e. run multiple times to make an measurement. This parameter accepts two values: BENCHMARK and QUERY.

nineinchnick commented 1 year ago

As I understand, we noticed an unexpected change in results and need to be able to restore the previous behavior until we can find the root cause. I wouldn't call this solution temporary, but we agree that repeating single queries should be deprecated in the future.

I think #61 is a better solution. Adding a new parameter for repeating queries, without other side effects, is better than adding a parameter that changes the behavior of another parameter. I also agree that naming any new parameter is hard, and it's not easy for Benchto users to figure out the execution model and how it's related to benchmark definitions (yaml files). I think we agree we can't improve this now. Let's move forward with either one of those PRs.

radek-kondziolka commented 1 year ago

For 6 consecutive benchmarks (tpch, orc_part, sf1000) the maximal difference between two of them was 11% cpu time.