search

smarr / ReBench

Execute and document benchmarks reproducibly.
MIT License
83 stars 24 forks source link

Rethink composition of experiments and value precedence #169

Open smarr opened 2 years ago

smarr commented 2 years ago

At the moment, we compose experiments/runIs bottom up, with the highest elements having priority. See https://rebench.readthedocs.io/en/latest/config/#priority-of-configuration-elements

Though, in some cases this doesn't seem to be very useful.

For instance, if I want to override the number of iterations that a benchmark is supposed to run, I can't, if it's a per-benchmark setting, which can be useful for large, diverse benchmark suites.

So, if I want to limit the number of iterations for a specific experiment, I need a separate suite definition, as I have been doing for many of my configurations.

Though, things might be more reusable, if priority would work the other way around.

Or, if I could mark a specific value as important. Perhaps similar to https://developer.mozilla.org/en-US/docs/Web/CSS/Specificity#the_!important_exception

smarr commented 2 years ago

A first version is implemented with https://github.com/smarr/ReBench/pull/170, but it's only partial.