search

IO500 / io500

IO500 Storage Benchmark source code
MIT License
95 stars 30 forks source link

Disabling one step of the benchmark zeroes out the score #42

Closed michaelb closed 2 years ago

michaelb commented 2 years ago

I'm trying to run the benchmark with one particular step disabled, with the key

RUN = FALSE

but while the step does indeed not run, it gives out a score of 0, which in turn zeroes out the geometric mean of the score aggregation. I guess that's mathematically correct, but it doesn't make much sense for as long as I'm not trying to submit a score but rather bench a system to test ameliorations

JulianKunkel commented 2 years ago

I see the point. We could skip the stage completely if you favor this.

michaelb commented 2 years ago

That would be an effective way to fix it, thanks for the quick reply!

Also, marking the final score as 'INVALID' if a step was skipped is not my decision to take, but maybe someone want to think/document about what 'makes sense'