search

tensor-compiler / array-programming-benchmarks

MIT License
3 stars 0 forks source link

(optimization): see if we can use fixtures to avoid reading tensor files multiple times #14

Closed rohany closed 3 years ago

rohany commented 3 years ago

We currently use google-benchmark with 10 repetitions and 1 iteration so that google-benchmark reports mean/median and deviations for us. However, this results in executing the benchmark function 10 times. This means that we execute the same file i/o 10 times, when it could be done just once.

rohany commented 3 years ago

My commits should have helped this substantially. A similar approach can be used by other benchmarks that are feeling this pain.

stephenchouca commented 3 years ago

What was the reason for not just using google-benchmark with 1 repetition and 10 iterations? Doesn't google-benchmark still report aggregate statistics even in that case?

stephenchouca commented 3 years ago

What was the reason for not just using google-benchmark with 1 repetition and 10 iterations? Doesn't google-benchmark still report aggregate statistics even in that case?

Actually never mind, I misunderstood how google-benchmark works.

rohany commented 3 years ago

Yeah it's a little annoying. google bench (and other frameworks) report only the average of the iterations, since normally the number of iterations is dynamically controlled to find an appropriate amount for a micro benchmark. with repetitions it will report median and stddev.