psf / pyperf

Toolkit to run Python benchmarks
http://pyperf.readthedocs.io/
MIT License
799 stars 78 forks source link

Add geometric mean #76

Closed vstinner closed 3 years ago

vstinner commented 4 years ago

There are two approaches to compute a geometric mean:

scipy uses log+exp: https://github.com/scipy/scipy/blob/bf3ea59982a552e65f0388d4dd17b256b962adbb/scipy/stats/stats.py#L349-L411

The geometric mean should help to compute N benchmarks with a single value.

For example, https://speed.pypy.org/ says "The geometric average of all benchmarks is 0.23 or 4.3 times faster than cpython."

vstinner commented 4 years ago

My use case is to compute a geometric mean for pyperformance to avoid comparing N benchmark results manually, ignoring some benchmark with an absolute difference lower than N percent (ex: 5%).

vstinner commented 4 years ago

compute the product and then the nth root

If we pick this approach, math.prod() added to Python 3.8 can be used.

So log+exp seems better to avoid overflow. See also https://bugs.python.org/issue41458 "Avoid overflow/underflow in math.prod()".

vstinner commented 4 years ago

I wrote PR #79.

vstinner commented 3 years ago

It's now implemented in the master branch: fiexd by PR #79.