search

psf / pyperf

Toolkit to run Python benchmarks
http://pyperf.readthedocs.io/
MIT License
799 stars 78 forks source link

Output geometric mean to 0.1% precision #123

Open markshannon opened 2 years ago

markshannon commented 2 years ago

Outputs such as "0% slower" or "1% faster" are unnecessarily vague. Is "1%" 0.5% or 1.5%?

vstinner commented 2 years ago

Do you suggest to add one more digit in the output? Do you want to propose a PR for that?

sweeneyde commented 2 years ago

While 0.4% on individual benchmarks is probably not noteworthy, it's nice to see on the geometric mean of an entire suite. Very few changes will ever move the pyperformance geometric mean more than a couple of percentage points, so most of the time we see one of (1.01 slower, 1.00 slower, 1.00 faster, 1.01 faster, 1.02x faster), which isn't very much information IMO.

Another option would be to add three decimal places to pyperformance only (which doesn't display geometric mean at all yet).

ericsnowcurrently commented 2 years ago

While 0.4% on individual benchmarks is probably not noteworthy, it's nice to see on the geometric mean of an entire suite.

+1