search

psf / pyperf

Toolkit to run Python benchmarks
http://pyperf.readthedocs.io/
MIT License
799 stars 78 forks source link

Output geometric mean to 0.1% precision #126

Closed brandtbucher closed 1 year ago

brandtbucher commented 2 years ago

Closes #123.

brandtbucher commented 2 years ago

CC @vstinner

vstinner commented 2 years ago

Do you really care of a difference smaller than 1%?

Could it be an option? I would prefer to display least digits as possible for the most common use cases.

vstinner commented 2 years ago

If possible, I would prefer to keep 2 digits after the dot ("1.02x faster") for benchmarks. Otherwise, people will fight to death for a digit which is not significant and means nothing. People often misunderstand and misread benchmark results. Maybe pyperf default output should be simplified to not digits :-D Only say: faster, slower or not significant ;-)

By the way, IMO pyperf should say "not significant" in more cases ;-)

But I'm open to the idea of adding a 3rd digit after the dot for the geometric mean which cumulates results of multiple benchmarks.

pyperf stores all data in JSON. pyperf can have many options to change the formatting of the output. Maybe it can be a command line option to show "more digits". How many? I don't know.

I am fine with adding options, people are free to use data as they want. I mostly care about keeping the default formatting unchanged to not open discussions about the least significant digit which is usually not significant.

It's hard to draw a line between "significant digit" and "not significant digit". It depends on the duration of a single benchmark run, the kind of benchmark, etc. pyperf knows little about the benchmark.