search

ionelmc / pytest-benchmark

py.test fixture for benchmarking code
BSD 2-Clause "Simplified" License
1.24k stars 119 forks source link

Description of output table #179

Closed sutarnilesh closed 4 years ago

sutarnilesh commented 4 years ago

I'm finding difficulties in understanding the output of table.

------------------------------------------------------------------------------------- benchmark 'mytest-fig': 2 tests --------------------------------------------------------------------------------------
pytest-benchmark-tests-fig_1        | Name (time in ms)                                    Min                 Max              Mean            StdDev            Median               IQR       Outliers       OPS            Rounds  Iterations
pytest-benchmark-tests-fig_1        | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
pytest-benchmark-tests-fig_1        | test_fig (NOW)                                1.0080 (1.0)      230.8245 (1.0)      1.5488 (1.14)     3.3283 (1.0)      1.4615 (1.19)     0.1965 (2.45)        3;312  645.6711 (0.87)      10000           5
pytest-benchmark-tests-fig_1        | test_fig (/tests_benchmark/baselines/fig)     1.1627 (1.15)     248.4515 (1.08)     1.3527 (1.0)      3.6721 (1.10)     1.2301 (1.0)      0.0802 (1.0)         7;974  739.2518 (1.0)       10000           5
pytest-benchmark-tests-fig_1        | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

What it means by values in () eg. 1.5488 (1.14) in first row? What is outliers? Also there is significant amount of difference in Min and Max. What is causing that?

ionelmc commented 4 years ago

Teteudvughrudjeufhw HdueufjjeifihrjdHEIOFIDUDHEIFDHHEJDHXBDJJWJDBDMMSJFHRJF

On Mon, Sep 21, 2020, 17:49 Nilesh Sutar notifications@github.com wrote:

I'm finding difficulties in understanding the output of table.

benchmark 'mytest-fig': 2 tests

pytest-benchmark-tests-fig_1 | Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations pytest-benchmark-tests-fig_1 |

pytest-benchmark-tests-fig_1 | test_fig (NOW) 1.0080 (1.0) 230.8245 (1.0) 1.5488 (1.14) 3.3283 (1.0) 1.4615 (1.19) 0.1965 (2.45) 3;312 645.6711 (0.87) 10000 5 pytest-benchmark-tests-fig_1 | test_fig (//tests_benchmark/baselines/fig) 1.1627 (1.15) 248.4515 (1.08) 1.3527 (1.0) 3.6721 (1.10) 1.2301 (1.0) 0.0802 (1.0) 7;974 739.2518 (1.0) 10000 5 pytest-benchmark-tests-fig_1 |

What it means by values in () eg. 1.5488 (1.14) in first row? What is outliers? Also there is significant amount of difference in Min and Max. What is causing that?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ionelmc/pytest-benchmark/issues/179, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAA7TXJFLVM3VIRH73YHTMLSG5RXJANCNFSM4RUUCALA .

ionelmc commented 4 years ago

Oooooof! Looks like my kid actually did something with my phone :)

The numbers in brackets are factors. In your table NOW is compared to /tests_benchmark/baselines/fig so for example in min column 1.0080 * 1.15 == 1.1627.

The outliers have two metrics, and are data points outside of these two partitions of the timings:

High min/max difference, or many outliers means that sometimes your tested function takes way more time than usual. It could mean that testing has outside interference (eg: you run your suite in a vm on a busy host, you have cpu turboboost/throtling going on etc). Also see https://pytest-benchmark.readthedocs.io/en/latest/faq.html

So then again what to do you ask? Well you have to consider what sort of function you have:

sutarnilesh commented 4 years ago

Thanks Ionel. Sounds good. The Outlier column has 2 values separated by ; may I know what they indicates? Also my benchmark service running in separates container, will underline host resources utilization affect performance? I noticed one more thing if there is significant difference in Max, Mean or StdDev, you can see difference in IQR and Outliers. Is this expected?

ionelmc commented 4 years ago

High max usually correlates with abnormal amount of outliers. The two numbers in the outliers column are exactly this:

The outliers have two metrics, and are data points outside of these two partitions of the timings:

  • 1 Standard Deviation from Mean
  • 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile
sutarnilesh commented 4 years ago

Closing this issue as my query was resolved!