HTML reports

[nirupbbnk]

Feature request Hi Team , When we use pytest benchmark with CI solutions like Jenkins , If we have option of saving results in HTML or exporting the results into image will be easier for end user to see results. Currently we need to go through console output to see results or export it to json (looking through json looks cumbersome ). So if we have option to export it to HTML will add value I believe. Tried with https://pytest-html.readthedocs.io/en/latest/user_guide.html , but doesn't contain much info for benchmark results.

[whyzdev]

A custom way to do this with pytest-html is by using the pytest_html_results_summary hook to read the saved benchmark json file. The following example shows how this may work already before potential enhancement.

Put this function in conftest.py under the test package,

import json
from glob import glob
import pandas as pd

def pytest_html_results_summary(prefix, summary, postfix):
    with open(sorted(glob('.benchmarks/**/*_Saved.json'))[-1]) as f:
        bmk_dict = json.open(f)
        bmk_df = pd.json_normalize(bmk_dict['benchmark'])
        # selected/reordered list from bmk_df.columns
        columns = ['group', 'name'] + [col for col in bmk_df.columns if col.startswith('stats')]         
        postfix.extend(bmk_dict['machine_info'], bmk_df[columns])

And run tests like this:

pytest --benchmark-only --benchmark-saved=Saved --html=test_output.html --self-contained-html tests/

Then check the html output file, and see if there is something in the console such as errors due to dependencies, or if I had typo above.