Improvements to Benchmark Scripts and Config Generation Workflow

[fabianlim]

Improvements to benchmarks

From now on, all benchmarks need to run in tox environment for package version hygiene.

tox -e run_benches

• note, since the PR is not yet merged we need to set FHT_BRANCH=accel-pr

In addition

• the tox command above accepts environment variables DRY_RUN, NO_DATA_PROCESSING, NO_OVERWRITE. See scripts/run_benchmarks.sh
• in run_benchmarks.sh we will clear the RESULT_DIR if it exists, to avoid contaimination with old results. To protect against overwrite, then always run with NO_OVERWRITE=true.
• also the run_benchmarks.sh script will produce two CSVs now.
  • raw_summary.csv: this is the original one previously called summary.csv. This one will contain the raw results
  • benchmarks.csv: this one is a processed version of raw_summary.csv. it only contains that columns that differ for easier viewing.
  • TODO: consider providing one more file for columns that are the same
• in run_benchmarks.sh we also pip freeze a requirements.txt file inside the tox environment.
  • we should check in this file as well
  • TODO: does tox have a version lock file?

Script for Producing CSV report

After running a few benchmarks, we can gather all the results into a single CSV report.

• can be done in incremental manner, even when the benches are still running
• works with multiple benchmark directories, just specify them one after another

# do the following in the repo directory
# activate the tox environment
source .tox/run-benches/bin/activate

# run the display-bench-results.py on a directory with benchmark results 
# - say "benchmark_outputs"
PYTHONPATH= python scripts/benchmarks/display-bench-results.py benchmark_outputs

This will produce an output like this, and then the .csv report can be read by pandas.read_csv

***************** Report Created ******************
Total lines: '48'
Number columns included: '20'
Number columns excluded: '20'
Excluding number of exceptions caught: '0'
Written report to 'results.csv'

Improvements to Generate Configs

We added a new tox -e verify-configs to ensure that the configs are correctly generated.

• this is now enabled as a workflow.
• this will run tox -e gen-configs as a worflow, and test against the file that was checked in. If it differs it will fail the build.
• This ensures that the sample-configuratinos are always up-to-date with the plugin configs