search

cloud-bulldozer / benchmark-wrapper

Python Library to run benchmarks
https://benchmark-wrapper.readthedocs.io
Apache License 2.0
19 stars 56 forks source link

Add Unit and Documentation Build Tests #306

Closed learnitall closed 3 years ago

learnitall commented 3 years ago

Description

Adds functionality for unit testing by leveraging pytest and tox. I believe that our goal for unit testing with benchmark-wrapper is to test the core functionality that is shared among our benchmark wrappers, as testing the wrappers themselves should be left to functional testing.

Here is how it works:

I've configured tox with eight different environments, allowing us to test across four versions of Python (3.6-3.9) and against unit and documentation tests: py{36,37,38,39}-{unit,docs}. For example, to run unit tests against Python 3.6, run: tox -e py36-unit. To run documentation build tests against Python 3.8, run tox -e py38-docs. Running tox without any arguments will tell tox to attempt to run all eight environments, ignoring any unavailable Python versions.

Coverage reports are saved as .coverage.<tox environment>. Configuration file is under .coveragerc and adds ignores for all benchmark-related code.

Additionally, I've added GH Actions workflows with will automatically run these tests on pull-requests, pushes or on a manual trigger. Coverage reports from the unit test workflow will automatically be uploaded to codecov, so we can share and review coverage reports in detail.

learnitall commented 3 years ago

I haven't included Python 3.9 in the GHA workflows due to this bug: https://github.com/cloud-bulldozer/benchmark-wrapper/pull/302

codecov-commenter commented 3 years ago

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@ec26c28). Click here to learn what that means. The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff            @@
##             master     #306   +/-   ##
=========================================
  Coverage          ?   42.69%           
=========================================
  Files             ?       14           
  Lines             ?      876           
  Branches          ?        0           
=========================================
  Hits              ?      374           
  Misses            ?      502           
  Partials          ?        0
Flag Coverage Δ
gha 42.69% <0.00%> (?)
python-3.6 42.69% <0.00%> (?)
unit 42.69% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update ec26c28...6e6c26e. Read the comment docs.

comet-perf-ci commented 3 years ago

Results for SNAFU CI Test

Test Result Runtime
snafu/hammerdb PASS 00:05:45
snafu/smallfile_wrapper PASS 00:05:41
snafu/flent_wrapper PASS 00:08:52
snafu/scale_openshift_wrapper PASS 00:11:29
snafu/sysbench PASS 00:01:21
snafu/benchmarks/uperf PASS 00:25:40
snafu/image_pull_wrapper PASS 00:03:47
snafu/ycsb_wrapper PASS 00:04:37
snafu/stressng_wrapper PASS 00:03:04
snafu/fio_wrapper FAIL 00:13:35
snafu/log_generator_wrapper PASS 00:04:00
snafu/upgrade_openshift_wrapper PASS 00:00:00
snafu/iperf PASS 00:05:51
snafu/vegeta_wrapper PASS 00:06:48
snafu/fs_drift_wrapper PASS 00:05:22
snafu/pgbench_wrapper PASS 00:04:09
learnitall commented 3 years ago

/rerun all

comet-perf-ci commented 3 years ago

Results for SNAFU CI Test

Test Result Runtime
snafu/hammerdb PASS 00:05:57
snafu/smallfile_wrapper PASS 00:05:41
snafu/flent_wrapper PASS 00:09:04
snafu/scale_openshift_wrapper PASS 00:09:38
snafu/sysbench PASS 00:01:22
snafu/benchmarks/uperf PASS 00:25:27
snafu/image_pull_wrapper PASS 00:03:41
snafu/ycsb_wrapper PASS 00:04:32
snafu/stressng_wrapper PASS 00:03:07
snafu/fio_wrapper FAIL 00:13:30
snafu/log_generator_wrapper PASS 00:03:58
snafu/upgrade_openshift_wrapper PASS 00:00:00
snafu/iperf PASS 00:05:44
snafu/vegeta_wrapper PASS 00:05:45
snafu/fs_drift_wrapper PASS 00:05:25
snafu/pgbench_wrapper PASS 00:04:04
comet-perf-ci commented 3 years ago

Results for SNAFU CI Test

Test Result Runtime
snafu/hammerdb PASS 00:06:09
snafu/smallfile_wrapper PASS 00:05:43
snafu/flent_wrapper PASS 00:09:28
snafu/scale_openshift_wrapper PASS 00:10:13
snafu/sysbench PASS 00:02:45
snafu/benchmarks/uperf PASS 00:26:32
snafu/image_pull_wrapper PASS 00:03:45
snafu/ycsb_wrapper PASS 00:04:48
snafu/stressng_wrapper PASS 00:03:11
snafu/fio_wrapper FAIL 00:13:28
snafu/log_generator_wrapper PASS 00:04:08
snafu/upgrade_openshift_wrapper PASS 00:00:00
snafu/iperf PASS 00:05:51
snafu/vegeta_wrapper PASS 00:05:22
snafu/fs_drift_wrapper PASS 00:05:41
snafu/pgbench_wrapper PASS 00:04:06
comet-perf-ci commented 3 years ago

Results for SNAFU CI Test

Test Result Runtime
snafu/hammerdb PASS 00:06:27
snafu/smallfile_wrapper PASS 00:05:51
snafu/flent_wrapper PASS 00:09:37
snafu/scale_openshift_wrapper PASS 00:10:32
snafu/sysbench PASS 00:02:33
snafu/benchmarks/uperf PASS 00:25:58
snafu/image_pull_wrapper PASS 00:03:51
snafu/ycsb_wrapper PASS 00:04:51
snafu/stressng_wrapper PASS 00:03:09
snafu/fio_wrapper FAIL 00:14:51
snafu/log_generator_wrapper PASS 00:03:48
snafu/upgrade_openshift_wrapper PASS 00:00:00
snafu/iperf PASS 00:05:40
snafu/vegeta_wrapper PASS 00:05:04
snafu/fs_drift_wrapper PASS 00:05:14
snafu/pgbench_wrapper PASS 00:04:08
comet-perf-ci commented 3 years ago

Results for SNAFU CI Test

Test Result Runtime
snafu/hammerdb PASS 00:05:51
snafu/smallfile_wrapper PASS 00:05:55
snafu/flent_wrapper PASS 00:09:14
snafu/scale_openshift_wrapper PASS 00:10:21
snafu/sysbench PASS 00:02:28
snafu/benchmarks/uperf PASS 00:25:46
snafu/image_pull_wrapper PASS 00:03:40
snafu/ycsb_wrapper PASS 00:04:40
snafu/stressng_wrapper PASS 00:03:05
snafu/fio_wrapper FAIL 00:13:42
snafu/log_generator_wrapper PASS 00:03:57
snafu/upgrade_openshift_wrapper PASS 00:00:00
snafu/iperf PASS 00:05:45
snafu/vegeta_wrapper PASS 00:05:00
snafu/fs_drift_wrapper PASS 00:05:26
snafu/pgbench_wrapper PASS 00:03:59
comet-perf-ci commented 3 years ago

Results for SNAFU CI Test

Test Result Runtime
snafu/hammerdb PASS 00:05:52
snafu/smallfile_wrapper PASS 00:05:51
snafu/flent_wrapper PASS 00:11:14
snafu/scale_openshift_wrapper PASS 00:10:45
snafu/sysbench PASS 00:02:36
snafu/benchmarks/uperf FAIL 00:18:52
snafu/image_pull_wrapper PASS 00:03:46
snafu/ycsb_wrapper PASS 00:04:41
snafu/stressng_wrapper PASS 00:03:11
snafu/fio_wrapper FAIL 00:13:36
snafu/log_generator_wrapper PASS 00:03:54
snafu/upgrade_openshift_wrapper PASS 00:00:00
snafu/iperf PASS 00:05:42
snafu/vegeta_wrapper PASS 00:05:08
snafu/fs_drift_wrapper PASS 00:05:23
snafu/pgbench_wrapper PASS 00:04:07
comet-perf-ci commented 3 years ago

Results for SNAFU CI Test

Test Result Runtime
snafu/hammerdb PASS 00:06:11
snafu/smallfile_wrapper FAIL 00:04:09
snafu/flent_wrapper PASS 00:09:21
snafu/scale_openshift_wrapper FAIL 00:07:09
snafu/sysbench PASS 00:02:42
snafu/benchmarks/uperf FAIL 00:19:40
snafu/image_pull_wrapper PASS 00:03:47
snafu/ycsb_wrapper FAIL 00:05:14
snafu/stressng_wrapper FAIL 00:03:25
snafu/fio_wrapper FAIL 00:06:24
snafu/log_generator_wrapper PASS 00:03:57
snafu/upgrade_openshift_wrapper PASS 00:00:00
snafu/iperf PASS 00:05:48
snafu/vegeta_wrapper FAIL 00:03:45
snafu/fs_drift_wrapper FAIL 00:03:41
snafu/pgbench_wrapper FAIL 00:04:29
comet-perf-ci commented 3 years ago

!!! Couldn't read commit file !!!

comet-perf-ci commented 3 years ago

!!! Couldn't read commit file !!!

comet-perf-ci commented 3 years ago

Results for SNAFU CI Test

Test Result Runtime
snafu/hammerdb PASS 00:06:54
snafu/smallfile_wrapper PASS 00:05:57
snafu/flent_wrapper PASS 00:12:27
snafu/scale_openshift_wrapper PASS 00:10:18
snafu/sysbench PASS 00:03:10
snafu/benchmarks/uperf PASS 00:28:58
snafu/image_pull_wrapper PASS 00:03:09
snafu/ycsb_wrapper FAIL 00:11:58
snafu/stressng_wrapper PASS 00:03:49
snafu/fio_wrapper PASS 00:17:54
snafu/log_generator_wrapper PASS 00:03:14
snafu/upgrade_openshift_wrapper PASS 00:00:00
snafu/iperf PASS 00:06:15
snafu/vegeta_wrapper FAIL 00:04:09
snafu/fs_drift_wrapper PASS 00:04:58
snafu/pgbench_wrapper PASS 00:04:10
comet-perf-ci commented 3 years ago

Results for SNAFU CI Test

Test Result Runtime
snafu/hammerdb PASS 00:06:27
snafu/smallfile_wrapper PASS 00:04:55
snafu/flent_wrapper PASS 00:10:11
snafu/scale_openshift_wrapper PASS 00:10:56
snafu/sysbench PASS 00:03:11
snafu/benchmarks/uperf PASS 00:26:43
snafu/image_pull_wrapper PASS 00:03:20
snafu/ycsb_wrapper FAIL 00:11:59
snafu/stressng_wrapper PASS 00:03:35
snafu/fio_wrapper PASS 00:18:19
snafu/log_generator_wrapper PASS 00:03:14
snafu/upgrade_openshift_wrapper PASS 00:00:00
snafu/iperf PASS 00:06:05
snafu/vegeta_wrapper PASS 00:04:06
snafu/fs_drift_wrapper PASS 00:04:41
snafu/pgbench_wrapper PASS 00:04:12
comet-perf-ci commented 3 years ago

Results for SNAFU CI Test

Test Result Runtime
snafu/hammerdb PASS 00:06:51
snafu/smallfile_wrapper PASS 00:05:28
snafu/flent_wrapper PASS 00:11:22
snafu/scale_openshift_wrapper PASS 00:10:28
snafu/sysbench PASS 00:03:22
snafu/benchmarks/uperf PASS 00:26:47
snafu/image_pull_wrapper PASS 00:03:03
snafu/ycsb_wrapper FAIL 00:11:54
snafu/stressng_wrapper PASS 00:03:44
snafu/fio_wrapper PASS 00:18:59
snafu/log_generator_wrapper PASS 00:03:23
snafu/upgrade_openshift_wrapper PASS 00:00:00
snafu/iperf PASS 00:06:13
snafu/vegeta_wrapper PASS 00:04:21
snafu/fs_drift_wrapper PASS 00:04:48
snafu/pgbench_wrapper PASS 00:04:12