Open johnsca opened 7 years ago
+1 the benchmarks bring a lot of value to the report in a CI context. Would be great to see how a commit affects performance. I don't think a lot of CI systems make this available in an automated repeatable fashion.
Ideally, we could add test-plan.yaml support into bundletester's tests/tests.yaml and define benchmarks in the tests.yaml.
Shorter term we could just assume unit 0 of the named application without parameters, and just have the user pass in the benchmark name.
Super +1 on supporting benchmarks in tests.yaml
. If we did that, this could be a pretty simple change to cwr's run_benchmarks
function (around https://github.com/juju-solutions/cloud-weather-report/blob/master/cloudweatherreport/run.py#L243):
for benchmark_plan in test_plan.benchmarks:
...
becomes
if test_plan.benchmarks:
benchmark_yaml = test_plan.benchmarks
else:
tests_yaml = ## read test_plan.bundle's tests.yaml ##
benchmark_yaml = tests_yaml.benchmarks
for benchmark_plan in benchmark_yaml:
....
And if that happens, benchmarking in layer-cwr comes for free.
One of the key advantages of CWR is benchmarks. Currently, anyone wanting to use CWR will not be able to see benchmark results.
Thus, if we would like to see adoption by other non-Canonical communities this should be a bug we fix asap.
-thanks, Antonio
CWR supports running benchmarks in addition to the tests, but we need a way to specify what benchmarks should be run. This could be done as action args or in the bundle's
tests.yaml
.