search

juju-solutions / layer-cwr

Layer for building the Juju Jenkins CI env
Other
2 stars 5 forks source link

Add support for benchmarks #106

Open johnsca opened 7 years ago

johnsca commented 7 years ago

CWR supports running benchmarks in addition to the tests, but we need a way to specify what benchmarks should be run. This could be done as action args or in the bundle's tests.yaml.

arosales commented 7 years ago

+1 the benchmarks bring a lot of value to the report in a CI context. Would be great to see how a commit affects performance. I don't think a lot of CI systems make this available in an automated repeatable fashion.

Ideally, we could add test-plan.yaml support into bundletester's tests/tests.yaml and define benchmarks in the tests.yaml.

Shorter term we could just assume unit 0 of the named application without parameters, and just have the user pass in the benchmark name.

kwmonroe commented 7 years ago

Super +1 on supporting benchmarks in tests.yaml. If we did that, this could be a pretty simple change to cwr's run_benchmarks function (around https://github.com/juju-solutions/cloud-weather-report/blob/master/cloudweatherreport/run.py#L243):

    for benchmark_plan in test_plan.benchmarks:
    ...

becomes

    if test_plan.benchmarks:
        benchmark_yaml = test_plan.benchmarks
    else:
        tests_yaml = ## read test_plan.bundle's tests.yaml ##
        benchmark_yaml = tests_yaml.benchmarks

    for benchmark_plan in benchmark_yaml:
    ....

And if that happens, benchmarking in layer-cwr comes for free.

arosales commented 7 years ago

One of the key advantages of CWR is benchmarks. Currently, anyone wanting to use CWR will not be able to see benchmark results.

Thus, if we would like to see adoption by other non-Canonical communities this should be a bug we fix asap.

-thanks, Antonio