Supporting Benchmark suites «à la JUnit»

[GoogleCodeExporter]

Would that be possible to provide an implementation of the interface Benchmark 
that enables to build suites of benchmarks that are expected to be run and 
reported together. Something like:

public class ArrayBenchmark extends BenchmarkSuite {
    public ArrayBenchmark() {
        addBenchmark(new ArraySortBenchmark("timeSort"));
        addBenchmark(new ArrayCopyBenchmark());
    }

    public static void main(String[] args) throws Exception {
        Runner.main(ArrayBenchmark.class, args);
}   }

Original issue reported on code.google.com by romain.r...@gmail.com on 12 Jun 2011 at 9:15

[GoogleCodeExporter]

Maybe; we have never felt the need for this yet, partly because it's weird that 
you wouldn't be able to use command-line arguments like -Dname=value, -b 
methodName, etc.

Original comment by kevinb@google.com on 14 Jun 2011 at 9:12

[GoogleCodeExporter]

My point is not about the parameter of ArraySortBenchmark, but rather that the 
documentation of caliper arguments says "<benchmark>: a benchmark class or 
suite", but I do not see how to create a benchmark suite for running several 
benchmark classes in a single run.

Original comment by romain.r...@gmail.com on 15 Jun 2011 at 6:10

[GoogleCodeExporter]

Yeah, that misleading language will be removed.  Benchmark classes containing 
benchmark methods, that's all we got right now.

Original comment by kevinb@google.com on 16 Jun 2011 at 12:31

[GoogleCodeExporter]

Ok, that's your choice... Any way of grouping results from several similar 
benchmarks into a single dataset in order to display them altogether on the 
same report?

Original comment by romain.r...@gmail.com on 17 Jun 2011 at 6:14

[GoogleCodeExporter]

Is it possible to refactor so that the results you want to view together 
actually come from the same benchmark class?  It would be a whole lot cleaner 
that way.  When mixing multiple classes, they might have all kinds of different 
parameters and method names and such.

"Benchmark suites" will still be considered post-1.0, but I think the case for 
them is not extraordinarily strong.

Original comment by kevinb@google.com on 19 Sep 2011 at 6:36

• Added labels: Milestone-Post-1.0, Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

Thanks for considering it. In my personal case, combining the benchmark methods 
into the same class is not applicable and modular enough to deal with the 
different software configuration we would like to compare.

Original comment by romain.r...@gmail.com on 19 Sep 2011 at 7:41

[GoogleCodeExporter]

Original comment by kevinb@google.com on 8 Feb 2012 at 9:49

• Added labels: Component-Runner

[GoogleCodeExporter]

Original comment by kevinb@google.com on 1 Nov 2012 at 8:32