search

dash-project / dash

DASH, the C++ Template Library for Distributed Data Structures with Support for Hierarchical Locality for HPC and Data-Driven Science
http://www.dash-project.org/
Other
154 stars 44 forks source link

Unified performance reports #83

Open fuchsto opened 8 years ago

fuchsto commented 8 years ago

For automated performance reports in CI, benchmark applications should provide an option to print performance measurements in a unified format.

For this, utility types/concepts dash::bench::BenchmarkResult, dash::bench::BenchmarkResultPrinter should be introduced to decouple benchmark parameters and performance metrics from output.

Illustrating example:

$ mpirun -n 32 -genv DASH_MAX_THREADS_PER_UNIT=4 \
               ./bin/bench.12.time-dilatation.mpi --nd=11 --self-aware
$ # output:
$ [bench.12.time-dilatation.mpi] [n:32;tpu:4] [nd:11;self-aware] [iteration:0] [123.4]
$ [bench.12.time-dilatation.mpi] [n:32;tpu:4] [nd:11;self-aware] [iteration:1] [226.6]
$ [bench.12.time-dilatation.mpi] [n:32;tpu:4] [nd:11;self-aware] [iteration:2] [559.0]

... with 123.4, 226.6, 559.0 as measures in a "unified" performance metric (higher is better) specific to the benchmark. The overall goal is to emit warnings if these values decrease in a nightly build.

fmoessbauer commented 7 years ago

During the Christmas days I did some research and came to the decision, that we definitely should not implement this our own. There are (at least two) pretty advanced benchmarking libraries for C++:

Both offer a highly configurable but easy to use interface similar to GoogleTest. While GoogleBenchmark might be easier to use, Celero is better fitted to our quite complex input parameters. Furthermore it suites well to automated performance regression testing. A comparison of these two (along with others) libraries can be found here.

I know, switching to one of the libraries will require quite some refactoring, but IMO it's worth the time. Maybe this could also be implemented as part of a bachelor thesis?

@dash-project/developers Please tell me your opinion about my proposal.

fuchsto commented 7 years ago

@fmoessbauer Apart from that: you, my dear, are officially forbidden to direct your attention to mundane tasks like this. You have more interesting stuff to invest your efforts into.

fuchsto commented 7 years ago

... the same applies to @devreal for the same reason. Seriously: if you do, I won't merge your pull requests or roll them back if someone else does. Leave CI alone. It's the evil witch. It drains your energy.