Closed fabbing closed 1 year ago
Thanks for the contribution, @fabbing and @MisterDA. As discussed in person, the benchmarks are currently run using different wrappers, like orun
, perfstat
or pausetimes
.
But, the use case of being able to measure specific parts of a program as part of a benchmark makes sense to me. We do have such benchmarks being run on different repositories, using current-bench
, which allows repositories to define their own custom benchmarks. I wonder if maybe that is a better place to have this benchmark? Or if we should add support for doing these kinds of benchmarking with micro-benchmarks in Sandmark.
@shakthimaan or @kayceesrk might have thoughts on this.
But, the use case of being able to measure specific parts of a program as part of a benchmark makes sense to me.
Sandmark is not built for measuring and reporting on specific parts of the program. All the reported metrics are for the entire program. Breaking this invariant complicates how we measure, report and analyse the benchmarks in Sandmark. I am not keen on breaking this.
current-bench
is probably the better place. The other option is to make the prefetching specific bits run much longer so that its effects dominate the program behaviour measured as a whole. This may be as simple as running the core parts of the algorithm repeatedly so that prefetching effects are magnified.
current-bench
is probably the better place. The other option is to make the prefetching specific bits run much longer so that its effects dominate the program behaviour measured as a whole. This may be as simple as running the core parts of the algorithm repeatedly so that prefetching effects are magnified.
The core part is actually already run several times (and can be easily tweaked with the launch parameter). The GC/tracing time of the benchmark largely dominates the setup time, so the benchmark can be kept as it is in Sandmark.
Sounds great.
Thanks for the contribution, @fabbing and @MisterDA ! I've merged it with a minor change - changed the tag from micro_bench
to macro_bench
to make sure that the benchmark runs in our nightly runs.
This PR, a joint effort with @MisterDA, adds a very slightly modified version of @stedolan
markbench
micro-benchmark. This micro-benchmark was first used in https://github.com/ocaml/ocaml/pull/10195 and then in https://github.com/ocaml/ocaml/pull/11827 to validate prefetching speedup while the GC is tracing blocks.It could be useful as a sort of regression test running in Sandmark.
It would be preferable to use the reported seconds/GC time as calculated by the benchmark, which is reported on
stdout
, since it avoids accounting for setup time. How could this could be achieved in Sandmark?