search

dart-lang / benchmark_harness

The official benchmark harness for Dart
https://pub.dev/packages/benchmark_harness
BSD 3-Clause "New" or "Revised" License
94 stars 26 forks source link

Add support for running perftools to use hardware performance counters when benchmarking. #98

Closed whesse closed 6 months ago

whesse commented 9 months ago

To use hardware performance counters to measure benchmark performance, this PR adds hooks (methods that can be overridden in subclasses) to the base benchmark harness that run immediately before and after the main benchmarking loop.

It also exposes the total count of benchmark iterations to the call that creates measurements, so this count can be made visible in the output.

The rest of the implementation, using environment variables to control optionally signalling perftools using named pipes, could be located in this package or put in a different package.

whesse commented 8 months ago

@jensjoha You might be interested in this change

jensjoha commented 8 months ago

Do you have numbers for how stable it is when it's started and stopped in the middle of a run? My immediate thought is that it isn't --- which would make it not useful --- so I'm hoping to be proven wrong :)

whesse commented 8 months ago

The benchmark harness is now starting up the "perf stat" subprocess and attaching it to the benchmark process. Example output is:

dart run BinaryTrees/dart/BinaryTrees.dart 
BinaryTrees(MajorPageFaults)(RunTime): 587.8266666666667 us.
BinaryTrees(CpuCycles)(RunTime): 250666196.78666666 us.
BinaryTrees.totalIterations(RunTime): 75.0 us.
BinaryTrees(RunTime): 60710.38636363636 us.

Remaining TODO: remove the extra (RunTime) from the output, and create and destroy the fifo named pipes in the harness.

whesse commented 7 months ago

This PR is now ready for review. It has been restructured to move the "perf stat" subprocess into the PerfBenchmarkBase class, and to create the named pipes needed for that class in the system temp directory.

An integration test is added, that will only pass when run on a machine with the "mkfifo" and "perf" commands.