Adding profiling and performance analysis during benchmarking

[gfursin]

We need to continue improving universal benchmarking and optimization capabilities in CM for different OS and hardware targets:

• For compiled code (C/C++ ...) we improve the following CM scripts

  • CM scripts:
  • compile-program
  • benchmark-program
  • benchmark-program-mlperf
  • TBD
  • [ ] support for gprof/oprofile/hardware counters
  • [ ] universal support to pin threads (numactl)
  • [ ] expose internal profiling info from ML frameworks and run-times if/when available (onnx, TFLite ...)
  • Sample apps:
  • Corner detection (C)
  • MLPerf ONNX C++ harness
  • MLPerf TFLite C++ harness

• For Python:

  • [ ] create CM script with a python package to collect various profiling info (memory utilization, etc) particularly to analyze ML/AI models (was asked by ABTF).
  • [ ] Collect function-level profiling

• Add support for universal performance analysis to CM experiment:

  • [ ] Aggregate profiling from multiple runs and perform stat analysis (variation, min/max, phases, etc)
  • [ ] Visualize experiments