Extension projects to enable collaborative benchmarking, design space exploration and optimization of ML and AI Systems

[gfursin]

• [ ] Test that MLPerf inference benchmark works using the CM interface across different implementations, models, frameworks, data sets and platforms. See the CM interface with available variations and flags here.

  • [ ] Increase the coverage of the reference implementation of the MLPerf inference benchmark. See the current status here.
  • [x] Add more ML models (Hugging Face, Nvidia benchmark, papers) compatible with the MLPerf inference benchmark as CM scripts, test the benchmark and add models as variations here.
  • [x] Test TFLite C++ implementation of the MLPerf inference benchmark and extend coverage to other models and platforms. Attempt to reproduce past MLperf submissions with this new interface: (data center and edge)
    • [x] Extend MLPerf CM script to reproduce past open division submissions from MLPerf inference v2.1 (data center and edge) and prepare new submissions using the latest versions of all dependencies
  • [ ] Test C++ implenentation of the MLPerf inference benchmark and increase the coverage
  • [x] Test DeepSparse implementation
  • [ ] Compare different implementations and attempt to optimize them using different batch sizes, thread numbers, etc
  • [x] Participate in MLPerf inference v3.0 submission
  • [ ] Improve tutorials and documentation to make it easier for the community to understand CK2 (CM) concepts, use CM interface for unified ML Systems benchmarking and extend CM automations!

• Some performance/accuracy experiments per model

  • ResNet50
  • [ ] Integrate tflite cpp code to generic-cpp code, make all scenarios run (currently only singlestream works)
  • [ ] Try cuda device for tflite
  • [x] Add new models which can work with imagenet dataset to open division
  • [ ] Check performance of quantized models on different backends
  • [x] Compare performance of reference and Nvidia implementation on GPUs
  • [ ] Compare performance of reference and Intel implementation on CPUs
  • [ ] Does TVM improve performance of any model/system.scenario?
  • Bert
  • [ ] Try different bert models trained on squad
  • [ ] Try quantized models on different backends
  • [x] Compare performance of reference and Nvidia implementation on GPUs
  • [ ] Compare performance of reference and Intel implementation on CPUs
  • [ ] Does TVM improve performance of any model/system.scenario?
  • RetinaNet
  • [ ] Dry different retinanet models trained on openimages
  • [ ] Try quantized models on different backends
  • [ ] Try running the NMS part on CPU and rest on GPU
  • [x] Compare performance of reference and Nvidia implementation on GPUs
  • [ ] Compare performance of reference and Intel implementation on CPUs
  • [ ] Does TVM improve performance of any model/system.scenario?

• [ ] Test/improve the CM script for the light MLPerf inference benchmark to benchmark and optimize any ONNX model with loadgen but without data sets and accuracy!

• [ ] Test/improve CM interface to automatically prepare and run TinyMLPerf and prepare official tutorial. Use OctoML's submission as a starting point: results, code

  • [ ] Try another device such as Arduino Nano 33 BLE sense if supported in MLPerf
  • [ ] Participate in TinyMLPerf inference v3.0 submission

• [ ] Test and improve individual CM scripts across different software/stacks to be reused in any R&D project - will be useful for our reproducibility initiatives and artifact evaluation at conferences

[arjunsuresh]

Reference for adding CUDA for tflite-cpp