Collective Mind (CM) is a small, modular, cross-platform and decentralized workflow automation framework with a human-friendly interface and reusable automation recipes to make it easier to build, run, benchmark and optimize AI, ML and other applications and systems across diverse and continuously changing models, data, software and hardware
[ ] Test that MLPerf inference benchmark works using the CM interface across different implementations, models, frameworks, data sets and platforms. See the CM interface with available variations and flags here.
[ ] Increase the coverage of the reference implementation of the MLPerf inference benchmark. See the current status here.
[x] Add more ML models (Hugging Face, Nvidia benchmark, papers) compatible with the MLPerf inference benchmark as CM scripts, test the benchmark and add models as variations here.
[x] Extend MLPerf CM script to reproduce past open division submissions from MLPerf inference v2.1 (data center and edge) and prepare new submissions using the latest versions of all dependencies
[ ] Compare different implementations and attempt to optimize them using different batch sizes, thread numbers, etc
[x] Participate in MLPerf inference v3.0 submission
[ ] Improve tutorials and documentation to make it easier for the community to understand CK2 (CM) concepts, use CM interface for unified ML Systems benchmarking and extend CM automations!
Some performance/accuracy experiments per model
ResNet50
[ ] Integrate tflite cpp code to generic-cpp code, make all scenarios run (currently only singlestream works)
[ ] Try cuda device for tflite
[x] Add new models which can work with imagenet dataset to open division
[ ] Check performance of quantized models on different backends
[x] Compare performance of reference and Nvidia implementation on GPUs
[ ] Compare performance of reference and Intel implementation on CPUs
[ ] Does TVM improve performance of any model/system.scenario?
Bert
[ ] Try different bert models trained on squad
[ ] Try quantized models on different backends
[x] Compare performance of reference and Nvidia implementation on GPUs
[ ] Compare performance of reference and Intel implementation on CPUs
[ ] Does TVM improve performance of any model/system.scenario?
RetinaNet
[ ] Dry different retinanet models trained on openimages
[ ] Try quantized models on different backends
[ ] Try running the NMS part on CPU and rest on GPU
[x] Compare performance of reference and Nvidia implementation on GPUs
[ ] Compare performance of reference and Intel implementation on CPUs
[ ] Does TVM improve performance of any model/system.scenario?
[ ] Test/improve the CM script for the light MLPerf inference benchmark to benchmark and optimize any ONNX model with loadgen but without data sets and accuracy!
[ ] Test/improve CM interface to automatically prepare and run TinyMLPerf and prepare official tutorial. Use OctoML's submission as a starting point: results, code
[ ] Try another device such as Arduino Nano 33 BLE sense if supported in MLPerf
[ ] Participate in TinyMLPerf inference v3.0 submission
[ ] Test and improve individual CM scripts across different software/stacks to be reused in any R&D project - will be useful for our reproducibility initiatives and artifact evaluation at conferences
[ ] Test that MLPerf inference benchmark works using the CM interface across different implementations, models, frameworks, data sets and platforms. See the CM interface with available variations and flags here.
Some performance/accuracy experiments per model
[ ] Test/improve the CM script for the light MLPerf inference benchmark to benchmark and optimize any ONNX model with loadgen but without data sets and accuracy!
[ ] Test/improve CM interface to automatically prepare and run TinyMLPerf and prepare official tutorial. Use OctoML's submission as a starting point: results, code
[ ] Test and improve individual CM scripts across different software/stacks to be reused in any R&D project - will be useful for our reproducibility initiatives and artifact evaluation at conferences