Collective Knowledge

Collective Knowledge (CK, CM, CM4MLOps, CM4MLPerf and CMX) is an educational community project to learn how to run AI, ML and other emerging workloads in the most efficient and cost-effective way across diverse models, data sets, software and hardware. It includes the following sub-projects.

Collective Mind (CM)

Collective Mind (CM) - a very lightweight Python-based framework with a unified CLI, Python API and minimal dependencies intended to help researchers and engineers automate their repetitive, tedious and time-consuming tasks to build, run, benchmark and optimize AI, ML and other applications and systems across diverse and continuously changing models, data, software and hardware.

Collective Mind is continuously enhanced through public and private CM4* Git repositories, which serve as the unified interface for various collections of reusable automations and artifacts.

The CM architecture diagram is available for viewing here.

Collective Mind repositories

CM4MLOps

CM4MLOPS repository powered by CM - a collection of portable, extensible and technology-agnostic automation recipes with a common CLI and Python API (CM scripts) to unify and automate all the manual steps required to compose, run, benchmark and optimize complex ML/AI applications on diverse platforms with any software and hardware.

The two key automations are script" and cache*: see online catalog at CK playground, online MLCommons catalog.

CM scripts extend the concept of cmake with simple Python automations, native scripts and JSON/YAML meta descriptions. They require Python 3.7+ with minimal dependencies and are continuously extended by the community and MLCommons members to run natively on Ubuntu, MacOS, Windows, RHEL, Debian, Amazon Linux and any other operating system, in a cloud or inside automatically generated containers while keeping backward compatibility.

See the online documentation at MLCommons to run MLPerf inference benchmarks across diverse systems using CM.

CM4ABTF

CM4ABTF repository powered by CM - a collection of portable automations and CM scripts to run the upcoming automotive MLPerf benchmark across different models, data sets, software and hardware from different vendors.

CM4MLPerf-results

CM4MLPerf-results powered by CM - a simplified and unified representation of the past MLPerf results in the CM format for further visualization and analysis using CK graphs.

CM4Research

CM4Research repository powered by CM - a unified interface designed to streamline the preparation, execution, and reproduction of experiments in research projects.

Projects powered by Collective Mind

Collective Knowledge Playground

Collective Knowledge Playground - a unified and open-source platform designed to index all CM scripts similar to PYPI, assist users in preparing CM commands to:

• run MLPerf benchmarks
• aggregate, process, visualize, and compare benchmarking results for AI and ML systems
• organize open, reproducible optimization challenges and tournaments.

These initiatives aim to help academia and industry collaboratively enhance the efficiency and cost-effectiveness of AI systems.

Artifact Evaluation

Artifact Evaluation automation - a community-driven initiative leveraging the Collective Mind framework to automate artifact evaluation and support reproducibility efforts at ML and systems conferences.

Incubator

CMX - the next evolution of the Collective Mind framework, designed to enhance simplicity, flexibility, and extensibility of automations based on user feedback. Follow the project's progress here.

Archived projects

• CM-MLOps - now CM4MLOps
• CK automation framework v1 and v2 - now CM

License

Apache 2.0

Copyright

• Copyright (c) 2021-2024 MLCommons
• Copyright (c) 2014-2021 cTuning foundation

Author

• Grigori Fursin (FlexAI, cTuning)

Maintainers

• Collective Mind (CM): Grigori Fursin
• CM4MLOps repository: Arjun Suresh and Anandhu Sooraj
• CMX (the next generation of CM and CM4MLOps): Grigori Fursin

Citing Collective Mind and Collective Knowledge

If you found the CM automation framework helpful, kindly reference this article: [ ArXiv ], [ BibTex ].

To learn more about the motivation behind CK and CM technology, please explore the following presentations:

• "Enabling more efficient and cost-effective AI/ML systems with Collective Mind, virtualized MLOps, MLPerf, Collective Knowledge Playground and reproducible optimization tournaments": [ ArXiv ]
• ACM REP'23 keynote about the MLCommons CM automation framework: [ slides ]
• ACM TechTalk'21 about Collective Knowledge project: [ YouTube ] [ slides ]
• Journal of Royal Society'20: [ paper ]

CM Documentation

• Collective Mind white paper
• CM/CMX architecture
• CM/CMX installation GUI
• CM Getting Started Guide and FAQ
  • Common CM interface to run MLPerf inference benchmarks
  • Common CM interface to re-run experiments from ML and Systems papers including MICRO'23 and the Student Cluster Competition @ SuperComputing'23
  • CM4MLOps automation recipes for MLOps and DevOps
  • Other CM tutorials
• Full documentation
• CM taskforce
• CK, CM and CMX history

Acknowledgments

The open-source Collective Knowledge project (CK, CM, CM4MLOps/CM4MLPerf, CM4Research and CMX) was created by Grigori Fursin and sponsored by cTuning.org, OctoAI and HiPEAC. Grigori donated CK to MLCommons to benefit the community and to advance its development as a collaborative, community-driven effort.

We thank MLCommons, FlexAI and cTuning for supporting this project, as well as our dedicated volunteers and collaborators for their feedback and contributions!