search

mlcommons / cm4mlops

A collection of portable, reusable and cross-platform automation recipes (CM scripts) with a human-friendly interface and minimal dependencies to make it easier to build, run, benchmark and optimize AI, ML and other applications and systems across diverse and continuously changing models, data sets, software and hardware (cloud/edge)
http://docs.mlcommons.org/cm4mlops/
Apache License 2.0
7 stars 12 forks source link

Feature request: Checkpoint labels for CM builds #63

Open WarrenSchultz opened 1 month ago

WarrenSchultz commented 1 month ago

Given the highly dynamic nature of ML testing development, it would be an extremely useful feature to be able to pull labels for guaranteed repeatability across multiple systems, or to be able to go back and replicate older test conditions with different hardware at a later date. (If this is already in there, then the request is for more detailed documentation on how to do so :)

arjunsuresh commented 4 weeks ago

We'll surely do more labels and releases starting next month once we have covered all of the Nvidia and Intel MLPerf implementations in CM. But even now, we do support git checkout with CM. This is shown in our 4.0 submission README

Further CM for MLPerf inference is also generating a version information file that shows the versions of the dependencies in use. After the submissions, we have also added git commit hash for the repositories where non-release branches are used. https://github.com/mlcommons/inference_results_v4.0/blob/main/closed/CTuning/measurements/GATE_Overflow_Intel_Sapphire_Rapids-nvidia_original-gpu-tensorrt-vdefault-default_config/resnet50/offline/cm-version-info.json

Please let us know if you have any feedback or further requirements.

WarrenSchultz commented 4 weeks ago

Thanks, let me take a look at this.