-
- coordinate with this task: https://github.com/mlcommons/modeltune/issues/21
- (ideally) Migrate the logic from this PR (https://github.com/mlcommons/modeltune/pull/2) to modelgauge where we can leve…
-
### Describe the bug
The MLCubes getting generated (model or metrics) are always the same version, as evidenced by the following files (just search for the string `0.0.1`):
- https://github.com/ml…
-
- data blocked on generating SUT responses for the 40k MLC prompts. See the other tasks first to generate SUT data
- once you have prompt + SUT response, then use the ensemble annotator (hopefully cre…
-
https://github.com/mlcommons/inference/issues/1740
Encountered same issue as above post. Had to manually change --user to -U , else cm would not be available globally, guess add the user path to PA…
-
Link: https://mlcommons.org/ai-safety/ai-safety-prompt-eoi/
-
Hello mlcommons team,
I want to run the "Automated command to run the benchmark via MLCommons CM" (from the example: https://github.com/mlcommons/inference/tree/master/language/llama2-70b), but I d…
-
Add reference code for `mixtral-8x7b`(https://github.com/mlcommons/inference/tree/master/language/mixtral-8x7b) in `axs`.
To do the following steps:
- add recipe for downloading dataset
- add recip…
-
Hello mlcommons team,
I want to run the "Automated command to run the benchmark via MLCommons CM" (from the example: https://github.com/mlcommons/inference/tree/master/language/llama2-70b) with a d…
-
## Overview
- "Croissant is a high-level format for machine learning datasets that brings together four rich layers."
- This issue tracks activities related to our collaboration with the Kaggle Tea…
-
From the step at https://github.com/mlcommons/training/blob/3a6a379305e6ef0a1c34461c823e9c99d52c1021/recommendation_v2/torchrec_dlrm/scripts/process_Criteo_1TB_Click_Logs_dataset.sh#L42, there was a d…