search

mlcommons / inference

Reference implementations of MLPerf™ inference benchmarks
https://mlcommons.org/en/groups/inference
Apache License 2.0
1.24k stars 536 forks source link

quesiton about Llama2-70b #1818

Open amine-ammor opened 3 months ago

amine-ammor commented 3 months ago

Hello, when downloading the processed dataset for llama2-70b with rclone: as specified on the file "language/llama2-70b/README.md" on the "get dataset section" I noticed the file "mlperf_log_accuracy.json" within the folder. Is it the accuracy file, linked to the execution of the benchmark on DGX-H100 with FP32 accuracy ? If not, is this file publicly accessible? Thank you, Regards,

Oseltamivir commented 2 weeks ago

Hi @amine-ammor , the accuracy refers to the accuracy run.

From https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc:

For each benchmark, MLPerf will provide pointers to: An accuracy data set, to be used to determine whether a submission meets the quality target, and used as a validation set A speed/performance data set that is a subset of the accuracy data set to be used to measure performance

If you would like to change the precision of your model, you can change it in the custom.py config file if you are running the NVIDIA implementation at https://github.com/mlcommons/inference_results_v4.0/tree/main/closed/NVIDIA

If this resolves your question, you can close the issue.