search

mlcommons / algorithmic-efficiency

MLCommons Algorithmic Efficiency is a benchmark and competition measuring neural network training speedups due to algorithmic improvements in both training algorithms and models.
https://mlcommons.org/en/groups/research-algorithms/
Apache License 2.0
321 stars 62 forks source link

Mismatch between target values in `workload.py` files and on `DOCUMENTATION.md` page #684

Closed tfaod closed 6 months ago

tfaod commented 6 months ago

A few weeks ago, I noticed that the target values on the “getting started” are not consistent with the target values specified in the workload.py files. I thought this was a typo and previously I updated the workload.py files with the validation values.

This week, I noticed that our training runs are hitting the target values specified in the workload.py but not the Getting Started documentation. For example, our fastmri workload consistently hits the test target, but never the val target from DOCUMENTATION.md.

Could we confirm that the workload.py target files are correct, and fix this discrepancy? Thanks!!

priyakasimbeg commented 6 months ago

Hi, we are working on updating the documentation with the correct targets and maximum runtimes. In the meantime please refer to the workload.py files as the source of truth.

@runame