mlcommons / training_policies

Issues related to MLPerf™ training policies, including rules and suggested changes
https://mlcommons.org/en/groups/training
Apache License 2.0
93 stars 66 forks source link

[v1.1 BERT] Update reference and rules to allow clipnorm-after-reduce #432

Open johntran-nv opened 3 years ago

johntran-nv commented 3 years ago

In v1.0, we discovered after code freeze that the reference does clipnorm-before-reduce. Mathematically, that is not as good as clipnom-after-reduce, but since it was after code freeze, we decided to allow either, but not let submitters take advantage of perf optimizations from clipnorm-after-reduce.

For v1.1, we should change the reference before code freeze, and allow optimizations.