Missing Resnet HP Norm vs Trunc norm initialization

mlcommons / training_policies

Issues related to MLPerf™ training policies, including rules and suggested changes

https://mlcommons.org/en/groups/training

Apache License 2.0

92 stars 66 forks source link

Missing Resnet HP Norm vs Trunc norm initialization #367

Open bitfort opened 4 years ago

bitfort commented 4 years ago

This is an HP Choice we didn't document; to discuss today.

bitfort commented 4 years ago

SWG:

Resnet has allowing choosing of norm initialization. This is a hyperparamter not listed in the HP table or logged. This has been tuned in previous rounds and both norms appear in the reference with an option to switch between them. As normal hyperparameters, this can be borrowed during review and tuned during review.