How to add Radam to fairseq ?

LiyuanLucasLiu / Transformer-Clinic

Understanding the Difficulty of Training Transformers

https://arxiv.org/abs/2004.08249

Apache License 2.0

326 stars 20 forks source link

Closed KelleyYin closed 3 years ago

KelleyYin commented 3 years ago

According to the process described in your README, I encountered an error, fairseq-train: error: argument --optimizer: invalid choice : 'radam'.

I wonder how I can add Radm to fairseq.

Thanks.

LiyuanLucasLiu commented 3 years ago

Thanks for asking. I already included the RAdam implementation at: https://github.com/LiyuanLucasLiu/Transformer-Clinic/tree/master/radam_fairseq. You can import that by: --user-dir ../radam_fairseq (if run under the nmt-experiments folder, as in: https://github.com/LiyuanLucasLiu/Transformer-Clinic/blob/master/nmt-experiments/wmt14_en-de.md).

You may need to change ../radam_fairseq based on where you run the training command.