Some results trained with Adam and AdamW on Librispeech

[luomingshuang]

1. I use the script (mmi_att_transformer_train.py) as the basic experiment setting and use train-clean-100 as the training data.
2. I use Adam or AdamW optimizer to replace Noam.
3. I compare the performance from the three optimizers.
4. The results are based on different learning rate and decay weight.
5. All the results are based on single GPU.

[danpovey]

OK, thanks. I guess this confirms that we need the noam-type learning rate scheduler. At some point I'd like to try something like Adam but with weight decay where it decays to the initial (random) value, not to 0. [and also with a Noam-type scheduler]

[luomingshuang]

Yes, maybe I can try to set the lr (learning rate) according to the epoch. When the epoch <=5 (or other value), setting the lr=0.0001. And when the epoch >5, setting the lr=0.00001.

OK, thanks. I guess this confirms that we need the noam-type learning rate scheduler. At some point I'd like to try something like Adam but with weight decay where it decays to the initial (random) value, not to 0. [and also with a Noam-type scheduler]

[danpovey]

Study the Noam optimizer, how it sets the learning rate. It is just a modification of Adam, to set a certain schedule.

On Tue, Jul 6, 2021 at 4:46 PM Mingshuang Luo @.***> wrote:

Yes, maybe I can try to set the lr (learning rate) according to the epoch. When the epoch <=5 (or other value), setting the lr=0.0001. And when the epoch >5, setting the lr=0.00001.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/k2-fsa/snowfall/issues/228#issuecomment-874578540, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZFLO536WF6ZRGIUXYEW5LTWK7F3ANCNFSM474ASOPQ .