Closed danieldk closed 4 years ago
We use Adam in all the models. I propose to set the standard warmup to 2000 for both train and pretrain, since it results in better models in most cases than having no warmup.
train
pretrain
We use Adam in all the models. I propose to set the standard warmup to 2000 for both
train
andpretrain
, since it results in better models in most cases than having no warmup.