Open XinDongol opened 1 month ago
In paper, the optimal learning rate is 2e-3. In the pretrain.sh, the learning rate is set to 5e-4. Could you please advise the best learning rate to train MobileLLM models?
2e-3
pretrain.sh
5e-4
In paper, the optimal learning rate is
2e-3
. In thepretrain.sh
, the learning rate is set to5e-4
. Could you please advise the best learning rate to train MobileLLM models?