Open hegang1-tal opened 1 year ago
hi, I found in deepspeed config file that the learning rate is le-8. I am wandering that is this too small for pretraining ?
Required prerequisites
Questions
hi, I found in deepspeed config file that the learning rate is le-8. I am wandering that is this too small for pretraining ?
Checklist