search

bigcode-project / Megatron-LM

Ongoing research training transformer models at scale
Other
376 stars 49 forks source link

Saved model checkpoint in different precision ? #88

Open Ankush2k opened 6 months ago

Ankush2k commented 6 months ago

Hi Team, Great work!! I would like to know the precision in which the model checkpoint is saved. For example, if trained with bf16 precision, what will the checkpoint precision will be?

Thanks

Ankush2k commented 6 months ago

Hi Team, above doubt have been resolved - Model weights dtype will be bf16. Another doubt, Is there any specific setting/argument that we need so that the model checkpoint will be stored in fp32?

Thanks