loubnabnl / santacoder-finetuning

Fine-tune SantaCoder for Code/Text Generation.
Apache License 2.0
184 stars 23 forks source link

Added the ability to train with bf16 through arguments #9

Closed esslushy closed 1 year ago

esslushy commented 1 year ago

While fp16 causes error while training, I used bf16 for training and saw that it worked fine. I added the ability to enable this option through arguments and added an explanation to the tutorial. By default bf16 is off, but if you enable both the no_fp16 flag and bf16 flag, you train using mixed precision bf16.

esslushy commented 1 year ago

that was a good suggestion! I forgot not every platform supports that form of mixed precision! Let me know if anything else needs to be changed.

loubnabnl commented 1 year ago

LGTM thanks!