While fp16 causes error while training, I used bf16 for training and saw that it worked fine. I added the ability to enable this option through arguments and added an explanation to the tutorial. By default bf16 is off, but if you enable both the no_fp16 flag and bf16 flag, you train using mixed precision bf16.
While fp16 causes error while training, I used bf16 for training and saw that it worked fine. I added the ability to enable this option through arguments and added an explanation to the tutorial. By default bf16 is off, but if you enable both the
no_fp16
flag andbf16
flag, you train using mixed precision bf16.