Closed DrewGalbraith closed 3 months ago
Before my single commit, training with the default issued the following error:
ValueError: Precision 'fp16' is invalid. Allowed precision values: ('transformer-engine', 'transformer-engine-float16', '16-true', '16-mixed', 'bf16-true', 'bf16-mixed', '32-true', '64-true', 64, 32, 16, '64', '32', '16', 'bf16')
My commit changed the default to bf16, which I can confirm works. For more info on why this is the best option for default, look at the PyTorch Lightning docs for Mixed Precision Training (or ask Jay or me).
Before my single commit, training with the default issued the following error:
My commit changed the default to bf16, which I can confirm works. For more info on why this is the best option for default, look at the PyTorch Lightning docs for Mixed Precision Training (or ask Jay or me).