Add default bf16 precision setting when bf16 config option is set but precision is unset.

EleutherAI / gpt-neox

An implementation of model parallel autoregressive transformers on GPUs, based on the Megatron and DeepSpeed libraries

https://www.eleuther.ai/

Apache License 2.0

6.95k stars 1.02k forks source link

Add default bf16 precision setting when bf16 config option is set but precision is unset. #1311

Closed AI-WAIFU closed 1 week ago

AI-WAIFU commented 4 weeks ago

https://github.com/EleutherAI/gpt-neox/issues/1305 Shows that you can set the bf16 config without setting the precision to bfloat16 and this causes an obscure crash. This should automatically set the precision when it's missing or throw crash with an assert if precision conflicts.