Closed lin88lin8850 closed 1 year ago
when I set fp16=true during training gpt-2, error happens:
This error come from the gpt2 model of transformers. Because the reorder_and_upcast_attn is True in config.
reorder_and_upcast_attn
True
thanks a lot! bug solved, the issue will be closed
when I set fp16=true during training gpt-2, error happens: