grad norm 为 NAN， loss为0

[lambda-lee]

Reminder

• [X] I have read the README and searched the existing issues.

System Info

• llamafactory version: 0.9.1.dev0
• Platform: Linux-5.4.241
• Python version: 3.9.16
• PyTorch version: 2.1.0+cpu (NPU)
• Transformers version: 4.45.0.dev0
• Datasets version: 2.21.0
• Accelerate version: 0.34.2
• PEFT version: 0.12.0
• TRL version: 0.9.6
• NPU type: Ascend910B2C
• CANN version: 8.0.RC2
• DeepSpeed version: 0.14.0

Reproduction

训练一段时间后，先出现 {'loss': 1.4596, 'grad_norm': nan, 'learning_rate': 4.552255167404752e-06, 'epoch': 0.66} 接着就都是 {'loss': 0.0, 'grad_norm': nan, 'learning_rate': 4.549912997495027e-06, 'epoch': 0.66} {'loss': 0.0, 'grad_norm': nan, 'learning_rate': 4.549912997495027e-06, 'epoch': 0.66} {'loss': 0.0, 'grad_norm': nan, 'learning_rate': 4.549912997495027e-06, 'epoch': 0.66}

Expected behavior

怎么实现训练过程中能够跳过出现错误的数据，继续执行训练，谢谢

Others

No response

[hiyouga]

换个模型试试？

[Eternal-Yan]

我也是这种情况，请问你解决了嘛？