search

shibing624 / MedicalGPT

MedicalGPT: Training Your Own Medical GPT Model with ChatGPT Training Pipeline. 训练医疗大模型，实现了包括增量预训练(PT)、有监督微调(SFT)、RLHF、DPO、ORPO。
Apache License 2.0
2.94k stars 452 forks source link

llama2，sft全参训练，fp32，loss若干step后降为0 #320

Closed wangrx33 closed 5 months ago

wangrx33 commented 5 months ago

Describe the bug

Please provide a clear and concise description of what the bug is. If applicable, add screenshots to help explain your problem, especially for visualization related problems.

大佬，看到之前的issue里有关于loss降为0的问题，您提到解决方法是设置torch_dtype=float32，但是我这边设置了之后loss还是会降为0。 使用了deepspeed zero2，deepspeed 版本是0.12.6