微调训练Loss异常 - Githubissues

baichuan-inc / Baichuan2

A series of large language models developed by Baichuan Intelligent Technology

https://huggingface.co/baichuan-inc

Apache License 2.0

4.08k stars 293 forks source link

微调训练Loss异常 #195

Open senzhen0725 opened 1 year ago

senzhen0725 commented 1 year ago

使用Baichuan2-7B-base作为预训练模型，belle-10k数据集进行微调，起始loss是2.x；使用Baichuan2-13B-Chat作为预训练模型，belle-10k数据集进行微调，起始loss是2.x；使用Baichuan2-13B-Base作为预训练模型，belle-10k数据集进行微调，起始loss是6.x；这个正常吗？

sxk000 commented 10 months ago

请问你是用哪个脚本预训练baichuan2的呀？