search

THUDM / ChatGLM2-6B

ChatGLM2-6B: An Open Bilingual Chat LLM | 开源双语对话语言模型
Other
15.68k stars 1.85k forks source link

chatglm2微调回合数该如何设置[BUG/Help] <title> #587

Open renllll opened 10 months ago

renllll commented 10 months ago

Is there an existing issue for this?

Current Behavior

当我在垂直领域使用时需要微调上百回合才能降低到一个很好的loss，能比较准确进行垂直领域问题的回答

Expected Behavior

No response

Steps To Reproduce

指令微调上百回合，在垂直领域的回答才算比较好，我看很多论文只要微调几个回合

Environment

- OS:Ubuntu 20.04
- Python:3.8
- Transformers:4.26.1
- PyTorch:2.0.1
- CUDA Support (`python -c "import torch; print(torch.cuda.is_available())"`) :

Anything else?

No response

VViiepan commented 10 months ago

我也是，用的lora方式，训练轮数至少要100+训练轮数，损失才降至较低水平。但问题是，损失基本不下降后还不能获得较准确的回答