MiniCPM-2B-sft-int4 lora微调后，似乎无法merge权重 - Githubissues

OpenBMB / MiniCPM

MiniCPM-2B: An end-side LLM outperforming Llama2-13B.

Apache License 2.0

4.38k stars 313 forks source link

MiniCPM-2B-sft-int4 lora微调后，似乎无法merge权重 #131

Closed lonelydancer closed 1 week ago

lonelydancer commented 2 months ago

Is there an existing issue ? / 是否已有相关的 issue ?

[X] I have searched, and there is no existing issue. / 我已经搜索过了，没有相关的 issue。

Describe the bug / 描述这个 bug

MiniCPM-2B-sft-int4 lora微调后，无法merge权重。

raise ValueError("Cannot merge LORA layers when the model is gptq quantized")

To Reproduce / 如何复现

https://github.com/yangjianxin1/Firefly/blob/master/script/merge_lora.py

Expected behavior / 期望的结果

No response

Screenshots / 截图

No response

Environment / 环境

- OS: [e.g. Ubuntu 20.04]
- Pytorch: [e.g. torch 2.0.0]
- CUDA: [e.g. CUDA 11.8]
- Device: [e.g. A10, RTX3090]

Additional context / 其他信息

No response

请问需要自己用bf16版本做微调merge后做量化？如果自己用gptq做量化，数据集应该用什么呢？

LDLINGLINGLING commented 3 weeks ago

可以使用bf16进行训练后再做量化，具体的量化操作可以参考这个github地址：https://github.com/LDLINGLINGLING/MiniCPM/tree/ollama_manual_installation 的readme文件中gptq量化部分，量化的数据集可以是通用数据集加上你自己的训练数据集。