search

GreenBitAI / green-bit-llm

A toolkit for fine-tuning, inferencing, and evaluating GreenBitAI's LLMs.
https://huggingface.co/blog/NicoNico/green-bit-llm
Apache License 2.0
62 stars 7 forks source link

Finetune issue with Qwen2 GPTQ Int4 #21

Closed greatzane closed 2 days ago

greatzane commented 1 week ago

Hello, I am encountering this message when finetuning with Qwen2 GPTQ Int4 series models: image And the finetuned model cannot be use for inference. The command line is: python -m green_bit_llm.sft.finetune --model ../Qwen2-0.5B-Instruct-GPTQ-Int4/ --dataset tatsu-lab/alpaca --batch-size 1 --tune-qweight-only Can you explain how to fix this? Thanks!

NicoNico6 commented 5 days ago

hi, the logging here is not a warning or issue from green-bit-llm, instead, it is a warning from the huggingface library. This is because there is no bias for down/gate/up/o_projlayers in transformers block for Qwen2. https://github.com/huggingface/transformers/blob/9fe3f585bb4ea29f209dc705d269fbe292e1128f/src/transformers/models/qwen2/modeling_qwen2.py#L174. This explains why thebias term in Qwen2-0.5B-Instruct-GPTQ-Int4 is not used for transformers AutoModelForCausalLM.

If you wanna try finetuning GPTQ model, please follow the below command: CUDA_VISIBLE_DEVICES=0 python -m green_bit_llm.sft.peft_lora --model astronomer/Llama-3-8B-Instruct-GPTQ-4-Bit --dataset tatsu-lab/alpaca --lr-fp 1e-6 CUDA_VISIBLE_DEVICES=0 python -m green_bit_llm.sft.finetune --model astronomer/Llama-3-8B-Instruct-GPTQ-4-Bit --dataset tatsu-lab/alpaca --tune-qweight-only --batch-size 1