search

InternLM / InternLM-XComposer

InternLM-XComposer2 is a groundbreaking vision-language large model (VLLM) excelling in free-form text-image composition and comprehension.
1.91k stars 120 forks source link

QLora fine tuning? #337

Open pbarker opened 6 days ago

pbarker commented 6 days ago

Hello, thank you for the amazing work, is it possible to use Qlora to fine tune the 4bit quant models?

nzomi commented 5 days ago

Hello, I tried using BitsAndBytesConfig to obtain and save a 4-bit model. However, I encountered an issue where it is crucial to generate a chat using the 4-bit model. Have you experienced a similar issue? By the way, I followed the instructions with AutoGPTQ to obtain another 4-bit model, but I received a message stating that

'internlmxcomposer2 isn't supported yet.'

Has anyone else encountered this issue? How can I resolve it?

pbarker commented 4 days ago

@nzomi https://github.com/AutoGPTQ/AutoGPTQ/pull/619 and https://github.com/AutoGPTQ/AutoGPTQ/pull/189

nzomi commented 4 days ago

@nzomi AutoGPTQ/AutoGPTQ#619 and AutoGPTQ/AutoGPTQ#189

@pbarker Thank you for mentioning that. Indeed, I also created an issue in their repository and the problem was fixed. However, I tried to quantize the 4KHD model, but its structure is a bit different from the 7B version, which has become another challenge...

pbarker commented 1 day ago

Hey @nzomi we are going to try and quant the 4khd model next week if you want to share notes, also if there is a maintainer that can give any tips we would appreciate it!