Open fusesid opened 4 weeks ago
Oh you need to use model.save_pretrained_merged
for AWQ conversion
@danielhanchen , yeah i have tried that as well. But the issue is autoawq searches for config.json but we only have adapter_config.json as i have finetuned using LORA peft
@danielhanchen , yeah i have tried that as well. But the issue is autoawq searches for config.json but we only have adapter_config.json as i have finetuned using LORA peft
You are currently only saving the LORA adapters. I think you need to do this to save the full model (with all the needed files like config.json
, etc) for conversion to AWQ
.
model.save_pretrained_merged("model", tokenizer, save_method = "merged_16bit",)
Right @danielhanchen?
@mahiatlinux, Yeah i have used same command to save my model but it didn't have all the files. Using same command i.e.
model.save_pretrained_merged("model", tokenizer, save_method = "merged_16bit",)
My huggingface repo looks like:
I guess it just saved model with 16_bit weight which is why file size of safetensors are large.
This was the reason which i shifted to :
from unsloth import FastLanguageModel
model, tokenizer = FastLanguageModel.from_pretrained(
model_name = my-model
max_seq_length = 1024,
dtype = None,
load_in_4bit = True,
resize_model_vocab= 128257,
token= my-token
)
model = model.merge_and_unload()
tokenizer.push_to_hub(repo)
model.push_to_hub(repo)
After this my repo looked like:
But using autoawq for quantization on this model gives issue : RuntimeError: output with shape [8388608, 1] doesn't match the broadcast shape [8388608, 4096]
You have to push it using this to get all the files needed:
model.push_to_hub_merged("model", tokenizer, save_method = "merged_16bit", token = "token")
Try it and let me know @sids07.
@mahiatlinux, Yeah i have used same command to save my model but it didn't have all the files. Using same command i.e.
model.save_pretrained_merged("model", tokenizer, save_method = "merged_16bit",)
My huggingface repo looks like:
I guess it just saved model with 16_bit weight which is why file size of safetensors are large.
This was the reason which i shifted to :
from unsloth import FastLanguageModel model, tokenizer = FastLanguageModel.from_pretrained( model_name = my-model max_seq_length = 1024, dtype = None, load_in_4bit = True, resize_model_vocab= 128257, token= my-token ) model = model.merge_and_unload() tokenizer.push_to_hub(repo) model.push_to_hub(repo)
After this my repo looked like:
But using autoawq for quantization on this model gives issue : RuntimeError: output with shape [8388608, 1] doesn't match the broadcast shape [8388608, 4096]
Hmmm... Sorry about that. I don't know. Maybe @danielhanchen knows?
For replicating the issue, you can find unsloth save_model using merged_16bit on : https://huggingface.co/hiiamsid/llama3.1-finetuned-effie and unloaded peft model trained using unsloth on: https://huggingface.co/hiiamsid/effie-llama-merged
Firstly, I saved finetuned LORA model as merged_16bit on my huggingface repo. And i have adapter_config.json and adapter_model.safetensor inside my repo. Now when trying to load with AutoAWQForCausalLM, it searches config.json and throws an issue.
So, what i did is: load the model using FastLanguageModel and applied merge_and_unload method as below:
Then, i tried to quantize it:
But this is showing : RuntimeError: output with shape [8388608, 1] doesn't match the broadcast shape [8388608, 4096]
Am i doing the quantization correctly? Or what is causing this issue? Please help