search

chuangchuangtan / LLaVA-NeXT-Image-Llama3-Lora

LLaVA-NeXT-Image-Llama3-Lora, Modified from https://github.com/arielnlee/LLaVA-1.6-ft
Apache License 2.0
15 stars 0 forks source link

LoRA merging doesn't work #1

Open NicoZenith opened 6 days ago

NicoZenith commented 6 days ago

Hi! Thank you again for this repo. The fine-tuning with llama3 works. However, when I try to merge with the obtained LoRA weights, using the merge_lora_weights.py script, and I compare the weights between the base model and the new merged model, I see changes in the vision encoder (which is surprising), and not in the bridger and language model. Do you have the same observation? Maybe I am doing something wrong

chuangchuangtan commented 4 days ago

Llava determines whether it is a base model or a lora model based on the path name, please pay attention to the model save path.