search

BigData-KSU / RS-LLaVA

33 stars 1 forks source link

Confusion about the weights of mm_projector #2

Open jian-rookie opened 5 months ago

jian-rookie commented 5 months ago

In the inference script, the default weights of RS-LLaVA are as follows: model_path = 'BigData-KSU/RS-llava-v1.5-7b-LoRA' model_base = 'Intel/neural-chat-7b-v3-3'

However, these two models do not include the weights of mm_projector in LLaVa. Thus, the following information appears when runing the the inference script.

Some weights of LlavaLlamaForCausalLM were not initialized from the model checkpoint at /home/yangjian/retrieval/RS-LLaVA-main/ckpts/neural-chat-7b-v3-3 and are newly initialized: ['model.mm_projector.0.weight', 'model.mm_projector.0.bias', 'model.mm_projector.2.weight', 'model.mm_projector.2.bias']

Even if the above information is ignored, RS-LLaVA can still work. But I have my doubts about this detail. Can you help explain this question? Thanks for your excellent work！

zhouzihui2001 commented 1 week ago

Hello! Have you modified the code to load the model? If I do not modify the code, an error occurs in loading the weight of the mm_projector, stating "mm_projector.bin does not exist". If I modify the code not to load the mm_projector's weight, the result is a few phrases and sentences beginning with "A", which is clearly wrong.

In the inference script, the default weights of RS-LLaVA are as follows: model_path = 'BigData-KSU/RS-llava-v1.5-7b-LoRA' model_base = 'Intel/neural-chat-7b-v3-3'

However, these two models do not include the weights of mm_projector in LLaVa. Thus, the following information appears when runing the the inference script.

Some weights of LlavaLlamaForCausalLM were not initialized from the model checkpoint at /home/yangjian/retrieval/RS-LLaVA-main/ckpts/neural-chat-7b-v3-3 and are newly initialized: ['model.mm_projector.0.weight', 'model.mm_projector.0.bias', 'model.mm_projector.2.weight', 'model.mm_projector.2.bias']

Even if the above information is ignored, RS-LLaVA can still work. But I have my doubts about this detail. Can you help explain this question? Thanks for your excellent work！