search

OpenBMB / MiniCPM-V

MiniCPM-V 2.6: A GPT-4V Level MLLM for Single Image, Multi Image and Video on Your Phone
Apache License 2.0
12.75k stars 893 forks source link

Finetuning issue #577

Closed ragesh2000 closed 1 month ago

ragesh2000 commented 2 months ago

I have a doubt regarding finetuning. My dataset contains images and text which are not familiar to model. So while finetuning should i keep the parameters tune_llm, tune_vision as true or false? By keeping it false the performance of my finetuned model was just as the base model. Any help is appreciated.

2U1 commented 2 months ago

It would be better for tuning the full-model if you need to fit your model to specific domain.

ragesh2000 commented 2 months ago

What i actually meant is, when do we need to keep those parameters to True? Only when we needed full-model tuning? If iam doing lora finetuning with some custom images do i need to keep it as true?

2U1 commented 2 months ago

When using LoRA, the you should make the tune_llm to False. Otherwise It would not work. tuen_vision is an option, but it would show a better performance in many cases when you set to True.