How to organize data, which can be fine-tuned with both image-text data, as well as purely textual data.

OpenBMB / MiniCPM-V

MiniCPM-Llama3-V 2.5: A GPT-4V Level Multimodal LLM on Your Phone

Apache License 2.0

7.86k stars 547 forks source link

How to organize data, which can be fine-tuned with both image-text data, as well as purely textual data. #250

Open hill2hill opened 3 weeks ago

hill2hill commented 3 weeks ago

When fine-tuning with LoRA, is it necessary to use data that includes images? If pure text data is used, would it affect the model's performance (it should not, as some open-source datasets for MLM models include SFT with pure text question-answer pairs)?

How should the JSON file be structured?

univa-JASON commented 3 weeks ago

22 how can i finetune this model with Text-only data and Image-Text data in same dataset?