search

haotian-liu / LLaVA

[NeurIPS'23 Oral] Visual Instruction Tuning (LLaVA) built towards GPT-4V level capabilities and beyond.
https://llava.hliu.cc
Apache License 2.0
19.37k stars 2.13k forks source link

[Question] Trying to finetune LLAVA on a custom dataset without the use of a GPU. #794

Open aa221 opened 10 months ago

aa221 commented 10 months ago

Question

Is there a way I can download the checkpoints of the pre-trained LLAVA, and then fine-tune the data on my custom data set? I don't have a GPU so any tips would be appreciated.

tosiyuki commented 10 months ago

The dataset must be in the same format as LLaVA and "--data_path" in the training script must be replaced with your data path. If you want to fine-tune the model after the second stage training, you will need to modify the training code slightly. https://github.com/haotian-liu/LLaVA/tree/main/scripts/v1_5

But to learn without GPU, the learning time is enormous...

aneet-javis commented 10 months ago

@tosiyuki Could you please tell what is the format of LLaVA dataset? I'm trying to create a custom dataset for fine-tuning.

tosiyuki commented 10 months ago

@tosiyuki Could you please tell what is the format of LLaVA dataset? I'm trying to create a custom dataset for fine-tuning.

You can check data format here. https://github.com/haotian-liu/LLaVA/blob/main/docs/Finetune_Custom_Data.md

cherry956 commented 7 months ago

@tosiyuki Now I have my own dataset in the same format as LLaVA , So how I finetune llava-v1.5-7b with lora? I don't know the next I should do?

linfei-mise commented 5 months ago

@cherry956 I have the same problem now, have you solved it please?

SamuelSchmidgall commented 4 months ago

Try reading this @linfei-mise @cherry956 https://github.com/ubiai-incorporated/colab_repos/blob/master/LLAVA_finetuning.ipynb

linfei-mise commented 3 months ago

@SamuelSchmidgall Thank you for your reply, I have resolved the issue.