Fine-tune a LLaVa model for Visual question answering task on some custom set of images.

[anjanakg]

Question

I want to fine-tune a LLaVa model for Visual question answering task on some custom set of images. I wanted to know the Dataset format required for training and then fine-tuning. I found this github folder: https://github.com/haotian-liu/LLaVA/tree/main/scripts/v1_5 . And I found this is an example dataset: https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K/blob/main/detail_23k.json . But since I am new in these tasks still I didn't get a good idea about how to do it correctly. Can someone direct me?

[gyupro]

I am not sure what you don't know. You can finetune with the finetune script with the llava format dataset.

[anjanakg]

I am not sure what you don't know. You can finetune with the finetune script with the llava format dataset.

Thank you so much! I am sorry, and I am confused. I know how to train CNN models. We need to access the image folders along with the labels. But here in those Github folders, I cannot see any image data folders. Can anyone tell me where we need to keep the image folder along with that json file for fine-tuning the LLaVA?

[gyupro]

@anjanakg oh I see. U can setup your dataset along with this script https://github.com/SkunkworksAI/BakLLaVA/blob/main/setup_finetune.sh . The script is from another repo. You need to download the datasets from different websites and it's annoying. Using this script makes it easier

[anjanakg]

@anjanakg oh I see. U can setup your dataset along with this script https://github.com/SkunkworksAI/BakLLaVA/blob/main/setup_finetune.sh . The script is from another repo. You need to download the datasets from different websites and it's annoying. Using this script makes it easier

Oh...! Understood! Thank you so much! I will try in that way.

So, I need to create a json data file(with required format) for my images and keep the image folder somewhere on my local computer or cloud. And, then give the link to the folder and the json file as in that fine-tuning script. Am I correct? If not could you please correct me?

[Kikzter]

@anjanakg Hi, hope you are doing well. I have certain doubts on training this model. if you don't mind can we connect regarding this. gmail md.nayeem18899@gmail.com

[Kikzter]

Hi @anjanakg , @gyupro ,

Could you please help me out how to fine tuning the model using custom dataset. it would be better if we connect on it. email: md.nayeem18899@gmail.com

Thank you