search

LLaVA-VL / LLaVA-NeXT

Apache License 2.0
2.44k stars 174 forks source link

location of llava_1_6.json #222

Closed starhiking closed 1 week ago

starhiking commented 1 week ago

Where is the llava_1_6.json referred to in the scripts/train/finetune_clip.sh

mylesgoose commented 1 week ago

i think you need to find what datasets you want to train your model with. and run a script to get the data and generate the json file for that dataset. for example:/home/myles/LLaVA-NeXT/LLaVA-ReCap-558K.json /home/myles/LLaVA-NeXT/LLaVA-ReCap-CC3M.json import os from datasets import load_dataset from tqdm import tqdm import json

data = load_dataset("lmms-lab/LLaVA-ReCap-558K", split="train")

image_folder = "./data/image"

converted_data = []

for da in tqdm(data): json_data = {} json_data["id"] = da["id"] if da["image"] is not None: json_data["image"] = f"{da['id']}.jpg" da["image"].save(os.path.join(image_folder, json_data["image"])) json_data["conversations"] = da["conversations"] converted_data.append(json_data)

with open("LLaVA-ReCap-558K.json", "w") as f: json.dump(converted_data, f, indent=4, ensure_ascii=False)

for example:that script downloasdsthat particular dataset from hugginface. it looks like yoru trying to train 1.6 i think perhaps all that is contained int he recap one below. probbaly have to go and find out what datasets 1.6 was trained on. i noticed thsi guy has collated all the datasets into one json for you for the 1.6 model. https://github.com/xiaoachen98/Open-LLaVA-NeXT/blob/master/scripts/v1_6/train/8b/finetune.sh in this docuemnt he explaisn hwo he prepared the data. https://github.com/xiaoachen98/Open-LLaVA-NeXT/blob/master/docs/Data.md i think perhaps this repo assumes you have followed the 1.5 and 1.6 github pages. datasets: