training code

[ehartford]

Hello, I am trying to find the training code, but it seems like there is just inference code.

Can you please point to the training code?

[NicoZenith]

That would be great to get the training scripts, as it was done in the original LLaVA repo :)

[carlos-havier]

I'd also love to use them for fine-tuning with several images, for few-shot image classification.

[NicoZenith]

what do you guys think of this ? https://github.com/NielsRogge/Transformers-Tutorials/blob/master/LLaVa/Fine_tune_LLaVa_on_a_custom_dataset_(with_PyTorch_Lightning).ipynb

By replacing llava by lava-next (processor and model)

[chuangchuangtan]

I implement a LLava-llama3 Lora finetuning https://github.com/chuangchuangtan/LLaVA-NeXT-Image-Llama3-Lora

[NicoZenith]

I implement a LLava-llama3 Lora finetuning https://github.com/chuangchuangtan/LLaVA-NeXT-Image-Llama3-Lora

Great thank you! Does it also work with Llama3 70b? Btw, does it train only the bridger and language model, or does it also train the vision encoder (that we want to avoid)? Can we train without LoRA ?

[chuangchuangtan]

I implement a LLava-llama3 Lora finetuning https://github.com/chuangchuangtan/LLaVA-NeXT-Image-Llama3-Lora

Great thank you! Does it also work with Llama3 70b? Btw, does it train only the bridger and language model, or does it also train the vision encoder (that we want to avoid)? Can we train without LoRA ?

It only trains the bridge and language model. We have set up to print the names of trainable parameters in the code, you can check them. We haven't tested it on 70b, but it should be work. You can set training commands to train without LoRA.