[Feature request] Support Llama3

[thesby]

feature

Could you please support Llama3 in Llava ?

[awzhgw]

+1

[HarryHsing]

+1

[iMountTai]

+1

[GoGoJoestar]

+1

[manbehindthemadness]

+1

[dingtine]

i have trained llava with llama3 model, but the generate result is not correct.

[Namzakku]

@dingtine can you specify more about the result? also, which base model did you train on?

[manbehindthemadness]

@dingtine there is some mention of an abnormality regarding an end/termination token mentioned here: https://x.com/bartowski1182/status/1782206933719515467?s=46&t=iIhAbXdfE1VCk7vAgMnlRQ as this came out just now, it might affect your results.

[mmaaz60]

Hi @thesby @awzhgw , @Namzakku @manbehindthemadness,

I hope you are doing well. We have just released our project LLaVA++: Extending Visual Capabilities with LLaMA-3 and Phi-3, which features LLaMA-3 and Phi-3-Mini based LLaVA models. Please have a look at this at LLaVA++.

• We have released the codes required to support both LLaMA-3 & Phi-3-Mini models in LLaVA framework. The chat formats and corresponding preprocess methods are available at our GitHub repo.
• We released all the checkpoints on Hugging Face
• On our GitHub repository we have provided .py files that needs to be replaced/added to official LLaVA repository to train and infer LLaMA-3 & Phi-3-Mini based models.

Further, as pointed out by @manbehindthemadness, the issues related to generation have been fixed in the recent update of generation_config.json and tokenizer.json at meta-llama/Meta-Llama-3-8B-Instruct.

In case if you face any issue in running/training LLaMA-3 or Phi-3-Mini based LLaVA models, please let me know.

[manbehindthemadness]

Fantastic!