search

mbzuai-oryx / LLaVA-pp

🔥🔥 LLaVA++: Extending LLaVA with Phi-3 and LLaMA-3 (LLaVA LLaMA-3, LLaVA Phi-3)
814 stars 61 forks source link

inference error #23

Closed TuuSiwei closed 6 months ago

TuuSiwei commented 6 months ago

I merge the lora weight and use it for infering with your infer script，but i encounter the error：

(llava) root@bj1oj9u6aucjn-0:/x/tsw/llavapp/LLaVA-pp/LLaVA# python run_llava.py [2024-05-10 09:15:01,902] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect) Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [02:38<00:00, 39.64s/it] Loading LoRA weights from ./checkpoints/llama3-finetune-lora Merging weights Convert to FP16... Traceback (most recent call last): File "/x/tsw/llavapp/LLaVA-pp/LLaVA/run_llava.py", line 152, in eval_model(args) File "/x/tsw/llavapp/LLaVA-pp/LLaVA/run_llava.py", line 67, in eval_model if model.config.mm_use_im_start_end: File "/x/sherlor/envs/llava/lib/python3.10/site-packages/transformers/configuration_utils.py", line 265, in getattribute return super().getattribute(key) AttributeError: 'LlamaConfig' object has no attribute 'mm_use_im_start_end'

the config file after merging is to simple

mmaaz60 commented 6 months ago

Hi @tsw123678,

Thank you for your interest in our work. Please note the following for running LLaMA-3-V model inference,

  1. The transformers version needs to be latest, e.g. pip install git+https://github.com/huggingface/transformers@a98c41798cf6ed99e1ff17e3792d6e06a2ff2ff3
  2. You need to use the llama3 chat template.

However, from the error you shared, it looks like something is wrong with the checkpoints. Could you please try if you can run the inference using our provided merged checkpoints at https://huggingface.co/MBZUAI/LLaVA-Meta-Llama-3-8B-Instruct?

Thank You