[Usage] Fail to load llava-llama-2-7b-chat-lightning-lora-preview KeyError: 'LlavaConfig'

[heylamourding]

Describe the issue

Issue:

Command:

tokenizer, model, image_processor, context_len = load_pretrained_model(model_path = 'liuhaotian/llava-llama-2-7b-chat-lightning-lora-preview', 
                     model_base = 'liuhaotian/llava-llama-2-7b-chat-lightning-lora-preview',
                     model_name = 'llava-llama-2-7b-chat-lightning-lora-preview')

Log:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[3], line 1
----> 1 tokenizer, model, image_processor, context_len = load_pretrained_model(model_path = 'liuhaotian/llava-llama-2-7b-chat-lightning-lora-preview', 
      2                      model_base = 'liuhaotian/llava-llama-2-7b-chat-lightning-lora-preview',
      3                      model_name = 'llava-llama-2-7b-chat-lightning-lora-preview')

File ~/jupyter_notebooks/LLaVA/llava/model/builder.py:45, in load_pretrained_model(model_path, model_base, model_name, load_8bit, load_4bit, device_map)
     43 if 'lora' in model_name.lower() and model_base is not None:
     44     lora_cfg_pretrained = AutoConfig.from_pretrained(model_path)
---> 45     tokenizer = AutoTokenizer.from_pretrained(model_base, use_fast=False)
     46     print('Loading LLaVA from base model...')
     47     model = LlavaLlamaForCausalLM.from_pretrained(model_base, low_cpu_mem_usage=True, config=lora_cfg_pretrained, **kwargs)

File ~/miniconda3/envs/llava/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:718, in AutoTokenizer.from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs)
    716 model_type = config_class_to_model_type(type(config).__name__)
    717 if model_type is not None:
--> 718     tokenizer_class_py, tokenizer_class_fast = TOKENIZER_MAPPING[type(config)]
    719     if tokenizer_class_fast and (use_fast or tokenizer_class_py is None):
    720         return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)

File ~/miniconda3/envs/llava/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:663, in _LazyAutoMapping.__getitem__(self, key)
    661 if key in self._extra_content:
    662     return self._extra_content[key]
--> 663 model_type = self._reverse_config_mapping[key.__name__]
    664 if model_type in self._model_mapping:
    665     model_name = self._model_mapping[model_type]

KeyError: 'LlavaConfig'

Screenshots: You may attach screenshots if it better explains the issue.

[haotian-liu]

Please check out the updated instruction for loading lora weights. You need to pass a corresponding --model-base, and in your case llama-2-7b-chat.

[zhangyupeng123]

请问可以生成类似“liuhaotian/llava-llama-2-13b-chat-lightning-preview”的，7b版本的文件吗？

[heylamourding]

Hi @haotian-liu thanks for your reply!

I tried to send python -m llava.serve.model_worker --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --model-path liuhaotian/llava-llama-2-7b-chat-lightning-lora-preview --model-base llama-2-7b-chat

I got following errors: OSError: llama-2-7b-chat is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'

May I know what value should I put under model-base?

Alternatively, is it possible to generate liuhaotian/llava-llama-2-7b-chat-lightning-preview?

Thanks!

[zhangyupeng123]

Hi @haotian-liu , Can we generate a 7b version file like “liuhaotian/llava-llama-2-13b-chat-lightning-preview” by ourselves?

[heylamourding]

Updated: error is solved. Need to get llama2 access in HF first. Then

python merge_lora_weights.py --model-path liuhaotian/llava-llama-2-7b-chat-lightning-lora-preview --model-base meta-llama/Llama-2-7b-chat-hf --save-model-path ./checkpoints/llava-7b-llama-2-7b-chat

Use ./checkpoints/llava-7b-llama-2-7b-chat in inference.

[zhangyupeng123]

Hi @heylamourding ,I have got the llava-2 access in HF, but how do I use it on the server?