search

cambrian-mllm / cambrian

Cambrian-1 is a family of multimodal LLMs with a vision-centric design.
https://cambrian-mllm.github.io/
Apache License 2.0
1.4k stars 88 forks source link

there is an error with 'llava_llama' #5

Closed zhangwei730 closed 1 week ago

zhangwei730 commented 1 week ago

here is my error when running the inference.py. I checked that there is no mention of llava_llama whatsoever in this git or in transformer git. So I am confused with this error. thank you !

python inference.py

python inference.py 

[2024-06-27 16:22:53,054] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
2024-06-27 16:22:53 PID=809676 __init__.py:49 setup_logging() INFO → 'standard' logger initialized.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Traceback (most recent call last):
  File "/data/share2/zw/miniconda3/envs/cambrian/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 1118, in from_pretrained
    config_class = CONFIG_MAPPING[config_dict["model_type"]]
  File "/data/share2/zw/miniconda3/envs/cambrian/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 813, in __getitem__
    raise KeyError(key)
KeyError: 'llava_llama'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/share2/zw/githubs/cambrian/inference.py", line 65, in <module>
    tokenizer, model, image_processor, context_len = load_pretrained_model(model_path, None, model_name)
  File "/data/share2/zw/githubs/cambrian/cambrian/model/builder.py", line 146, in load_pretrained_model
    model = AutoModelForCausalLM.from_pretrained(model_path, low_cpu_mem_usage=True, **kwargs)
  File "/data/share2/zw/miniconda3/envs/cambrian/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 526, in from_pretrained
    config, kwargs = AutoConfig.from_pretrained(
  File "/data/share2/zw/miniconda3/envs/cambrian/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 1120, in from_pretrained
    raise ValueError(
ValueError: The checkpoint you are trying to load has model type `llava_llama` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.
phamhe commented 1 week ago

so how do u fix this issue

phamhe commented 1 week ago

ur model name/store path must cotain "cambrian",