Closed enogalaxies closed 3 months ago
Currently, only turbomind backend support vision language model, and turbomind backend doesn't support moe architecture, so you can't deploy mistral model.
Sorry, we are not using moe architecture mistral,we use llava-v1.6-mistral-7b in hugging face link: https://huggingface.co/liuhaotian/llava-v1.6-mistral-7b
when i use pytorch backend the same error occur blew:
lmdeploy serve api_server /data/kai.qiao/model_repo/llava/liuhaotian/llava-v1.6-mistral-7b --server-port 3333 --backend pytorch --model-name mistral
ValueError: The checkpoint you are trying to load has model type llava_mistral
but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.
2024-05-10 17:58:49,668 - lmdeploy - ERROR -
Currently, we only support llava llama. To support llava-mistral, the LlavaMistralConfig
and LlavaMistralForCausalLM
should be used here, but I am not sure the if modification is enough. (including the https://github.com/InternLM/lmdeploy/pull/1579 modification)
We will check and support the model later if it doesn't use the moe or windows attention that turbomind backend doesn't support currently.
Thank you very much for your patience
i compile src and execute always occur error ,like blew
python mistra_7b.py
Fetching 15 files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15/15 [00:00<00:00, 194180.74it/s]
2024-05-13 18:55:03,879 - lmdeploy - INFO - Using turbomind engine
Traceback (most recent call last):
File "/home/kai.qiao/.local/lib/python3.8/site-packages/lmdeploy/vl/model/llava.py", line 21, in check_llava_install
import llava # noqa: F401
File "/home/kai.qiao/.local/lib/python3.8/site-packages/llava/init.py", line 1, in
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "mistra_7b.py", line 4, in
ImportError: To use LlavaVLModel, please install llava by pip install git+https://github.com/haotian-liu/LLaVA.git --no-deps
As guided by the log, pip install llava please
after install the error the same
This issue is marked as stale because it has been marked as invalid or awaiting response for 7 days without any further response. It will be closed in 5 days if the stale label is not removed or if there is no further response.
This issue is closed because it has been stale for 5 days. Please open a new issue if you have similar issues or you have any new updates now.
Checklist
Describe the bug
When attempting to deploy the llava-v1.6-mistral-7b model from HuggingFace using lmdeploy, I encountered an error indicating that the model type llava_mistral is not recognized by the Transformers library. This issue occurred despite the lmdeploy documentation stating that any model in HuggingFace format should be supported for inference. The same process worked previously for a model of type vicuna, but is failing for mistral.Local inference with the model works correctly.
Reproduction
Attempt to deploy the llava-v1.6-mistral-7b model using lmdeploy. Observe the error message in the logs.
Environment
Error traceback