Closed XkunW closed 2 months ago
Are you using a local model?
python3 -m vllm.entrypoints.openai.api_server \
--model /llava-v1.6-mistral-7b \
I suspect that you have a typo in the model name.
Are you using a local model?
python3 -m vllm.entrypoints.openai.api_server \ --model /llava-v1.6-mistral-7b \
I suspect that you have a typo in the model name.
Sorry I made a typo in my description, it should be /llava-v1.6-mistral-7b-hf
, which is what I have in my actual launching command, and yes it's a local model.
Are you using vLLM w/ CPU (you may have to build from source)? I haven't tested LLaVA using CPU yet so there may be some incompatibilities. Try setting --dtype float
?
Are you using vLLM w/ CPU (you may have to build from source)? I haven't tested LLaVA using CPU yet so there may be some incompatibilities. Try setting
--dtype float
?
I'm using a single A40 GPU, and setting --dtype float
did help it go beyond this error, I just need to figure out the right image feature size now, thanks!
Your current environment
🐛 Describe the bug
I'm trying to use a single A40 GPU to host
llava-hf/llava-v1.6-mistral-7b-hf
using the OpenAI compatible server withvllm 0.5.0
. Here is the command I used to launch the server:And I got this error after loading the model weights: