search

vllm-project / vllm

A high-throughput and memory-efficient inference and serving engine for LLMs
https://docs.vllm.ai
Apache License 2.0
27.91k stars 4.12k forks source link

api_server.py: error: unrecognized arguments: --lora-modules sql-lora=~/.cache/huggingface/hub/models--yard1--llama-2-7b-sql-lora-test/ #2909

Open xueyongfu11 opened 7 months ago

xueyongfu11 commented 7 months ago

run codes below python -m vllm.entrypoints.api_server \ --model meta-llama/Llama-2-7b-hf \ --enable-lora \ --lora-modules sql-lora=~/.cache/huggingface/hub/models--yard1--llama-2-7b-sql-lora-test/

raise an error: api_server.py: error: unrecognized arguments: --lora-modules sql-lora=~/.cache/huggingface/hub/models--yard1--llama-2-7b-sql-lora-test/

jacob-hansen commented 7 months ago

Same error when trying to launch docker from vllm/vllm-openai:latest

I'm going to try repulling the docker with the commits from a few days ago. (pending)

jacob-hansen commented 7 months ago

Ok, I still get the error with the new docker. It isn't an available argument.

I recommend checking out this example for how to load the lora after launching the engine: https://github.com/vllm-project/vllm/blob/main/examples/multilora_inference.py

1jsingh commented 7 months ago

yes same error as well. Is there a solution for this? ... to serve lora trained models using vlllm? Thanks!

JessyMu commented 7 months ago

build env from source can fix it. https://docs.vllm.ai/en/latest/getting_started/installation.html#build-from-source

Stannislav commented 6 months ago

Upgrading to vllm==0.3.3 solved this for me.

victoraranda commented 3 months ago

Instead of filing a new issue, tagging on to this open issue. I am having a similar experience with vllm-openai:0.5.0 where it is giving me the same message. It seems that the --lora-modules argument is unrecognized. I'll try building from source, but is this a bug?