Closed xs818818 closed 2 weeks ago
我希望是在网页上部署类似效果 CUDA_VISIBLE_DEVICES=0 python -m vllm.entrypoints.openai.api_server \ --trust-remote-code \ --max-model-len 4096 \ --model ~/qwen/Qwen1.5-14B-Chat \ --enable-lora \ --lora-modules lora1=~/lora/xxx lora2=~/lora/xxx curl --request POST \ --url http://localhost:8000/v1/chat/completions \ --header 'content-type: application/json' \ --data '{ "model": "lora2", "messages": [ { "role": "system", "content": "You are a helpful assistant." }, { "role": "user", "content": "China is a" } ], "stop_token_ids": [151645, 151644, 151643], "max_tokens": 5, "temperature": 0.7 }'
@xs818818 , v0.9.2
支持了lora的集成,参考文档:https://inference.readthedocs.io/zh-cn/latest/models/lora.html
但是目前不对lora模型进行管理,目前是用户自行下载,与LLM image模型一起launch。
现在UI里如果填了lora config,应该如何请求呢? OpenAI的接口。
This issue is stale because it has been open for 7 days with no activity.
This issue was closed because it has been inactive for 5 days since being marked as stale.
我想在openai接口使用时,可以自由选择不同lora或者原版模型