Closed firmament2008 closed 5 months ago
老兄,请问部署这个项目是否必须要显卡?我能否直接用openAI 的API?因为我只想体验下chat to excel
老兄,请问部署这个项目是否必须要显卡?我能否直接用openAI 的API?因为我只想体验下chat to excel
用 代理模式,不需一定要显卡。 CPU也能推理。 如 通过 litellm 代理 ollama 的模型,前提是部署好ollama模型。
# 兼容 openai 接口代理
litellm --model ollama/qwen:7b --port 4000 &
# db-gpt 的 proxyllm 配置如
PROXY_API_KEY=abc
PROXY_SERVER_URL=http://localhost:4000/v1/chat/completions
Search before asking
Operating system information
MacOS(x86)
Python version information
3.10
DB-GPT version
main
Related scenes
Installation Information
[x] Installation From Source
[ ] Docker Installation
[ ] Docker Compose Installation
[ ] Cluster Installation
[ ] AutoDL Image
[ ] Other
Device information
CPU
Models information
proxyllm
What happened
When registering a worker for dbgpt, the Manage Host information incorrectly changes to the worker's host and port details, which is incorrect..
What you expected to happen
dbgpt start webserver --port 8088 --model_name proxyllm
dbgpt start worker --port 8100 --model_name proxyllm-qwen-7b --worker_type llm --model_path '' --proxy_server_url http://localhost:4000/v1/chat/completions --proxy_api_key 'abc' --controller_addr http://127.0.0.1:8088
How to reproduce
branch : main commit ID : dce03862d582dab50ba8be583d97b667006f96fd
code change info
model_config add line "proxyllm-qwen-7b": "chatgpt_proxyllm", proxy_adapter change: def do_match(self, lower_model_name_or_path: Optional[str] = None): return lower_model_name_or_path in ["chatgpt_proxyllm", "proxyllm" ,"proxyllm-qwen-7b"]
Additional context
No response
Are you willing to submit PR?