Vllm completions/chat api connectivity [Roadmap]

enricoros / big-AGI

Generative AI suite powered by state-of-the-art models and providing advanced AI/AGI functions. It features AI personas, AGI functions, multi-model chats, text-to-image, voice, response streaming, code highlighting and execution, PDF import, presets for developers, much more. Deploy on-prem or in the cloud.

https://big-agi.com

MIT License

4.64k stars 1.06k forks source link

Vllm completions/chat api connectivity [Roadmap] #512

Closed Playerrrrr closed 1 month ago

Playerrrrr commented 1 month ago

Why i like running localhost port local llms via VLLM, would be awesome if I had an UI to interact and continue my work as llm dev on big-agi, instead of running my own custom python scripts. If there is a way to connect them, please enlighten me, thank you in advance!

enricoros commented 1 month ago

Can you explain me more, I don't understand the workflow. Have you tried using a generic "OpenAI" model vendor, with advanced options to set host and port?