Support for Ollama Planned/Possible?

lm-sys / RouteLLM

A framework for serving and evaluating LLM routers - save LLM costs without compromising quality!

Apache License 2.0

2.78k stars 204 forks source link

Support for Ollama Planned/Possible? #2

Closed Heavy02011 closed 1 month ago

Heavy02011 commented 1 month ago

Any ideas / plans to support ollama? maybe following a similar approach as for TextGrad which runs perfectly on an embedded device like Jetson Orin.

PS. Awesome work, thx for sharing.

iojw commented 1 month ago

Hi there! Since Ollama supports an OpenAI-compatible server (https://ollama.com/blog/openai-compatibility), you can already use it with RouteLLM. Just launch the Ollama server, specify the --alt-base-url flag to point RouteLLM your local Ollama server, and everything should work. Let me know if you have any other questions.

I'll update the README to make this clearer.