Closed Heavy02011 closed 1 month ago
Hi there! Since Ollama supports an OpenAI-compatible server (https://ollama.com/blog/openai-compatibility), you can already use it with RouteLLM. Just launch the Ollama server, specify the --alt-base-url
flag to point RouteLLM your local Ollama server, and everything should work. Let me know if you have any other questions.
I'll update the README to make this clearer.
Any ideas / plans to support ollama? maybe following a similar approach as for TextGrad which runs perfectly on an embedded device like Jetson Orin.
PS. Awesome work, thx for sharing.