Introduces a Controller interface for RouteLLM that allows it to be used as a Python SDK without having to launch a server. The controller can act as a drop-in replacement for the OpenAI client and has parity with the server API.
from routellm.controller import Controller
# Replace OpenAI client with Controller
# client = OpenAI()
client = Controller(
# By default, Controller uses the best performing configuration.
routers=["mf"],
strong_model="gpt-4-1106-preview",
weak_model="anyscale/mistralai/Mixtral-8x7B-Instruct-v0.1",
)
client.chat.completions.create(
# Or, you can specify the router and threshold directly as keyword arguments.
model="router-mf-0.116"
messages=[
{"role": "user", "content": "Hello!"},
],
)
The controller also supports returning the model name to route to for a given prompt.
routed_model = controller.route(prompt="What's the square root of 144?", router="mf", threshold=0.116)
The server and evaluation framework now use Controller to manage their routers through a unified interface!
Introduces a
Controller
interface for RouteLLM that allows it to be used as a Python SDK without having to launch a server. The controller can act as a drop-in replacement for the OpenAI client and has parity with the server API.The controller also supports returning the model name to route to for a given prompt.
The server and evaluation framework now use
Controller
to manage their routers through a unified interface!TODO: