Allow Separate Base URLs for Strong and Weak Models in Controller

[Splasha545]

For my use case, I have different models running on different servers which all replicate the OpenAI completions endpoint. However, from what I can see, it is currently not possible to use both the default OpenAI base URL and a custom server's base URL when defining the controller (or two separate server base URLs). There is functionality to create a Controller client from within a local server running a model, but what if there is no access to run the RouteLLM code from within the server hosting the model.

It would be great if the controller could be provided with a separate base URL for a strong and weak model as, from my understanding, right now, if a base_url is provided, it overrides the base_url used for both the strong and weak model.

Example Use Case: I want to use an OpenAI model like GPT-4o (using the OpenAI completions endpoint) and an open-source model like Mistral running on a custom server with a custom URL (replicating the OpenAI completion endpoint).

Actual Behavior:

• Without providing a base_url parameter to the Controller class, the controller cannot call the Mistral model as it defaults to using the official OpenAI completions endpoint.
• If a custom base_url is provided, the Mistral model works but GPT-4o does not, as the GPT-4o model is not found at this new endpoint.

Steps to Reproduce:

• Define a controller without a base_url parameter.
• Attempt to call a model (e.g., Mistral) hosted on a custom server with its own URL.
• Define a controller with a custom base_url.
• Attempt to call an OpenAI model (e.g., GPT-4o).

Current Workaround: I am currently solving this in a rudimentary way by checking whether the model called in the Controller.completion function (found within kwargs["model"]) matches the strong_model string or the weak_model string and using the corresponding base_url ive provided for each model.

Proposed Solution: Introduce functionality in the Controller class to allow specifying separate base URLs for the strong and weak models.

controller = Controller(
    strong_model="gpt-4o",
    weak_model="openai/mistral",
    strong_model_base_url="https://api.openai.com/v1" # or just None,
    weak_model_base_url="http://custom-endpoint.com/v1"
)

Not sure if I am just misunderstanding something and this functionality does exist. Thank you!

[iojw]

Hi there, thank you very much for the detailed write-up! You're right, this isn't possible with the Controller interface, and I can see how it can be useful.

I'll think about how we can best add this to the API.