For my use case, I have different models running on different servers which all replicate the OpenAI completions endpoint. However, from what I can see, it is currently not possible to use both the default OpenAI base URL and a custom server's base URL when defining the controller (or two separate server base URLs). There is functionality to create a Controller client from within a local server running a model, but what if there is no access to run the RouteLLM code from within the server hosting the model.
It would be great if the controller could be provided with a separate base URL for a strong and weak model as, from my understanding, right now, if a base_url is provided, it overrides the base_url used for both the strong and weak model.
Example Use Case:
I want to use an OpenAI model like GPT-4o (using the OpenAI completions endpoint) and an open-source model like Mistral running on a custom server with a custom URL (replicating the OpenAI completion endpoint).
Actual Behavior:
Without providing a base_url parameter to the Controller class, the controller cannot call the Mistral model as it defaults to using the official OpenAI completions endpoint.
If a custom base_url is provided, the Mistral model works but GPT-4o does not, as the GPT-4o model is not found at this new endpoint.
Steps to Reproduce:
Define a controller without a base_url parameter.
Attempt to call a model (e.g., Mistral) hosted on a custom server with its own URL.
Define a controller with a custom base_url.
Attempt to call an OpenAI model (e.g., GPT-4o).
Current Workaround:
I am currently solving this in a rudimentary way by checking whether the model called in the Controller.completion function (found within kwargs["model"]) matches the strong_model string or the weak_model string and using the corresponding base_url ive provided for each model.
Proposed Solution:
Introduce functionality in the Controller class to allow specifying separate base URLs for the strong and weak models.
controller = Controller(
strong_model="gpt-4o",
weak_model="openai/mistral",
strong_model_base_url="https://api.openai.com/v1" # or just None,
weak_model_base_url="http://custom-endpoint.com/v1"
)
Not sure if I am just misunderstanding something and this functionality does exist. Thank you!
Hi there, thank you very much for the detailed write-up! You're right, this isn't possible with the Controller interface, and I can see how it can be useful.
I'll think about how we can best add this to the API.
For my use case, I have different models running on different servers which all replicate the OpenAI completions endpoint. However, from what I can see, it is currently not possible to use both the default OpenAI base URL and a custom server's base URL when defining the controller (or two separate server base URLs). There is functionality to create a
Controller
client from within a local server running a model, but what if there is no access to run the RouteLLM code from within the server hosting the model.It would be great if the controller could be provided with a separate base URL for a strong and weak model as, from my understanding, right now, if a base_url is provided, it overrides the base_url used for both the strong and weak model.
Example Use Case: I want to use an OpenAI model like GPT-4o (using the OpenAI completions endpoint) and an open-source model like Mistral running on a custom server with a custom URL (replicating the OpenAI completion endpoint).
Actual Behavior:
Steps to Reproduce:
Current Workaround: I am currently solving this in a rudimentary way by checking whether the model called in the
Controller.completion
function (found withinkwargs["model"]
) matches thestrong_model
string or theweak_model
string and using the corresponding base_url ive provided for each model.Proposed Solution: Introduce functionality in the Controller class to allow specifying separate base URLs for the strong and weak models.
Not sure if I am just misunderstanding something and this functionality does exist. Thank you!