search

lm-sys / RouteLLM

A framework for serving and evaluating LLM routers - save LLM costs without compromising quality!
Apache License 2.0
2.78k stars 204 forks source link

Provide support for more than two models and provide a training guide. #23

Open BrainSlugs83 opened 1 month ago

BrainSlugs83 commented 1 month ago

It looks like this only supports two models, a strong and a weak model. But there are other things to consider like if privacy is a concern, or if the question is math heavy, or if the question has a visual element, etc.

Why not have a RouteLLM that could route to several arbitrary models (including local, self-hosted, or models as a service like GPT4).

And provide some example training scripts and/or a training guide that we could use to fine tune this.

iojw commented 1 month ago

Hi, thanks for raising this! That's definitely something we've been thinking about for while.

We did our initial research focusing on only 2 models as a start, but more research is required to build routers that work well with multiple models.

jonpsy commented 4 weeks ago

Hey @iojw , would the team need some help in this regard? I could help with benchmarks and reporting results and/or write code to extend this feature. What do you say?

bitnom commented 2 weeks ago

Hi, thanks for raising this! That's definitely something we've been thinking about for while.

We did our initial research focusing on only 2 models as a start, but more research is required to build routers that work well with multiple models.

This is the direction I want to go personally. Please share your thoughts on testing and implementation. Are you already working on it or not yet?

villqrd commented 2 weeks ago

@bitnom currently working on more than 2 models, you can see some of our results and read more about it here: https://tryplurally.com/.