Ability to bypass usage of `litellm.Router` - Githubissues

Future-House / paper-qa

High accuracy RAG for answering questions from scientific documents with citations

Apache License 2.0

6.46k stars 618 forks source link

Ability to bypass usage of `litellm.Router` #563

Closed jamesbraza closed 1 month ago

jamesbraza commented 1 month ago

There are multiple outstanding issues with LiteLLM's Router as of litellm==1.48.10:

It doesn't support TPM: https://github.com/BerriAI/litellm/issues/5783
The Router has several race conditions that keep killing our training runs

https://github.com/Future-House/paper-qa/pull/520 in-housed rate limits management, which was most of the reason we adopted the Router in https://github.com/Future-House/paper-qa/pull/315. Now the Router just enables model grouping, which we don't really use internally at FutureHouse. Paired with the Router's race conditions, we decided to make usage of the Router optional.

So, this PR:

Creates a back door to bypass Router usage, instead using LiteLLM's free functions
Removes "router" mentions from the code base where stuff is actually just "model list" specific

krrishdholakia commented 1 month ago

The Router has several race conditions that keep killing our training runs

Hi @jamesbraza what are these race conditions?