[Feature]: Add cooldowns to router

[krrishdholakia]

The Feature

If a call fails against a deployment, add a 1min cooldown to the deployment

Motivation, pitch

user request

Twitter / LinkedIn details

No response

[krrishdholakia]

received 2 user requests for this. will aim to have this done by EOD.

[krrishdholakia]

[krrishdholakia]

v0 logic

• use litellm.failure_callback to identify a failed call -> put deployment in cool down for 60s

[krrishdholakia]

initial version pushed - https://github.com/BerriAI/litellm/commits/main

Missing

• storing overloaded servers in global store (e.g. redis). Keeping it in-memory in router, means even basic multi-threaded tests have a separate cooldown dict for each thread (not ideal - leads to multiple wasted calls to known overloaded servers).

[krrishdholakia]

v2 pushed with redis support - https://github.com/BerriAI/litellm/commit/5d5ca9f7efcd0b912472d677aa6d651d79d3f5d1

Here's how to run it:

def test_multiple_deployments(): 
    import concurrent, time
    # litellm.set_verbose=True
    futures = {}
    model_list = [{ # list of model deployments 
        "model_name": "gpt-3.5-turbo", # openai model name 
        "litellm_params": { # params for litellm completion/embedding call 
            "model": "azure/chatgpt-v-2", 
            "api_key": "bad-key",
            "api_version": os.getenv("AZURE_API_VERSION"),
            "api_base": os.getenv("AZURE_API_BASE")
        },
        "tpm": 240000,
        "rpm": 1800
    }, 
    {
        "model_name": "gpt-3.5-turbo", # openai model name 
        "litellm_params": { # params for litellm completion/embedding call 
            "model": "gpt-3.5-turbo", 
            "api_key": os.getenv("OPENAI_API_KEY"),
        },
        "tpm": 1000000,
        "rpm": 9000
    }
    ]

    router = Router(model_list=model_list, 
                 redis_host=os.getenv("REDIS_HOST"), 
                 redis_password=os.getenv("REDIS_PASSWORD"), 
                 redis_port=int(os.getenv("REDIS_PORT")), 
                 routing_strategy="simple-shuffle",
                 num_retries=1) # type: ignore
    kwargs = {
            "model": "gpt-3.5-turbo",
            "messages": [{"role": "user", "content": """Context:

In the historical era of Ancient Greece, a multitude of significant individuals lived, contributing immensely to various disciplines like science, politics, philosophy, and literature. For instance, Socrates, a renowned philosopher, primarily focused on ethics. His notable method, the Socratic Method, involved acknowledging one's own ignorance to stimulate critical thinking and illuminate ideas. His student, Plato, another prominent figure, founded the Academy in Athens. He proposed theories on justice, beauty, and equality, and also introduced the theory of forms, which is pivotal to understanding his philosophical insights. Another student of Socrates, Xenophon, distinguished himself more in the domain of history and military affairs.

Aristotle, who studied under Plato, led an equally remarkable life. His extensive works have been influential across various domains, including science, logic, metaphysics, ethics, and politics. Perhaps most notably, a substantial portion of the Western intellectual tradition traces back to his writings. He later tutored Alexander the Great who went on to create one of the most vast empires in the world.

In the domain of mathematics, Pythagoras and Euclid made significant contributions. Pythagoras is best known for the Pythagorean theorem, a fundamental principle in geometry, while Euclid, often regarded as the father of geometry, wrote "The Elements", a collection of definitions, axioms, theorems, and proofs. 

Apart from these luminaries, the period also saw a number of influential political figures. Pericles, a prominent and influential Greek statesman, orator, and general of Athens during the Golden Age, specifically between the Persian and Peloponnesian wars, played a significant role in developing the Athenian democracy.

The Ancient Greek era also witnessed extraordinary advancements in arts and literature. Homer, credited with the creation of the epic poems 'The Iliad' and 'The Odyssey,' is considered one of the greatest poets in history. The tragedies of Sophocles, Aeschylus, and Euripides left an indelible mark on the field of drama, and the comedies of Aristophanes remain influential even today.

---
Question: 

Who among the mentioned figures from Ancient Greece contributed to the domain of mathematics and what are their significant contributions?"""}],
    }

    results = [] 

    for _ in range(10): 
        print(f"starting!!!")
        response = router.completion(**kwargs)
        results.append(response)

test_multiple_deployments()

[krrishdholakia]

pushed a fix for cooldowns with caching improvements - supports using a dual cache (in-memory + redis) and just works more consistently.