search

stanfordnlp / dspy

DSPy: The framework for programming—not prompting—language models
https://dspy.ai
MIT License
19.37k stars 1.47k forks source link

Configuring rate limiter (throttler) #1572

Closed denisergashbaev closed 1 month ago

denisergashbaev commented 1 month ago

LiteLLM provides rate limit aware routing strategy that routes the call to the deployment with the lowest tokens per minute value (see https://github.com/BerriAI/litellm/discussions/4510, https://docs.litellm.ai/docs/routing#advanced---routing-strategies-%EF%B8%8F).

How could we configure it in DSPy?

denisergashbaev commented 1 month ago

essentially a duplicate of https://github.com/stanfordnlp/dspy/issues/1570