(Feat) Add provider specific budget routing

[ishaan-jaff]

Provider Budget Routing

Use this to set budgets for LLM Providers - example $100/day for OpenAI, $100/day for Azure.

model_list:
    - model_name: gpt-3.5-turbo
      litellm_params:
        model: openai/gpt-3.5-turbo
        api_key: os.environ/OPENAI_API_KEY
    - model_name: gpt-3.5-turbo
      litellm_params:
        model: azure/chatgpt-functioncalling
        api_key: os.environ/AZURE_API_KEY
        api_version: os.environ/AZURE_API_VERSION
        api_base: os.environ/AZURE_API_BASE

router_settings:
  redis_host: <your-redis-host>
  redis_password: <your-redis-password>
  redis_port: <your-redis-port>
  provider_budget_config: 
    openai: 
            budget_limit: 0.000000000001 # float of $ value budget for time period
        time_period: 1d # can be 1d, 2d, 30d 
    azure:
        budget_limit: 100
        time_period: 1d
    anthropic:
        budget_limit: 100
        time_period: 10d
    vertexai:
        budget_limit: 100
        time_period: 12d
    gemini:
        budget_limit: 100
        time_period: 12d

general_settings:
  master_key: sk-1234

How provider-budget-routing works

1. Budget Tracking:

   • Uses Redis to track spend for each provider
   • Tracks spend over specified time periods (e.g., "1d", "30d")
   • Automatically resets spend after time period expires

2. Routing Logic:

   • Routes requests to providers under their budget limits
   • Skips providers that have exceeded their budget
   • If all providers exceed budget, raises an error

3. Supported Time Periods:

   • Format: "Xd" where X is number of days
   • Examples: "1d" (1 day), "30d" (30 days)

4. Requirements:

   • Redis required for tracking spend across instances
   • Provider names must be litellm provider names. See Supported Providers

Relevant issues

Type

🆕 New Feature 🐛 Bug Fix 🧹 Refactoring 📖 Documentation 🚄 Infrastructure ✅ Test

Changes

[REQUIRED] Testing - Attach a screenshot of any new tests passing locall

If UI changes, send a screenshot/GIF of working UI fixes

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
litellm	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Nov 20, 2024 1:54am

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 90.81633% with 9 lines in your changes missing coverage. Please review.	Files with missing lines	Patch %	Lines
litellm/router_strategy/provider_budgets.py	89.65%	9 Missing :warning:

:loudspeaker: Thoughts on this report? Let us know!