BerriAI / litellm

Python SDK, Proxy Server (LLM Gateway) to call 100+ LLM APIs in OpenAI format - [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, Replicate, Groq]
https://docs.litellm.ai/docs/
Other
14.29k stars 1.68k forks source link

[Budget Manager] Expose - budgetmanager.user_under_budget() #354

Closed ishaan-jaff closed 1 year ago

ishaan-jaff commented 1 year ago

The current interface feels complicated

Desired interface:

if budget_manager.user_under_budget(user=user_id):
    response = completion(model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Hey, how's it going?"}], user_id=user_id)

Here is what my existing code looks like:

current_spend_for_user = budget_manager.get_current_cost(user=user_id)
budget_for_user = budget_manager.get_total_budget(user_id)
print(f"User: {user_id} has spent ${current_spend_for_user}, budget for user: ${budget_for_user}\n")
if current_spend_for_user <= budget_for_user:
    response = completion(model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Hey, how's it going?"}])
    budget_manager.update_cost(completion_obj=response, user=user_id)
krrishdholakia commented 1 year ago

acknowledging this issue - i don't feel comfortable with abstracting too much of this away from the user but i think you might be right. Let me think about it tonight. Will update ticket.