search

BerriAI / litellm

Python SDK, Proxy Server to call 100+ LLM APIs using the OpenAI format - [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, Replicate, Groq]
https://docs.litellm.ai/docs/
Other
12.28k stars 1.43k forks source link

[Help]: Set individual budgets for members in a team #5345

Closed edwins closed 3 weeks ago

edwins commented 3 weeks ago

What happened?

I am currently using litellm proxy via api only.

This is the container image version ghcr.io/berriai/litellm:main-v1.44.2

TLDR; team budget seems to work, individual member budgets do not (when using /user/new)

I attempted to create a team with a budget and then add users linked to a team. Creating the user on the team seemed to work as expected along with the budget, but when we test the individual max_budget, it doesn't seem to work.

Here is the curl to create the team:

curl --location 'https://example.com/team/new' --header "Authorization: Bearer $LITELLM_MASTER_KEY" \
--header 'Content-Type: application/json' \
--data-raw '{
  "team_alias": "llm-budget-testing",
  "organization_id":"some-org-id",
  "models":["gpt-4o", "gpt-4o-mini", "gpt-4-turbo", "gpt-4","openai/*"],
  "max_budget": 5,
  "budget_duration":"30d"
}'

The output:

{"team_alias":"llm-budget-testing","team_id":"some-team-id","organization_id":"some-org-id","admins":[],"members":[],"members_with_roles":[{"role":"admin","user_id":"default_user_id","user_email":null}],"metadata":{},"tpm_limit":null,"rpm_limit":null,"max_budget":5.0,"budget_duration":"30d","models":["gpt-4o","gpt-4o-mini","gpt-4-turbo","gpt-4","openai/*"],"blocked":false,"spend":0.0,"max_parallel_requests":null,"budget_reset_at":"2024-09-14T19:06:08.946000Z","model_id":null}

Here is the curl to create the user, linked to a team and what I thought would create an individual budget on the team:

curl --location 'https://example.com/user/new' --header "Authorization: Bearer $LITELLM_MASTER_KEY" \
--header 'Content-Type: application/json' \
--data-raw '{
  "user_id": "budget10c",
  "user_email": "budget10c@cyverse.org",
  "team_id": "some-team-id",
  "organization_id":"some-org-id",
  "auto_create_key": true,
  "max_budget": 0.1,
  "budget_duration":"30d"
}'

The output

{"models":[],"spend":0.0,"max_budget":0.1,"user_id":"budget10c","team_id":"some-team-id","max_parallel_requests":null,"metadata":{},"tpm_limit":null,"rpm_limit":null,"budget_duration":"30d","allowed_cache_controls":[],"soft_budget":null,"key_alias":null,"duration":null,"aliases":{},"config":{},"permissions":{},"model_max_budget":{},"send_invite_email":null,"key":"some-key","key_name":null,"expires":null,"token_id":null,"user_email":"budget10c@cyverse.org","user_role":null,"teams":null,"organization_id":null}

If we should be creating individual budgets on a team in a different way, feel free to let me know.

Thank you, E

Relevant log output

No response

Twitter / LinkedIn details

No response

ishaan-jaff commented 3 weeks ago

hi @edwins - you should use team member budgets for this use case: https://docs.litellm.ai/docs/proxy/users

Does team member budgets solve your problem ?

Screenshot 2024-08-23 at 2 17 30 PM
krrishdholakia commented 3 weeks ago

This isn't a bug @edwins, setting budget on user applies to that user id, independent of teams.

I believe the approach suggested by ishaan should solve this. Will leave the ticket open in case the suggested approach doesn't work for you @edwins

edwins commented 3 weeks ago

I tried /team/member_add to update the budget, which resulted in an internal server error. Should I /team/member_delete and then /team/member_add?

ishaan-jaff commented 3 weeks ago

can you share a stacktrace of the error you got @edwins ?

edwins commented 3 weeks ago

So I tried the following:

First deleting

curl -s -X POST 'https://example.com/team/member_delete' --header "Authorization: Bearer $LITELLM_MASTER_KEY" --header 'Content-Type: application/json' --data-raw '{
    "team_id": "team-id-here",
    "user_id": "budget50c"
}'

The resulting output does not display budget50c in output, as expected.

Then I executed the /team/member_add:

curl --location 'https://example.com/team/member_add' --header "Authorization: Bearer $LITELLM_MASTER_KEY" --header 'Content-Type: application/json' --data-raw '{
  "team_id": "team-id-here",
  "max_budget_in_team": 0.5,
  "member": {"role": "user", "user_id": "budget50c"}
}'
Internal Server Error

The logs show:

db-1       | 2024-08-23 21:50:36.331 UTC [3011] ERROR:  duplicate key value violates unique constraint "LiteLLM_TeamMembership_pkey"
db-1       | 2024-08-23 21:50:36.331 UTC [3011] DETAIL:  Key (user_id, team_id)=(budget50c, team-id-here) already exists.
db-1       | 2024-08-23 21:50:36.331 UTC [3011] STATEMENT:  INSERT INTO "public"."LiteLLM_TeamMembership" ("user_id","team_id","spend","budget_id") VALUES ($1,$2,$3,$4) RETURNING "public"."LiteLLM_TeamMembership"."user_id", "public"."LiteLLM_TeamMembership"."team_id", "public"."LiteLLM_TeamMembership"."spend", "public"."LiteLLM_TeamMembership"."budget_id" /* traceparent=00-00000000000000000000000000000000-0000000000000000-01 */
litellm-1  | INFO:     172.18.0.1:53350 - "POST /team/member_add HTTP/1.0" 500 Internal Server Error
litellm-1  | ERROR:    Exception in ASGI application
litellm-1  | Traceback (most recent call last):
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 411, in run_asgi
litellm-1  |     result = await app(  # type: ignore[func-returns-value]
litellm-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
litellm-1  |     return await self.app(scope, receive, send)
litellm-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
litellm-1  |     await super().__call__(scope, receive, send)
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
litellm-1  |     await self.middleware_stack(scope, receive, send)
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
litellm-1  |     raise exc
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
litellm-1  |     await self.app(scope, receive, _send)
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
litellm-1  |     await self.app(scope, receive, send)
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
litellm-1  |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
litellm-1  |     raise exc
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
litellm-1  |     await app(scope, receive, sender)
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
litellm-1  |     await self.middleware_stack(scope, receive, send)
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
litellm-1  |     await route.handle(scope, receive, send)
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
litellm-1  |     await self.app(scope, receive, send)
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
litellm-1  |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
litellm-1  |     raise exc
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
litellm-1  |     await app(scope, receive, sender)
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
litellm-1  |     response = await func(request)
litellm-1  |                ^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
litellm-1  |     raw_response = await run_endpoint_function(
litellm-1  |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
litellm-1  |     return await dependant.call(**values)
litellm-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/proxy/management_helpers/utils.py", line 338, in wrapper
litellm-1  |     raise e
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/proxy/management_helpers/utils.py", line 253, in wrapper
litellm-1  |     result = await func(*args, **kwargs)
litellm-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/proxy/management_endpoints/team_endpoints.py", line 518, in team_member_add
litellm-1  |     await add_new_member(
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/proxy/management_helpers/utils.py", line 121, in add_new_member
litellm-1  |     await prisma_client.db.litellm_teammembership.create(
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/prisma/actions.py", line 12517, in create
litellm-1  |     resp = await self._client._execute(
litellm-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/prisma/client.py", line 525, in _execute
litellm-1  |     return await self._engine.query(builder.build(), tx_id=self._tx_id)
litellm-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/prisma/engine/query.py", line 244, in query
litellm-1  |     return await self.request(
litellm-1  |            ^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/prisma/engine/http.py", line 141, in request
litellm-1  |     return utils.handle_response_errors(resp, errors_data)
litellm-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/prisma/engine/utils.py", line 192, in handle_response_errors
litellm-1  |     raise exc(error)
litellm-1  | prisma.errors.UniqueViolationError: Unique constraint failed on the fields: (`user_id`,`team_id`)
krrishdholakia commented 3 weeks ago

curl -s -X POST 'https://example.com/team/member_delete' --header "Authorization: Bearer $LITELLM_MASTER_KEY" --header 'Content-Type: application/json' --data-raw '{

it looks like this user wasn't successfully deleted from the team. I'll try and repro this + push a fix

krrishdholakia commented 3 weeks ago

@edwins unable to repro the issue

What do your logs show on /member_delete?

I'm able to successfully add/delete/re-add with no errors

Screenshot 2024-08-24 at 8 48 43 AM Screenshot 2024-08-24 at 8 48 39 AM
krrishdholakia commented 3 weeks ago

@edwins exposed 2 new fields - updated_users and updated_team_memberships for /team/member_add to show the max budget per team member being added to db