feat: track the cost associated with each user

[changchiyou]

Is your feature request related to a problem? Please describe. Currently, it's challenging to track the cost associated with each user.

Describe the solution you'd like

1. Modify the current method of generating user API keys: implement a process where user keys are generated by invoking the <litellm>/key/generate API.
2. Users initiate conversations through LiteLLM models using their unique API keys.
3. Provide an admin link for accessing https://litellm.vercel.app/docs/proxy/ui from the admin panel.

Describe alternatives you've considered

Modify the current method of generating user API keys: implement a process where user keys are generated by invoking the <litellm>/key/generate API.

1. Disable registration function.
2. (The admin) Register an open-webui(hosted web) account for user and retrieves <litellm>/key/generate api using the email address
3. Manually updating the generated API into the API field.
4. After all, provide the account with email/password to user, notify user update the default password with their own password.

Additional context

Reference:

1. https://litellm.vercel.app/docs/proxy/virtual_keys
2. https://litellm.vercel.app/docs/proxy/users

[tjbck]

Related: open-webui/pipelines#20, we should try to add this feature to the rest of our proxy servers as well.

[max00346]

This would also solve my Feature-Request: open-webui/open-webui#1320

[krrishdholakia]

@changchiyou do you want to create a key per user?

LiteLLM also allows for tracking by the 'user' param in /chat/completions -

curl --location 'http://0.0.0.0:4000/chat/completions' \
        --header 'Content-Type: application/json' \
        --header 'Authorization: Bearer sk-zi5onDRdHGD24v0Zdn7VBA' \
        --data ' {
        "model": "azure-gpt-3.5",
        "user": "ishaan3", # 👈 TRACKING COST FOR THIS USER
        "messages": [
            {
            "role": "user",
            "content": "what time is it"
            }
        ]
        }'

https://docs.litellm.ai/docs/proxy/users

[changchiyou]

@krrishdholakia Does the method you provide, which uses user instead of an API key to track users, require additional manual adjustments/settings on the Open WebUI? I observed that Open WebUI does not send out the user parameter by default:

https://github.com/open-webui/open-webui/blob/d43ee0fc5b018cec183de99e8047472c454737ae/src/routes/(app)/%2Bpage.svelte#L602-L666

and the logs in LiteLLM show that the user is always default_user_id:

litellm-proxy  | Request to litellm:
litellm-proxy  | litellm.acompletion(model='azure/gpt-35-turbo_0125', api_key='xxx', api_base='xxx', messages=[{'role': 'user', 'content': '說「HI」'}], caching=False, client=<openai.lib.azure.AsyncAzureOpenAI object at 0x7fe75dce7310>, timeout=6000, stream=True, proxy_server_request={'url': 'http://litellm-proxy:8000/v1/chat/completions', 'method': 'POST', 'headers': {'host': 'litellm-proxy:8000', 'user-agent': 'python-requests/2.31.0', 'accept-encoding': 'gzip, deflate', 'accept': '*/*', 'connection': 'keep-alive', 'authorization': 'Bearer sk-1234', 'content-type': 'application/json', 'content-length': '109'}, 'body': {'model': 'gpt-3.5-turbo', 'stream': True, 'messages': [{'role': 'user', 'content': '說「HI」'}]}}, user='default_user_id', metadata={'user_api_key': 'sk-1234', 'user_api_key_alias': None, 'user_api_key_user_id': 'default_user_id', 'user_api_key_team_id': None, 'user_api_key_metadata': {}, 'headers': {'host': 'litellm-proxy:8000', 'user-agent': 'python-requests/2.31.0', 'accept-encoding': 'gzip, deflate', 'accept': '*/*', 'connection': 'keep-alive', 'content-type': 'application/json', 'content-length': '109'}, 'endpoint': 'http://litellm-proxy:8000/v1/chat/completions', 'model_group': 'gpt-3.5-turbo', 'deployment': 'azure/gpt-35-turbo_0125', 'model_info': {'base_model': 'azure/gpt-3.5-turbo-0125', 'id': '69f51c2e-d0fb-4b40-b722-7664a58366e4'}, 'caching_groups': None}, request_timeout=600, model_info={'base_model': 'azure/gpt-3.5-turbo-0125', 'id': '69f51c2e-d0fb-4b40-b722-7664a58366e4'}, max_retries=0)

[juancarlosm]

same problem here! We are unable to track costs per user:

[juancarlosm]

If it helps, a dirty hack in backend/apps/[openai|litellm]/main.py at proxy function:

_tmp = json.loads(body) _tmp['user'] = user.email body = json.dumps(_tmp)

And now I can see the right user at langfuse:

[kroonen]

We've added LiteLLM support to our pipelines examples (see : https://github.com/open-webui/pipelines/blob/main/pipelines/examples/litellm_manifold_pipeline.py and https://github.com/open-webui/pipelines/blob/main/pipelines/examples/litellm_subprocess_manifold_pipeline.py), enabling per-user cost tracking. This will be included with the v0.2.0 release, currently in the :dev branch. (We'll be leveraging OpenAI's endpoint to create plugins. Full docs will be available soon!)

[juancarlosm]

https://github.com/open-webui/open-webui/discussions/3494