[Feature]: Proxy - Support passing through OpenAI, anthropic API Keys from request headers

ishaan-jaff commented 4 months ago

The Feature

User wants to pass their OpenAI, Anthropic API Key in the request header
LiteLLM should use the api key passed in the header to make the LLM API Call
This should be opt in, litellm.use_llm_key_in_header=True

Motivation, pitch

user request

Twitter / LinkedIn details

No response

paneru-rajan commented 4 months ago

If we add this, then request header key can be added to the data, which then goes to the usual flow.

File: proxy_server.py

        # Reading API KEY From header
        if req_header_api_key := request.headers.get('X-API-KEY'):
            data["api_key"] = req_header_api_key

My concern is:

Is this correct way?
litellm.use_llm_key_in_header=True this can only work if one is using import litellm, correct, then what about the rest calls, how the rest client can set that parameter?
I think we need to make some changes in the documentations as well
Which will get more priority if someone happen to pass keys from config and or header etc.

paneru-rajan commented 4 months ago

After looking into the Pull-Request feedback, i spend quite sometime figuring things out, here's is my findings:

OpenAPI and Anthropic usage SDK and HTTP requests to serve the responses, here is how it looks like:

OpenAI Key format

Following is rest api call, api key is being passed as Authorization: Bearer ...

curl -X POST --location "https://api.openai.com/v1/chat/completions" \
    -H "Authorization: Bearer sk-proj-.........." \
    -H "Content-Type: application/json" \
    -d .......

Following is with Python sdk:

from openai import OpenAI

client = OpenAI(api_key='sk-proj-....')

stream = client.chat.completions.create(
    model="gpt-3.5-turbo",
....

Anthropic API Key format

Following is the rest api call, api key is being passed as x-api-key: ....

curl https://api.anthropic.com/v1/messages --header "x-api-key: YOUR_API_KEY" ...

Following is using python sdk:

import anthropic

client = anthropic.Anthropic(
    api_key="my_api_key",
....

We support starting the proxy in two form:

With config: This approach looks for api_key to be there:

# run:  litellm --config config.yaml 
model_list:
- model_name: gpt-3.5-turbo
litellm_params:
  model: gpt-3.5-turbo

without config: This does not require the api_key to be there:
```
litellm --model gpt-3.5-turbo
```

And we want to support if someone uses following format:

from openai import OpenAI

client = OpenAI(api_key='sk-proj-...', base_url='http://0.0.0.0:4000')

stream = client.chat.completions.create(
    model="gpt-3.5-turbo",
...

Correct me here @ishaan-jaff . Supporting this, means it should also support for the HTTP request for both Anthropic & OpenAPI.

While trying the above approach i am getting:

    raise AuthenticationError(
litellm.exceptions.AuthenticationError: OpenAIException - The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable
INFO:     127.0.0.1:64570 - "POST /chat/completions HTTP/1.1" 500 Internal Server Error

Further diving deep, i found out that the litellm poxy server also capable of using the Bearer:

if isinstance(api_key, str):
    passed_in_key = api_key
    api_key = _get_bearer_token(api_key=api_key)

If a user is sending the Bearer to access the litellm proxy vs if a user is requesting using the OpenAI SDK passing the api_key will conflict with each other.

However, I've adjusted the code, which should work. Tested for OpenAI SDK.

This works in two ways:

Set litellm.use_llm_key_in_header=True
Don't initialize the api_key, and pass it using the SDK or http header

CC: @ishaan-jaff

paneru-rajan commented 3 months ago

Following up, let me know feedback if there are any.

CC: @krrishdholakia / @ishaan-jaff

BerriAI / litellm