Open ishaan-jaff opened 4 months ago
If we add this, then request header key can be added to the data, which then goes to the usual flow.
File: proxy_server.py
# Reading API KEY From header
if req_header_api_key := request.headers.get('X-API-KEY'):
data["api_key"] = req_header_api_key
My concern is:
litellm.use_llm_key_in_header=True
this can only work if one is using import litellm
, correct, then what about the rest calls, how the rest client can set that parameter?After looking into the Pull-Request feedback, i spend quite sometime figuring things out, here's is my findings:
OpenAPI and Anthropic usage SDK and HTTP requests to serve the responses, here is how it looks like:
Following is rest api call, api key is being passed as Authorization: Bearer ...
curl -X POST --location "https://api.openai.com/v1/chat/completions" \
-H "Authorization: Bearer sk-proj-.........." \
-H "Content-Type: application/json" \
-d .......
Following is with Python sdk:
from openai import OpenAI
client = OpenAI(api_key='sk-proj-....')
stream = client.chat.completions.create(
model="gpt-3.5-turbo",
....
Following is the rest api call, api key is being passed as x-api-key: ....
curl https://api.anthropic.com/v1/messages --header "x-api-key: YOUR_API_KEY" ...
Following is using python sdk:
import anthropic
client = anthropic.Anthropic(
api_key="my_api_key",
....
We support starting the proxy in two form:
# run: litellm --config config.yaml
model_list:
- model_name: gpt-3.5-turbo
litellm_params:
model: gpt-3.5-turbo
litellm --model gpt-3.5-turbo
And we want to support if someone uses following format:
from openai import OpenAI
client = OpenAI(api_key='sk-proj-...', base_url='http://0.0.0.0:4000')
stream = client.chat.completions.create(
model="gpt-3.5-turbo",
...
Correct me here @ishaan-jaff . Supporting this, means it should also support for the HTTP request for both Anthropic & OpenAPI.
While trying the above approach i am getting:
raise AuthenticationError(
litellm.exceptions.AuthenticationError: OpenAIException - The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable
INFO: 127.0.0.1:64570 - "POST /chat/completions HTTP/1.1" 500 Internal Server Error
Further diving deep, i found out that the litellm poxy server also capable of using the Bearer:
if isinstance(api_key, str):
passed_in_key = api_key
api_key = _get_bearer_token(api_key=api_key)
If a user is sending the Bearer to access the litellm proxy vs if a user is requesting using the OpenAI SDK passing the api_key will conflict with each other.
However, I've adjusted the code, which should work. Tested for OpenAI SDK.
This works in two ways:
litellm.use_llm_key_in_header=True
CC: @ishaan-jaff
Following up, let me know feedback if there are any.
CC: @krrishdholakia / @ishaan-jaff
The Feature
litellm.use_llm_key_in_header=True
Motivation, pitch
user request
Twitter / LinkedIn details
No response