Open cynthiajiangatl opened 1 year ago
Thanks @cynthiajiangatl and ack on the settings. Can you do a PR to fix them please?
@lordlinus @cynthiajiangatl
I'm the maintainer of LiteLLM we provide an Open source proxy for load balancing Azure + OpenAI It can process (500+ requests/second)
From this thread it looks like you're trying to load balance between OpenAI instance - I hope our solution makes it easier for you. (i'd love feedback if you're trying to do this)
Doc: https://docs.litellm.ai/docs/simple_proxy#load-balancing---multiple-instances-of-1-model
model_list:
- model_name: gpt-4
litellm_params:
model: azure/chatgpt-v-2
api_base: https://openai-gpt-4-test-v-1.openai.azure.com/
api_version: "2023-05-15"
api_key:
- model_name: gpt-4
litellm_params:
model: azure/gpt-4
api_key:
api_base: https://openai-gpt-4-test-v-2.openai.azure.com/
- model_name: gpt-4
litellm_params:
model: azure/gpt-4
api_key:
api_base: https://openai-gpt-4-test-v-2.openai.azure.com/
litellm --config /path/to/config.yaml
curl --location 'http://0.0.0.0:8000/chat/completions' \
--header 'Content-Type: application/json' \
--data ' {
"model": "gpt-4",
"messages": [
{
"role": "user",
"content": "what llm are you"
}
],
}
'
This issue is for a: (mark with an
x
)Minimal steps to reproduce
Any log messages given by the failure
Expected/desired behavior
OS and Version?
Versions
Mention any other details that might be useful
client.py has errors in the code when calling search client. need to remove the additional line 134 credential=AzureKeyCredential(AZURE_SEARCH_KEY))
Since three OpenAI services are used for load balancing, need to add those to azfunctions.bicep or provide instructions on the configuration updates to be made, default configuration for the storage account was incorrect, should be the storage account for documents.