[Feature]: Please support Azure APIM

BerriAI / litellm

Call all LLM APIs using the OpenAI format. Use Bedrock, Azure, OpenAI, Cohere, Anthropic, Ollama, Sagemaker, HuggingFace, Replicate (100+ LLMs)

https://docs.litellm.ai/docs/

Other

10.3k stars 1.15k forks source link

[Feature]: Please support Azure APIM #1611

Open MypaceEngine opened 5 months ago

MypaceEngine commented 5 months ago

The Feature

For security, system engineer need to configure APIM setting on endpoint of Azure OpenAI Service. Some times, Engineer need to assign custom http header and certificate of client authentication . Please support these customizable feature. Sample of configuration: ''' model_list:

model_name: gpt-3.5-turbo litellm_params: model: azure/chatgpt-v-2 api_base: https://openai-gpt-4-test-v-1.openai.azure.com/ api_version: "2023-05-15" api_key: "zzzzzzzzzzzzzzzzzzzzz" extra_headers: Ocp-Apim-Subscription-Key: "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" client_certificate:
- "client_certificate.cer"
- "client_key.key" ''' Sample of source: https://github.com/MypaceEngine/litellm/commit/239c802cee24d6d9c3d9989c0282eae8d385a83a

Motivation, pitch

litellm is good tool for enterprise llm management. But enterprise user need to configure high security setting. I think litellm support APIM, Litellm will be popular OSS for LLM management.

Twitter / LinkedIn details

No response

ishaan-jaff commented 5 months ago

Hi @MypaceEngine any chance we can hop on a call to better understand this issue and how we can get litellm to work for you ?

Link to my cal for your convenience https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat?month=2024-02

cakriwut commented 2 weeks ago

Hi @MypaceEngine , @ishaan-jaff

I am able to integrate LiteLLM with APIM without any changes in LiteLLM. My configuration

┌───────┐     ┌──────────┐     ┌────────────┐
│LiteLLM├───► │Azure APIM├───► │Azure OpenAI│
└───────┘     └──────────┘     └────────────┘

The key is to deploy Azure OpenAI (AOI) in the APIM URL ending with openai , example https://apim.public.url/openai or https://apim.public.url/subpath/sub-subpath/openai and configure litellm config to use azure provider, example:

       - model_name: gpt4-vision
         litellm_params:
            model: "azure/gpt4-vision"
            custom_llm_provider: "azure"
            api_base: "https://apim.public.url/subpath/sub-subpath"  # Before stop before openai suffix
            api_key: "xxxxx"     # This is APIM subscription key
            api_version: "2024-02-15-preview"

@ishaan-jaff may be we can add in the Wiki

ishaan-jaff commented 1 week ago

Hi @cakriwut - yes can you make a PR on this please

@cakriwut any chance we can hop on a call ? I'd love to learn how how we can improve litellm for you. Sharing a link to my cal for your convenience: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat