[Discussion]: Setting model name during client init

[MypaceEngine]

What happened?

litellm create wrong URL pattern for Azure OpenAI. Example of Azure OpenAI Endpoint: http://xxxx.openai.azure.com/openai/deployments/chatgpt-2 But litellm create http://xxxx.openai.azure.com/openai/ and use this for communication to Azure OpenAI service. litellm get HTTP error code and communication is faulure. OpenAI Proxy does not work in case of Azure OpenAI.

I define config.yaml:

model_list:
  - model_name: gpt-3.5-turbo
    litellm_params:
      model: azure/chatgpt-v-2
      api_base: https://openai-gpt-4-test-v-1.openai.azure.com/
      api_version: "2023-05-15"
      api_key: "xxxxxxxxxxxxxxxxxxxxx"

I think https://cognitiveservices.azure.com is not permitted in some company for company rule. In this situation, deployments id is needed. like https://github.com/BerriAI/litellm/blob/main/litellm/llms/azure.py. If user assign deployment id like azure/chatgpt-v-2. Please assign deployment id to openai.AzureOpenAI/openai.AsyncAzureOpenAI.

Sample source of patch: https://github.com/BerriAI/litellm/commit/00513f31bbc8d959c129bf12970c8a14317d775a

Relevant log output

No response

Twitter / LinkedIn details

No response

[krrishdholakia]

Hi @MypaceEngine, azure openai accepts model being passed in during the completion call, and that's where we set it - https://github.com/BerriAI/litellm/blob/7e06944d80aeecfd5f02372a91cd72be052f8572/litellm/llms/azure.py#L351

Openai client supports this by reading the optional params passed in during a call and appending the model to the url - https://github.com/openai/openai-python/blob/15488ce07cb97535d8564e82dd5cda3481bc1a81/src/openai/lib/azure.py#L56

Can you share any specific error you hit? This passes our testing

[MypaceEngine]

In azure.py, AzureOpenAI client is created by connection information include "deployment id". but in line 336-line 339, if client is received from parent, the client is used. the AzureOpenAI client of azure.py is not used. https://github.com/BerriAI/litellm/blob/7e06944d80aeecfd5f02372a91cd72be052f8572/litellm/llms/azure.py#L321-L339

On proxy mode, this method (acompletion) received client. https://github.com/BerriAI/litellm/blob/7e06944d80aeecfd5f02372a91cd72be052f8572/litellm/llms/azure.py#L299-L311 I trucked where do client come from. At startup time, These client is created on router.py: https://github.com/BerriAI/litellm/blob/00513f31bbc8d959c129bf12970c8a14317d775a/litellm/router.py#L1339C5-L1339C18 And these client is cached on memory. on all of azure proxy request, cached client use for execution. But "deployment id" is not assign on creating AzureOpenAI client. This AzureOpenAI client create URL not include "deployment id" and connection is failure. I would like to fix this probrem.

[MypaceEngine]

For example: In this area, connection information of AzureOpenAI client is created. "deployment id" is not assigned https://github.com/BerriAI/litellm/blob/652f4f89d028cde0cbd54462c6e84e9691684c0e/litellm/router.py#L1513-L1517

[krrishdholakia]

@MypaceEngine Repeating earlier request. Can you share a code script or stacktrace of the problem you had.

The openai client uses deployment id and model interchangeably. The code I shared shows how they set it dynamically at runtime when model is passed.

You can also see the azure QuickStart on openai python's GitHub for this.