Unable to use the GPT4 Vision enhancement with the latest version of GPT4 Vision Preview

[saikiran-bandaru]

While setting up LiteLLM to use with the Azure Vision enhancements, the documentation suggests to use the /extensions as part of the URL. However, for the latest recommended preview version 2024-02-15-preview, it is unable to find the resource. It was working fine with 2023-12-01-preview.

This is the the Router model that is working fine for 2023-12-01-preview:

{
    "model_name": <model-name>,
    "litellm_params": {
            "api_key": **********,
            "api_version": "2023-12-01-preview",
            "model": "azure/<azure-openai-deployment-name>",
            "api_base": "<azure-api-base>/openai/deployments/<azure-openai-deployment-name>/extensions",
            "enhancements": {
                    "ocr": { "enabled": True },
                    "grounding": {"enabled": True},
             },
            "dataSources": [
                            {
                                "type": "AzureComputerVision",
                                "parameters": {
                                    "endpoint": "https://<endpoint>.cognitiveservices.azure.com",
                                    "key": **********,
                                },
                            }
            ],
     },
    "tpm": <tpm>,
    "rpm": <rpm>
}

When using the 2024-02-15-preview version, the regular chat completions are working fine with the following setup:

{
    "model_name": <model-name>,
    "litellm_params": {
            "api_key": **********,
            "api_version": "2024-02-15-preview",
            "model": "azure/<azure-openai-deployment-name>",
            "api_base": "<azure-api-base>/openai/deployments/<azure-openai-deployment-name>/chat/completions?api-version=2024-02-15-preview",
            "enhancements": {
                    "ocr": { "enabled": False },
                    "grounding": {"enabled": False},
             }
     },
    "tpm": <tpm>,
    "rpm": <rpm>
}

However, when using the GPT4 Vision enhancement with the following Router model setup:

{
    "model_name": <model-name>,
    "litellm_params": {
            "api_key": **********,
            "api_version": "2024-02-15-preview",
            "model": "azure/<azure-openai-deployment-name>",
            "api_base": "<azure-api-base>/openai/deployments/<azure-openai-deployment-name>/extensions",
            "enhancements": {
                    "ocr": { "enabled": False },
                    "grounding": {"enabled": False},
             },
            "dataSources": [
                            {
                                "type": "AzureComputerVision",
                                "parameters": {
                                    "endpoint": "https://<endpoint>.cognitiveservices.azure.com",
                                    "key": **********,
                                },
                            }
            ],
     },
    "tpm": <tpm>,
    "rpm": <rpm>
}

I am getting the following error.

Response:

Exception AzureException - Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}}

Is there any change required to the Router model setup or URLs?

Relevant log output

Exception AzureException - Error code: 404 - {'error': {'code': '404', 'message': 'Resource not found'}}

[krrishdholakia]

cc: @ishaan-jaff ?