How to enable Vision enhancements for Azure Open AI?

[ishaan-jaff]

Discussed in https://github.com/BerriAI/litellm/discussions/1433

^{Originally posted by **kishorek** January 12, 2024} How can we enable Vision enhancements for Azure Open AI GPT 4 Vision preview? It is suggested to enable it for enhancing the OCR - https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/gpt-with-vision#use-vision-enhancement-with-images The below parameters need to be added to the payload for `chat/completions`: ```json "enhancements": {"ocr": {"enabled": true}, "grounding": {"enabled": false}}, "dataSources": [ { "type": "AzureComputerVision", "parameters": { "endpoint": "https://.cognitiveservices.azure.com", "key": "***", }, } ]

[ishaan-jaff]

I think we already support this - perhaps we need to verify + add to docs

[ishaan-jaff]

okay so from looking at this we need to pass the following in extra_body for Azure Requests:

"enhancements": {
            "ocr": {
              "enabled": true
            },
            "grounding": {
              "enabled": true
            }
    },
    "dataSources": [
    {
        "type": "AzureComputerVision",
        "parameters": {
            "endpoint": "<your_computer_vision_endpoint>",
            "key": "<your_computer_vision_key>"
        }
    }],

[ishaan-jaff]

@kishorek can you test this on 1.17.6 and let me know if it works ?

[ishaan-jaff]

https://docs.litellm.ai/docs/providers/azure#usage---with-azure-vision-enhancements

[kishorek]

Thanks for the response @ishaan-jaff . Unfortunately, I am getting the below error.

Traceback (most recent call last):
  File "<home>/.cache/pypoetry/virtualenvs/litellm_test-px-5zgsC-py3.10/lib/python3.10/site-packages/litellm/llms/azure.py", line 250, in completion
    response = azure_client.chat.completions.create(**data, timeout=timeout)  # type: ignore
  File "<home>/.cache/pypoetry/virtualenvs/litellm_test-px-5zgsC-py3.10/lib/python3.10/site-packages/openai/_utils/_utils.py", line 271, in wrapper
    return func(*args, **kwargs)
TypeError: Completions.create() got an unexpected keyword argument 'enhancements'

litellm==1.17.5

[kishorek]

Tested the same in 1.17.6 from litellm_bedrock_provisioned_throughput branch. Still the same error

litellm.exceptions.APIError: AzureException - Completions.create() got an unexpected keyword argument 'enhancements'

[ishaan-jaff]

Hi @kishorek can you try this on the latest version of litellm https://github.com/BerriAI/litellm/releases/tag/v1.17.13

I'm trying to get this set up on my Azure account too, to test

[ishaan-jaff]

was able to test this on my Azure account, fix here: https://github.com/BerriAI/litellm/pull/1475

[ishaan-jaff]

@kishorek can you try this release and lmk if it works for you: https://github.com/BerriAI/litellm/releases/tag/v1.17.17

docs: https://docs.litellm.ai/docs/providers/azure#usage---with-azure-vision-enhancements

Note,base_url needs /extensions to work on Azure

[ishaan-jaff]

bump on this @kishorek ? can you confirm it's working ?

[kishorek]

Hi @ishaan-jaff This is perfect! Its working as expected. Thank you so much. We can close this issue