[Bug]: litellm.exceptions.APIError: AzureException - {"error": {"code": "content_filter_error", "message": "The contents are not filtered"}}

[langgg0511]

What happened?

def tesy_async_acompletion():
    litellm.set_verbose = False
    print("test_async_completion with stream")

    async def test_get_response():
        try:
            response = await litellm.acompletion(
                model="azure/gpt-35-turbo-1106",
                messages=[{"role": "user", "content": "say 1"}],
                temperature=0,
                top_p=1,
                stream=True,
                max_tokens=512,
                presence_penalty=0,
                frequency_penalty=0,
            )
            print(f"response: {response}")

            num_finish_reason = 0
            async for chunk in response:
                print(chunk)
                if chunk["choices"][0].get("finish_reason") is not None:
                    num_finish_reason += 1
                    print("finish_reason", chunk["choices"][0].get("finish_reason"))

            assert (
                num_finish_reason == 1
            ), f"expected only one finish reason. Got {num_finish_reason}"
        except Exception as e:
            pytest.fail(f"GOT exception for gpt-3.5 instruct In streaming{e}")

    asyncio.run(test_get_response())

https://github.com/BerriAI/litellm/blob/main/litellm/utils.py#L7376 if str_line.choices[0].content_filter_result=={} will raise AzureOpenAIError

Relevant log output

Traceback (most recent call last):                                                                                                        
  File "/root/.pyenv/versions/llm-pool/lib/python3.10/site-packages/litellm/utils.py", line 7738, in chunk_creator
    response_obj = self.handle_openai_chat_completion_chunk(chunk)                                                                        
  File "/root/.pyenv/versions/llm-pool/lib/python3.10/site-packages/litellm/utils.py", line 7318, in handle_openai_chat_completion_chunk  
    raise e                                                                                                                               
  File "/root/.pyenv/versions/llm-pool/lib/python3.10/site-packages/litellm/utils.py", line 7306, in handle_openai_chat_completion_chunk  
    raise litellm.AzureOpenAIError(                                                                                                       
litellm.llms.azure.AzureOpenAIError: {"error": {"code": "content_filter_error", "message": "The contents are not filtered"}}

During handling of the above exception, another exception occurred:

Twitter / LinkedIn details

No response

[Extremys]

I can confirm I have also this error since at least 7 or 8 versions. for gpt-4 model only, for the gpt-4-32k model no issue weirdly with Azure. There my config file if it's helping:

model_list:
  - model_name: gpt-4-32k
    litellm_params:
      model: azure/gpt-4-32k
      api_version: "2023-05-15"
      api_base: "os.environ/AZURE_API_BASE"
      api_key: "os.environ/AZURE_API_KEY"
  - model_name: gpt-4
    litellm_params:
      model: azure/gpt-4
      api_version: "2023-05-15"
      api_base: "os.environ/AZURE_API_BASE"
      api_key: "os.environ/AZURE_API_KEY"

litellm_settings:
  set_verbose: True
  drop_params: True
  success_callback: ["langfuse"]

environment_variables:
  LANGFUSE_PUBLIC_KEY: {{ LANGFUSE_PUBLIC_KEY }}
  LANGFUSE_SECRET_KEY: {{ LANGFUSE_SECRET_KEY }}
  LANGFUSE_HOST: "{{ LANGFUSE_HOST }}"
  AZURE_API_KEY: {{ AZURE_API_KEY }}
  AZURE_API_BASE: {{ AZURE_API_URL }}

[krrishdholakia]

Thanks for filing this @Extremys @langgg0511 will investigate and push a fix today hopefully

[krrishdholakia]

Just pushed a fix w/ testing - https://github.com/BerriAI/litellm/commit/23b59ac9b8a78ece3809a2ed33390063a75319fa

We now check for content policy violation errors during streaming by checking the finish reason. This is based on the azure docs - https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/content-filter?tabs=warning%2Cpython#scenario-details

Will update ticket once a release is out

[Extremys]

Fixed on my side :) thanks!

[krrishdholakia]

Closing issue as the fix is live in v1.18.12 onwards

[hooman-bayer]

@krrishdholakia this issue still occurs occasionally. In my case with "model": "gpt-35-turbo-1106" and I am using 1.19.2

6741, in exception_type
    raise e
  File "python3.11/site-packages/litellm/utils.py", line 6680, in exception_type
    raise APIError(
          └ <class 'litellm.exceptions.APIError'>

litellm.exceptions.APIError: AzureException - {"error": {"code": "content_filter_error", "message": "The contents are not filtered"}}

[krrishdholakia]

@hooman-bayer isn't that just azure raising the content filter error?

We're now just explicitly checking if the finish reason is a content filter and re-raising that