search

openai / openai-python

The official Python library for the OpenAI API
https://pypi.org/project/openai/
Apache License 2.0
22.91k stars 3.21k forks source link

Better (Azure OpenAI’s content management policy) exception management #786

Closed solyarisoftware closed 12 months ago

solyarisoftware commented 12 months ago

Confirm this is a feature request for the Python library and not the underlying OpenAI API.

Describe the feature or improvement you're requesting

Something between a feature request and better Python module exception management.

When using an OpenAI (chat) create completion API endpoint through an Azure deployment sometime I got an UNEXPECTED exception raised by the Azure content filtering component.

As far as I understand, Azure "prepend" ANY OpenAI completion model deployment by a content filter on top. 

┌────────────────────────────────────────────────────────────┐
│                                                            │
│ Completion API Request with prompt containing the sentence │
│                                                            │
│ “vorrei vedere la cosa più bella ad Ercolano. Qual’è?”     │
│                                                            │
└─────────────────────────────┬──────────────────────────────┘
                              │
                              │            InvalidRequestError
                              │                   ▲
                              │                   │
                              │                   │
                              │                   │
               ┌──────────────┴──────────────┐    │
               │                             │    │
               │   (Python) OpenAI module    │    │
               │                             │    │
               └──────────────┬──────────────┘    │
                              │                   │
                              │                   │
               ┌──────────────▼──────────────┐    │
               │                             │    │
               │    Azure Content Filter     │    │
               │                             │    │
               └──────────────┬──────────────┘    │
                              │                   │
               exception      ▼                   │
               prompt triggering Azure OpenAI’s   │
               content management policy          │
                                                  │
                              │                   │
               ┌──────────────▼──────────────┐    │
               │                             │    │
               │                             │    │
               │   Azure OpenAI Deployment   │    │
               │    (Any Model Behind)       │    │
               │                             │    │
               │                             │    │
               │                             │    │
               └──────────────┬──────────────┘    │
                              │                   │
                              │                   │
                              └───────────────────┘

When testing a chatbot (made with Azure OpenAI completions, e.g. a deployment using a GPT3.5-Turbo policy) in Italian language, I got an exception 

              except openai.error.InvalidRequestError as e:
                  # Handle invalid request error, e.g. validate parameters or log
                  # WARNING: The response was filtered due to the prompt triggering Azure OpenAI’s content management policy.
                  sys.stderr.write(f"OpenAI API request was invalid: {e}")
                  return "Non ho capito bene. Cosa intendi?"

OpenAI API request was invalid: The response was filtered due to the prompt triggering Azure OpenAI’s content management policy. Please modify your prompt and retry. To learn more about our content filtering policies please read our documentation: https://go.microsoft.com/fwlink/?linkid=2198766

I experienced that sometime the Azure content filter component UNEXPECTEDLY trigger an exception with a content category management violation.

As rightly stated in the closed issue https://github.com/openai/openai-python/issues/331#issuecomment-1805017174 that problem is apparently NOT an OpenAI model issue but a maybe WRONG triggering due to a not correct behavior of the Azure filter management. So I already opened an issue to Azure.

Nevertheless, when concerning the Python module engineering I'm a bit perplexed about the fact openAI python SDK raises an InvalidRequestError for an exception NOT related to the OpenAI but raised by an external component (in this case the Azure Filtering component).

I didn't inspected the code, but I think the specific error due to the Azure filter management component could reserve a specific error AzureContentManagementPolicy instead of a generic and misleading InvalidRequestError (sse documentation: reading documentation here: https://help.openai.com/en/articles/6897213-openai-library-error-types-guidance).

Thanks Giorgio

Additional context

$ py --version
Python 3.11.6

$ pip show openai | grep Version
Version: 0.28.1
RobertCraigie commented 12 months ago

Can you share a specific prompt that triggers this error?

The reason InvalidRequestError is raised is because you're calling an API and the API is returning a 400 status code. We don't have any specific exceptions for the Azure API but I agree that would be quite nice.

solyarisoftware commented 12 months ago

We don't have any specific exceptions for the Azure API

That's the point :-)

I can not share the entire prompt, because it's related to an enterprise (Italian language) chatbot for a customer of my company. I can say that 100% the content policy is triggered by apparently innocuous user utterance (part of chat history of the chat completion request) :

User sentence (in Italian): “vorrei vedere la cosa più bella ad Ercolano. Qual’è?”

BTW, translated to English, means: “I would like to see the most beautiful thing in Herculaneum. What is it?”

More details here: https://github.com/openai/openai-python/issues/331#issuecomment-1784833243 .

We agreed that the reason of exception is a possible bug of the Azure content policy component. Nevertheless I separately opened this related "software engineering" enhancement request, as suggested by @rattrayalex here: https://github.com/openai/openai-python/issues/331#issuecomment-1806642248

rattrayalex commented 12 months ago

Ok, now I understand a little better. Thanks for filing this!

I agree it'd be ideal to use a different error class in python for this error; we'll try to do that at some point but I don't think it'll be soon.

In the meantime, while I definitely understand that "invalid request" adds insult to injury in this case, it is the appropriate error class for a generic 400 error from the API right now. After all, while the content policy filter may be wrong, it does think the request is not OK.

So I'm going to close this for now.

solyarisoftware commented 12 months ago

@rattrayalex, all, just FYI:

I got a temporary answer from Azure that suggested me to enable Content Filter "annotations", as described here: https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/content-filter?tabs=python#annotations-preview

That's interesting for debug purposes because, enabling a recent version of API version, e.g.:

Annotations are returned for all scenarios when using 2023-06-01-preview. See also: https://learn.microsoft.com/en-us/azure/ai-services/openai/reference

I now can print out the specific reason of the content filter trigger with a code like: 

except openai.error.InvalidRequestError as e:
    # Handle invalid request error, e.g. validate parameters or log
    error = f"OpenAI API request was invalid: {e}"

    # WARNING: The response was filtered due to the prompt triggering Azure OpenAI’s content management policy.
    if e.error.code == "content_filter" and e.error.innererror:
        content_filter_result = e.error.innererror.content_filter_result

        error += "\nAzure Content Filter Error Result:\n"
        for category, details in content_filter_result.items():
        error += f"{category}: filtered={details['filtered']}, severity={details['severity']}. "

Obtaining the following log:

U: Ciao, come mi puoi aiutare?
A: Ciao! Sono Ercole, l'assistente virtuale per il turismo di Ercolano. Posso aiutarti ad esplorare la città, cosa ti serve sapere?

U: vorrei vedere la cosa più bella ad Ercolano. Qual'è?
A: Non ho capito bene. Cosa intendi?
ERROR: OpenAI API request was invalid: The response was filtered due to the prompt triggering Azure OpenAI's content management policy. Please modify your prompt and retry. To learn more about our content filtering policies please read our documentation: https://go.microsoft.com/fwlink/?linkid=2198766

Azure Content Filter Error Result:
hate: filtered=False, severity=safe. self_harm: filtered=False, severity=safe. sexual: filtered=True, severity=medium. violence: filtered=False, severity=safe.

All in all, having this exception error details, developer could exactly define the reason of the exception. At the end I agree to close the issue because the python module management is specific and complete.

In my opinion the triggering of the content_filter is wrong in the example, but Azure is in charge for this.

Thanks giorgio