Add additional logging capabilities to AI Proxy

[iMicknl]

Since Azure API Management is not able to do end to end logging on streaming requests, we will need to support this in the AI Proxy. It should have multiple options

• Option 1: log all requests and response

• Option 2: log only requests and response when content filters are triggered, for abuse monitoring

  • Only log request where content filter (blocking - filtered)

  • Only log request where content filter (detected - annotated) - only available when turned off

  • Only log request where Additional models (blocking - filtered)

  • Only log request where Additional models (detected - annotated)

  • Only log request where blocklist (custom or pre-built) (blocking - filtered)

  • Only log request where blocklist (custom or pre-built) (detected - annotated)

Logging options: CosmosDB, Azure Monitor/Log Analytics, Blob Storage

Since many customers have turned off abuse monitoring, this is an important capability to implement.

Key	Value	Content Type
LoggingStrategy	"all"	string

Valid options: ["all", "abuse-monitoring", "none"]

[iMicknl]

See https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/content-filter?tabs=warning%2Cpython.

We need to double check if this currently works for our streaming requests, or if our AI proxy will throw an error due to a different payload.

Example response (non streaming) (api-version=2023-03-15-preview)

{
    "error": {
        "message": "The response was filtered due to the prompt triggering Azure OpenAI's content management policy. Please modify your prompt and retry. To learn more about our content filtering policies please read our documentation: https://go.microsoft.com/fwlink/?linkid=2198766",
        "type": null,
        "param": "prompt",
        "code": "content_filter",
        "status": 400
    }
}

Example response (non streaming) (api-version=2023-10-01-preview)

{
    "error": {
        "message": "The response was filtered due to the prompt triggering Azure OpenAI's content management policy. Please modify your prompt and retry. To learn more about our content filtering policies please read our documentation: https://go.microsoft.com/fwlink/?linkid=2198766",
        "type": null,
        "param": "prompt",
        "code": "content_filter",
        "status": 400,
        "innererror": {
            "code": "ResponsibleAIPolicyViolation",
            "content_filter_result": {
                "hate": {
                    "filtered": false,
                    "severity": "safe"
                },
                "self_harm": {
                    "filtered": false,
                    "severity": "safe"
                },
                "sexual": {
                    "filtered": false,
                    "severity": "safe"
                },
                "violence": {
                    "filtered": true,
                    "severity": "medium"
                }
            }
        }
    }
}

Example response, with jailbreak detection enabled (non streaming) (api-version=2023-10-01-preview) Additional objects can be present in the content_filter_result.

{
    "error": {
        "message": "The response was filtered due to the prompt triggering Azure OpenAI's content management policy. Please modify your prompt and retry. To learn more about our content filtering policies please read our documentation: https://go.microsoft.com/fwlink/?linkid=2198766",
        "type": null,
        "param": "prompt",
        "code": "content_filter",
        "status": 400,
        "innererror": {
            "code": "ResponsibleAIPolicyViolation",
            "content_filter_result": {
                "hate": {
                    "filtered": false,
                    "severity": "safe"
                },
                "jailbreak": {
                    "detected": true,
                    "filtered": true
                },
                "self_harm": {
                    "filtered": false,
                    "severity": "safe"
                },
                "sexual": {
                    "filtered": false,
                    "severity": "safe"
                },
                "violence": {
                    "filtered": false,
                    "severity": "safe"
                }
            }
        }
    }
}

[azureholic]

The proxy detects errors already (but there is no implementation yet). We need to figure out what the error is and handle it accordingly.

The proxy will respond to the client with the original error response. Handling applies to token calculation and eventual policy violation logging (feature not present yet).