Support for content filters in Azure OpenAI

dkmiller commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues

Community Note

Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Description

Azure OpenAI has introduced the ability to configure content filters (How to configure content filters with Azure OpenAI Service).

Terraform already supports specifying a custom policy name via rai_policy_name. However, it does not appear that Terraform currently supports creating custom policies.

The underlying REST call made to create a new content filter is looks like below.

PUT https://management.azure.com/subscriptions/5001b330-4456-4782-8a95-5be13fe6f00a/resourceGroups/my-resource-group/providers/Microsoft.CognitiveServices/accounts/my-openai-account/raiPolicies/MyCustomRaiPolicy?api-version=2023-06-01-preview

with a body like

{
  "name": "Test",
  "displayName": "",
  "properties": {
    "basePolicyName": "Microsoft.Default",
    "type": "UserManaged",
    "contentFilters": [
      {
        "name": "hate",
        "blocking": false,
        "enabled": true,
        "allowedContentLevel": "medium",
        "source": "prompt"
      },
      {
        "name": "sexual",
        "blocking": true,
        "enabled": true,
        "allowedContentLevel": "high",
        "source": "prompt"
      },
      {
        "name": "selfharm",
        "blocking": true,
        "enabled": true,
        "allowedContentLevel": "low",
        "source": "prompt"
      },
      {
        "name": "violence",
        "blocking": true,
        "enabled": true,
        "allowedContentLevel": "medium",
        "source": "prompt"
      },
      {
        "name": "hate",
        "blocking": true,
        "enabled": true,
        "allowedContentLevel": "low",
        "source": "completion"
      },
      {
        "name": "sexual",
        "blocking": false,
        "enabled": true,
        "allowedContentLevel": "medium",
        "source": "completion"
      },
      {
        "name": "selfharm",
        "blocking": true,
        "enabled": true,
        "allowedContentLevel": "medium",
        "source": "completion"
      },
      {
        "name": "violence",
        "blocking": true,
        "enabled": true,
        "allowedContentLevel": "high",
        "source": "completion"
      }
    ]
  }
}

New or Affected Resource(s)/Data Source(s)

azurerm_cognitive_rai_policy

Potential Terraform Configuration

resource "azurerm_cognitive_rai_policy" "my_policy" {
  name                 = "MyCustomRaiPolicy"
  display_name         = "My custom RAI policy"
  cognitive_account_id = "/subscriptions/5001b330-4456-4782-8a95-5be13fe6f00a/resourceGroups/my-resource-group/providers/Microsoft.CognitiveServices/accounts/my-openai-account"
  content_filters = [
    {
      name = "hate"
      blocking = false
      enabled = true // This is the default.
      allowed_content_level = "high"
      source = "prompt"
    },
    {
      name = "sexual"
      blocking = true // This is the default.
      // Omit default value for `enabled`.
      allowed_content_level = "high"
      source = "completion"
    }
  ]
}

References

RBAC docs for Microsoft.CognitiveServices/accounts/raiPolicies/*: https://learn.microsoft.com/en-us/azure/role-based-access-control/resource-provider-operations

Original, incorrectly-filed, issue: https://github.com/Azure/terraform-azurerm-openai/issues/28.

liuwuliuyun commented 1 year ago

Hi @dkmiller, thanks for raising this enhancement request. I will track this request and try to implement it. For anyone needs this right now, I would recommend using azapi provider to deploy resource with preview API version.

dkmiller commented 1 year ago

@liuwuliuyun TIL the azapi provider exists, thanks so much! For anyone wishing to use Terraform to create RAI policies in the interim, here's the code I used:

terraform {
  required_providers {
    azapi = {
      source  = "azure/azapi"
      version = "~> 1.8.0"
    }
  }
}

resource "azapi_resource" "no_moderation_policy" {
  type      = "Microsoft.CognitiveServices/accounts/raiPolicies@2023-06-01-preview"
  name      = "NoModerationPolicy"
  parent_id = var.parent_id

  schema_validation_enabled = false

  body = jsonencode({
    displayName = ""
    properties = {
      basePolicyName = "Microsoft.Default"
      type           = "UserManaged"
      contentFilters = [
        { name = "hate", blocking = false, enabled = true, allowedContentLevel = "High", source = "Prompt" },
        { name = "sexual", blocking = false, enabled = true, allowedContentLevel = "High", source = "Prompt" },
        { name = "selfharm", blocking = false, enabled = true, allowedContentLevel = "High", source = "Prompt" },
        { name = "violence", blocking = false, enabled = true, allowedContentLevel = "High", source = "Prompt" },
        { name = "hate", blocking = false, enabled = true, allowedContentLevel = "High", source = "Completion" },
        { name = "sexual", blocking = false, enabled = true, allowedContentLevel = "High", source = "Completion" },
        { name = "selfharm", blocking = false, enabled = true, allowedContentLevel = "High", source = "Completion" },
        { name = "violence", blocking = false, enabled = true, allowedContentLevel = "High", source = "Completion" },
      ]
    }
  })
}

tombuildsstuff commented 1 year ago

hey @dkmiller

Thanks for opening this issue.

Taking a quick look into this one we're currently using API Version 2023-05-01 for CognitiveServices - however this functionality is only available in API Version 2023-06-01-preview (per above, and it not being present within 2023-05-01).

The SDK we use (hashicorp/go-azure-sdk) is generated from the Azure API Definitions found within the Azure/azure-rest-api-specs repository - however unfortunately the Preview API version isn't available at this time, and so unfortunately we're unable to take advantage of this API version at this point-in-time.

As such I've gone ahead and labelled this issue to indicate that it's blocked on the API Definitions becoming available - but once those are available then we can take a look into supporting this natively within the AzureRM Provider.

Thanks!

dkmiller commented 1 year ago

Thanks @tombuildsstuff! I'm kicking off a conversation with our contacts in Azure, will circle back as soon as there is an update.

dsoltz1 commented 10 months ago

Hi @tombuildsstuff and @dkmiller ! I wondered if there is now a possibility to create the content filter with Azurerm provider or if the only valid way is to use Azapi. Thanks!

illgitthat commented 8 months ago

Hi @tombuildsstuff and @dkmiller !

I wondered if there is now a possibility to create the content filter with Azurerm provider or if the only valid way is to use Azapi. Thanks!

It looks like 2023-05-01 is still being used. @liuwuliuyun Are you the right person to ask if there is any update on this enhancement?

dkmiller commented 8 months ago

@dsoltz1 — sadly, to the best of my knowledge Azapi is still the only valid way. The sample snippet I shared above should work.

We have an open ask out to Azure to publish API schemas including content filters, after which Terraform can pick those up. cc @cenhao, who is also interested in this from Airbnb.

liuwuliuyun commented 8 months ago

@illgitthat, thanks for reaching out, I will comment in this thread once there is update on this issue. In the meanwhile, I would suggest using AzAPI provider for any resource in preview API.

kamikaze commented 4 months ago

I'm getting following error:

│ --------------------------------------------------------------------------------
│ RESPONSE 400: 400 Bad Request
│ ERROR CODE: Resource has invalid content filter
│ --------------------------------------------------------------------------------
│ {
│   "error": {
│     "code": "Resource has invalid content filter",
│     "message": "Policy does not have necessary permission to override base policy. Please check aka.ms/oai/rai/exceptions"
│   }
│ }
│ --------------------------------------------------------------------------------
│ 
│ 
│   with module.azure_infra.azapi_resource.no_moderation_policy,
│   on modules/azure_infra/02-cognitive.tf line 9, in resource "azapi_resource" "no_moderation_policy":
│    9: resource "azapi_resource" "no_moderation_policy" {
│ 
╵

illgitthat commented 4 months ago

@kamikaze you can't disable content filters entirely without first getting approval. Reference: https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/content-filter?tabs=warning%2Cpython-new

LeonKalt commented 4 months ago

Would it be possible to also activate async streaming mode via terraform?

mymeatball commented 4 months ago

How to disable content flters and acticate async straming mode via Go SDK?

DanHyland commented 4 months ago

@liuwuliuyun any updates given the new 2024-02-01 API release? Having azurerm support will make things like dependency management easier.

beliven-daniele-sarnari commented 3 months ago

Support 👍

liuwuliuyun commented 2 months ago

Hi everyone, I would like to answer some questions in this thread by Q&A format.

Q: How could we create content filter in AOAI now? A: I just checked that as of now, content filter in AOAI is still only able to create by preview versions of API. So I would still recommend that using AzAPI provider to create this resource.

Q: Where do we know which API version I should use and is there a stable version 2024-02-01 released? A: Here is the swagger repo that shows API versions of Cognitive resource provider. You could see that latest stable API version 2023-05-01, creation of this resource is still not supported. There is no version 2024-02-01, we only support cognitive management plane APIs here, version 2024-02-01 could be a data plane API that used for inference. You could use 2023-06-01-preview or later preview versions to create this resource.

Q: Why not support preview versions of API here? A: There might be potential regression issues to have AzureRM support Azure preview features, because preview features do not guarantee its backward and forward feature compatibility. E.g. If AzureRM supports it now but this feature will have new versions in the future which however would include breaking changes to the current preview versions, it would be uneasy for AzureRM to catch up with that new version because it's on purpose to bring as few breaking changes as possible to Terraform AzureRM.

Hope this helps~

Mithun2507 commented 1 month ago

Would it be possible to also activate async streaming mode via terraform? you can achieve this using below parameter in the API request mode = "Deferred"

vjayarag commented 1 month ago

Thanks below code worked for me

resource "azapi_resource" "content_filter" {
  type      = "Microsoft.CognitiveServices/accounts/raiPolicies@2023-10-01-preview"
  name      = var.content_filter_name
  parent_id = azurerm_cognitive_account.example.id

  schema_validation_enabled = false

  body = jsonencode({
    properties = {
      mode           = "Default",
      basePolicyName = "Microsoft.Default",
      contentFilters = [
        { name = "hate", blocking = true, enabled = true, allowedContentLevel = "Low", source = "Prompt" },
        { name = "sexual", blocking = true, enabled = true, allowedContentLevel = "Low", source = "Prompt" },
        { name = "selfharm", blocking = true, enabled = true, allowedContentLevel = "Low", source = "Prompt" },
        { name = "violence", blocking = true, enabled = true, allowedContentLevel = "Low", source = "Prompt" },
        { name = "hate", blocking = true, enabled = true, allowedContentLevel = "Low", source = "Completion" },
        { name = "sexual", blocking = true, enabled = true, allowedContentLevel = "Low", source = "Completion" },
        { name = "selfharm", blocking = true, enabled = true, allowedContentLevel = "Low", source = "Completion" },
        { name = "violence", blocking = true, enabled = true, allowedContentLevel = "Low", source = "Completion" },
        { name = "jailbreak", blocking = true, enabled = true, source = "Prompt" },
        { name = "indirect_attack", blocking = true, enabled = true, source = "Prompt" },
        { name = "protected_material_text", blocking = true, enabled = true, source = "Completion" },
        { name = "protected_material_code", blocking = true, enabled = true, source = "Completion" }
      ]
    }
  })
  depends_on = [
    azurerm_cognitive_account.example
  ]
}

And passing the value from var.content_filter_name into var.rai_policy_name as below deployment config


resource "azurerm_cognitive_deployment" "example" {
  name                 = var.name
  cognitive_account_id = var.openai_account_id
  model {
    format  = "OpenAI"
    name    = var.model_name
    version = var.model_version
  }
  scale {
    type = var.scale
    capacity = var.model_capacity
  }
  rai_policy_name = var.rai_policy_name
}```

DanHyland commented 1 month ago

@vjayarag the issue is not the ability to manage this via azapi its that we want azurerm support for multiple reasons including:

Proper resource dependency management via TF graph
Remove the need to manage API versions

The latter issue is being seen today within my environment when using the 2023-06-01-preview API, given that preview APIs are mutable.....

The naming convention for the categories (or names) seems to have changed and now are upper case for Hate, Sexual and Violence, where selfharm is still lower case. This type of churn causes a perpetual diff within our environment without us mutating anything on our end. This is extremely sub optimal.

dkmiller commented 1 month ago

Hey @tombuildsstuff, I finally heard back from Microsoft! They've got an updated API version now GA at 2024-06-01.

It should have all latest features including latest content filter annotations.

Hopefully that should unblock implementing this feature request?

hashicorp / terraform-provider-azurerm