Cognitive services broken by Azure API change

phil-bevan commented 3 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Affected Resource(s)

azurerm_cognitive_account (Bing.SpellCheck.v7, Bing.Search.v7, Bing.CustomSearch)

Terraform Configuration Files

resource "azurerm_cognitive_account" "spellcheck" {
  name                = "BingSpellCheck-${var.naming_suffix_global}"
  location            = "global"
  resource_group_name = var.resourcegroup
  kind                = "Bing.SpellCheck.v7"
  sku_name = "S1"
}
resource "azurerm_cognitive_account" "bing-search" {
  name                = "BingSearch-${var.naming_suffix_global}"
  location            = "global"
  resource_group_name = var.resourcegroup
  kind                = "Bing.Search.v7"
  sku_name = "S1"
}
resource "azurerm_cognitive_account" "bing-custom" {
  name                = "BingCustomSearch-${var.naming_suffix_global}"
  location            = "global"
  resource_group_name = var.resourcegroup
  kind                = "Bing.CustomSearch"
  sku_name = "S1"
}

Debug Output

Error: creating Cognitive Services Account "BingSpellCheck-flextfone-prod-global" (Resource Group "rg-flextfone-prod-uks"): cognitiveservices.AccountsClient#Create: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="ApiSetDisabledForCreation" Message="It's not allowed to create new accounts with type 'Bing.SpellCheck.v7'."

on Modules\CogSvcs\cogsvcs.tf line 25, in resource "azurerm_cognitive_account" "spellcheck": 25: resource "azurerm_cognitive_account" "spellcheck" {

Error: creating Cognitive Services Account "BingSearch-flextfone-prod-global" (Resource Group "rg-flextfone-prod-uks"): cognitiveservices.AccountsClient#Create: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="ApiSetDisabledForCreation" Message="It's not allowed to create new accounts with type 'Bing.Search.v7'."

on Modules\CogSvcs\cogsvcs.tf line 46, in resource "azurerm_cognitive_account" "bing-search": 46: resource "azurerm_cognitive_account" "bing-search" {

Error: creating Cognitive Services Account "BingCustomSearch-flextfone-prod-global" (Resource Group "rg-flextfone-prod-uks"): cognitiveservices.AccountsClient#Create: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="ApiSetDisabledForCreation" Message="It's not allowed to create new accounts with type 'Bing.CustomSearch'."

on Modules\CogSvcs\cogsvcs.tf line 53, in resource "azurerm_cognitive_account" "bing-custom": 53: resource "azurerm_cognitive_account" "bing-custom" {

Panic Output

Expected Behavior

Bing Search Cognitive Services should deploy correctly

Actual Behavior

As of today, they do not. Microsoft has moved Bing Search services out of Cognitive Services and into Azure Marketplace, and appears to have disabled the original APIs which makes it impossible to currently deploy these services via Terraform. This is obviously a huge issue for those of use that require these services!

Steps to Reproduce

terraform apply

Important Factoids

References

0000

tombuildsstuff commented 3 years ago

Further details: https://azure.microsoft.com/en-us/updates/bing-search-apis-will-transition-from-azure-cognitive-services-to-azure-marketplace-on-31-october-2023/

tombuildsstuff commented 3 years ago

Opened an upstream issue for this, since this isn't how API deprecations are supposed to work in ARM: https://github.com/Azure/azure-rest-api-specs/issues/11460

Notably the product page for this is still available - https://azure.microsoft.com/en-us/services/cognitive-services/bing-web-search-api/ - so this appears to have been deprecated rather quickly / without any notice?

jaggerbodas-ms commented 3 years ago

Hi @tombuildsstuff ,
As you noted, Bing Search APIs are transitioning out of Cognitive Services platform. All the existing instances of Bing Search APIs provisioned via Cognitive services platform will be supported till the end of deprecation period i.e. till 31st Oct 2023 (or till end of specific enterprise agreement), however there will not be an option to create new Bing APIs resources via existing Cognitive services platform related methodologies.

And relevant email communication was sent to respective resource owners. However updates to existing Azure product pages were delayed due to few technical issues.

bhanuchintha commented 3 years ago

You might have to use the below new format for creating the Bing resource -

{ "$schema": "http://schema.management.azure.com/schemas/2014-04-01-preview/deploymentTemplate.json#", "contentVersion": "1.0.0.0", "parameters": { "name": { "type": "string" }, "sku": { "type": "string" } }, "resources": [ { "apiVersion": "2020-06-10", "name": "[parameters('name')]", "location": "global", "type": "Microsoft.Bing/accounts", "kind": "Bing.Search.v7", "sku": { "name": "[parameters('sku')]" }, "properties": { "statisticsEnabled": false } } ] }

mehfuzh commented 3 years ago

Similar issue:


resource "azurerm_cognitive_account" "spellcheck" {
  name                = "bing-spellcheck-${azurerm_resource_group.rg.name}"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  kind                = "Bing.SpellCheck.v7"

  sku_name = "S1"

  tags = {
    Owner = "sl"
  }
}

The documentation needs an update as well since this is still broken: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/cognitive_account

gpnikhil commented 12 months ago

Hi, Is there no solution for this yet?

samirbanjanovic commented 9 months ago

@bhanuchintha is on the right track. If you follow the documentation on how to create the new Bing Search resources through market place you'll end up with the following arm templates for CustomSearch and Search.v7, respectively.

CustomSearch

{
    "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "name": {
            "type": "string"
        },
        "location": {
            "type": "string"
        },
        "sku": {
            "type": "string"
        },
        "tagValues": {
            "type": "object"
        }
    },
    "resources": [
        {
            "apiVersion": "2020-06-10",
            "name": "[parameters('name')]",
            "location": "[parameters('location')]",
            "type": "Microsoft.Bing/accounts",
            "kind": "Bing.CustomSearch",
            "tags": "[if(contains(parameters('tagValues'), 'Microsoft.Bing/accounts'), parameters('tagValues')['Microsoft.Bing/accounts'], json('{}'))]",
            "sku": {
                "name": "[parameters('sku')]"
            }
        }
    ]
}

Search.v7

{
    "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "name": {
            "type": "string"
        },
        "location": {
            "type": "string"
        },
        "sku": {
            "type": "string"
        },
        "tagValues": {
            "type": "object"
        }
    },
    "resources": [
        {
            "apiVersion": "2020-06-10",
            "name": "[parameters('name')]",
            "location": "[parameters('location')]",
            "type": "Microsoft.Bing/accounts",
            "kind": "Bing.Search.v7",
            "tags": "[if(contains(parameters('tagValues'), 'Microsoft.Bing/accounts'), parameters('tagValues')['Microsoft.Bing/accounts'], json('{}'))]",
            "sku": {
                "name": "[parameters('sku')]"
            }
        }
    ]
}

As it currently stands azurerm doesn't have support for this specific azure resource type. If you search you'll also notice documentation is lacking. However, using the ARM above and network observations on Azure Portal you can leverage the terraform provider azure/azapi to accomplish your goal. I was ultimately able to achieve this using the two resource blocks defined below -- azapi_resource and azapi_resource_action. It's a two step process because of the data that is needed. The resource creation doesn't respond with the bing api key you may need to pass to aps for interaction, thus the action to list keys.

You must set `schema_validation_enabled = false` otherwise azapi will throw a schema validation error due to the absence of the type definition details for `Microsoft.Bing/accounts`

Also, make sure the service principal/account you're using to execute this terraform is on the tenant where you're creating the Bing search resource. Here's what worked for me:

resource "azapi_resource" "bingSearchAccount" {
  type                      = "Microsoft.Bing/accounts@2020-06-10"
  schema_validation_enabled = false
  name                      = var.bingSearchAPIName
  parent_id                 = var.resourceGroupId
  location                  = "global"
  body = jsonencode({
    sku = {
      name = "S1"
    }
    kind = "Bing.Search.v7" # or "Bing.CustomSearch"
  })
  response_export_values = ["*"]
}

# get the bing search api access keys
data "azapi_resource_action" "bingSearchAccount" {
  type        = "Microsoft.Bing/accounts@2020-06-10"
  resource_id = azapi_resource.bingSearchAccount.id
  action      = "listKeys"
  method      = "POST"
  response_export_values = ["*"]
}

rcskosir commented 3 months ago

Thank you for taking the time to open this issue. Since this is an upstream Microsoft change to the Azure API that broke cognitive services and we are now past the October 31st, 2023 deprecation period I am going to close this issue. For further feedback please follow the linked upstream issue: https://github.com/Azure/azure-rest-api-specs/issues/11460

jongio commented 3 months ago

This is how I got the values:

output "BING_SEARCH_ENDPOINT" {
  value = jsondecode(azapi_resource.bing.output).properties.endpoint
}

output "BING_SEARCH_KEY" {
  value     = jsondecode(data.azapi_resource_action.bing.output).key1
  sensitive = true
}

and add this to provider list:

terraform {
  backend "azurerm" {}
  required_version = ">= 1.1.7, < 2.0.0"
  required_providers {

    azapi = {
      source  = "Azure/azapi"
    }

}
}

github-actions[bot] commented 2 months ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

rcskosir commented 1 month ago

I am reopening this issue as a feature request for Bing Search, it is still waiting on the swagger upstream: https://github.com/Azure/azure-rest-api-specs/issues/11460

hashicorp / terraform-provider-azurerm