When applying changes to azurerm_cognitive_account_customer_managed_key.key, provider │ "provider[\"registry.terraform.io/hashicorp/azurerm\"]" produced an unexpected new value: Root resource was present, but now absent.

aatifnaseem commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues

Community Note

I am creating a cognitive account with a private endpoint and trying to attach a CMK stored in key vault but I am consistently getting below error.

│ Error: Provider produced inconsistent result after apply │ │ When applying changes to azurerm_cognitive_account_customer_managed_key.key, provider │ "provider[\"registry.terraform.io/hashicorp/azurerm\"]" produced an unexpected new value: Root resource was present, but now
│ absent. │ │ This is a bug in the provider, which should be reported in the provider's own issue tracker.

Terraform Version

1.0.0

AzureRM Provider Version

3.73.0

Affected Resource(s)/Data Source(s)

azurerm_cognitive_account_customer_managed_key

Terraform Configuration Files

terraform {
  required_providers {
    azurerm = {
        source = "hashicorp/azurerm"
        version = "3.68.0"
    }
  }
}

provider "azurerm" {
  features {

  }
}
data "azurerm_client_config" "current" {

}
data "azurerm_resource_group" "rg" {
  name = "rg-practice-tst"
}

data "azurerm_virtual_network" "vnet" {
  name = "vnet-dev-abx000-01"
  resource_group_name = data.azurerm_resource_group.rg.name
}

data "azurerm_subnet" "snet" {
    for_each = toset(data.azurerm_virtual_network.vnet.subnets)
    name = each.key
    resource_group_name = data.azurerm_virtual_network.vnet.resource_group_name
    virtual_network_name = data.azurerm_virtual_network.vnet.name
}

data "azurerm_key_vault" "kv" {
  name = "kv-dev-abx000-001"
  resource_group_name = data.azurerm_resource_group.rg.name
}

data "azurerm_key_vault_key" "key" {
  key_vault_id = data.azurerm_key_vault.kv.id
  name = "key-dev-abx000-01"
}

resource "azurerm_cognitive_account" "account" {
  name = "fr-dev-abx0000-02s"
  location = data.azurerm_virtual_network.vnet.location
  resource_group_name = data.azurerm_resource_group.rg.name
  kind = "FormRecognizer"
  sku_name = "S0"
  custom_subdomain_name = "fr-dev-abx0000-01"
  public_network_access_enabled = false

  identity {
    type = "SystemAssigned"
  }
}

resource "azurerm_key_vault_access_policy" "policy" {
  key_vault_id = data.azurerm_key_vault.kv.id
  tenant_id = data.azurerm_client_config.current.tenant_id
  object_id = azurerm_cognitive_account.account.identity.0.principal_id

  key_permissions = [
    "Get",
    "WrapKey",
    "UnwrapKey"
  ]
  secret_permissions = [
    "Get"
  ]
}

resource "azurerm_cognitive_account_customer_managed_key" "key" {
    cognitive_account_id = azurerm_cognitive_account.account.id
    key_vault_key_id = data.azurerm_key_vault_key.key.id

    depends_on = [ null_resource.sleep ]
}

resource "null_resource" "sleep" {
  triggers = {
    CA_ID = azurerm_cognitive_account.account.id
  }

  provisioner "local-exec" {
    interpreter = [ "pwsh", "-c" ]
    command = "Start-Sleep -s 180"
  }
}
resource "azurerm_private_endpoint" "pvt" {

  name = "cognitive-pvt"
  resource_group_name = data.azurerm_resource_group.rg.name
  location = data.azurerm_virtual_network.vnet.location

  subnet_id = data.azurerm_subnet.snet["snet-pvt-endpoints"].id
  private_service_connection {
    name = "cognitive-pvt-conn"
    private_connection_resource_id = azurerm_cognitive_account.account.id
    is_manual_connection = false
    subresource_names = ["account"]
  }

 depends_on = [ null_resource.sleep ]
}

Debug Output/Panic Output

│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to azurerm_cognitive_account_customer_managed_key.key, provider
│ "provider[\"registry.terraform.io/hashicorp/azurerm\"]" produced an unexpected new value: Root resource was present, but now  
│ absent.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

Expected Behaviour

Customer managed key should have been attached to created cognitive account of type Form Recognizer.

Actual Behaviour

Getting below error every time consistently. Tried changing provider version to 3.71.0, 3.69.0, 3.68 but no luck.

│ Error: Provider produced inconsistent result after apply │ │ When applying changes to azurerm_cognitive_account_customer_managed_key.key, provider │ "provider[\"registry.terraform.io/hashicorp/azurerm\"]" produced an unexpected new value: Root resource was present, but now
│ absent. │ │ This is a bug in the provider, which should be reported in the provider's own issue tracker.

Steps to Reproduce

Create a cognitive account of type Form Recognizer with system assigned managed identity enabled on it.
Grant Access Policy permissions to identity
Attach customer managed key stored in key vault using terraform resource and pass key id and cognitive account id.

Important Factoids

NA

References

NA

liuwuliuyun commented 1 year ago

Hi @aatifnaseem , thanks for raising this issue. This seems to be a bug in the provider, I will try to reproduce it locally and update in this thread.

liuwuliuyun commented 1 year ago

Reproduced error with following config

provider "azurerm" {
  features {
    key_vault {
      purge_soft_delete_on_destroy       = false
      purge_soft_deleted_keys_on_destroy = false
    }
  }
}

data "azurerm_client_config" "current" {}

resource "azurerm_resource_group" "test" {
  name     = "yun-GHIssue-23353"
  location = "East US"
}

resource "azurerm_cognitive_account" "test" {
  name                  = "testacc"
  location              = azurerm_resource_group.test.location
  resource_group_name   = azurerm_resource_group.test.name
  kind = "FormRecognizer"
  sku_name = "S0"
  custom_subdomain_name = "fr-dev-abx0000-01"
  public_network_access_enabled = false

  identity {
    type = "SystemAssigned"
  }
}

resource "azurerm_key_vault" "test" {
  name                     = "acctestkvyunliugh"
  location                 = azurerm_resource_group.test.location
  resource_group_name      = azurerm_resource_group.test.name
  tenant_id                = data.azurerm_client_config.current.tenant_id
  sku_name                 = "standard"
  purge_protection_enabled = true

  access_policy {
    tenant_id = azurerm_cognitive_account.test.identity.0.tenant_id
    object_id = azurerm_cognitive_account.test.identity.0.principal_id
    key_permissions = [
      "Get", "Create", "List", "Restore", "Recover", "UnwrapKey", "WrapKey", "Purge", "Encrypt", "Decrypt", "Sign", "Verify"
    ]
    secret_permissions = [
      "Get",
    ]
  }

  access_policy {
    tenant_id = data.azurerm_client_config.current.tenant_id
    object_id = data.azurerm_client_config.current.object_id
    key_permissions = [
      "Get", "Create", "Delete", "List", "Restore", "Recover", "UnwrapKey", "WrapKey", "Purge", "Encrypt", "Decrypt", "Sign", "Verify", "GetRotationPolicy"
    ]
    secret_permissions = [
      "Get",
    ]
  }
}

resource "azurerm_key_vault_key" "test" {
  name         = "acctestkvkey"
  key_vault_id = azurerm_key_vault.test.id
  key_type     = "RSA"
  key_size     = 2048
  key_opts     = ["decrypt", "encrypt", "sign", "unwrapKey", "verify", "wrapKey"]
}

resource "azurerm_cognitive_account_customer_managed_key" "test" {
  cognitive_account_id = azurerm_cognitive_account.test.id
  key_vault_key_id     = azurerm_key_vault_key.test.id
}

aatifnaseem commented 1 year ago

@liuwuliuyun Thank you for quick response and reproducing it. I tried a workaround by passing the "identity_client_id" for system assigned managed identity through Azure AD data source for service principal and it worked. But ideally, it should work without passing anything to it in case of system assigned identity as it is mentioned as optional property in azurerm_cognitive_account_customer_managed_key.

liuwuliuyun commented 1 year ago

For any future future readers, using an user-assigned identity id is also a protential workaround. I will continue look into this.

liuwuliuyun commented 1 year ago

This issue is caused by an eventual consistancy issue of the cognitive REST API. After the creation of the resource succeeds, the GET operation of the resource does not return the updated keyvault settings. Therefore the code returns nil and triggers this issue.

https://github.com/hashicorp/terraform-provider-azurerm/blob/0b84d31c9c97ad8920a54d47e09274155c8353ec/internal/services/cognitive/cognitive_account_customer_managed_key_resource.go#L145-L148

hashicorp / terraform-provider-azurerm