hashicorp / terraform-provider-azurerm

Terraform provider for Azure Resource Manager
https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs
Mozilla Public License 2.0
4.59k stars 4.62k forks source link

MySql Flexible server CMK bug? polling after Create: Code="AzureKeyVaultKeyNotFound" #20846

Open igor-marinho opened 1 year ago

igor-marinho commented 1 year ago

Is there an existing issue for this?

Community Note

Terraform Version

1.3.9

AzureRM Provider Version

3.46.0

Affected Resource(s)/Data Source(s)

azurerm_mysql_flexible_server

Terraform Configuration Files

resource "azurerm_mysql_flexible_server" "main" {
  provider                     = azurerm.current_sub
  name                         = var.resource_name
  resource_group_name          = var.rg_name
  location                     = var.resource_location
  version                      = "5.7"
  delegated_subnet_id          = azurerm_subnet.delegation.id
  private_dns_zone_id          = local.private_dns_zone_id
  geo_redundant_backup_enabled = false
  backup_retention_days        = 30
  administrator_login          = "dbaadmin"
  administrator_password       = "Test##00!"
  sku_name                     = lookup(var.sku_name, terraform.workspace)
  tags = {
    created-by = "Terraform"
  }

  identity {
    type         = "UserAssigned"
    identity_ids = [var.user_assigned_identity_id]
  }

  customer_managed_key {
    key_vault_key_id                  = var.keyvault_vault_key_id
    primary_user_assigned_identity_id = var.user_assigned_identity_id
  }

  storage {
    auto_grow_enabled = false
    size_gb = lookup(var.size_gb, terraform.workspace)
  }
  lifecycle {
    ignore_changes = [tags, zone]
  }
}

Debug Output/Panic Output

│ Error: creating Flexible Server (Subscription: "*****"
│ Resource Group Name: "mysqlflexiac-rg"
│ Flexible Server Name: "mysqlflexiac"): polling after Create: Code="AzureKeyVaultKeyNotFound" Message="Could not find Azure Key Vault Key with key name 'https://my-keyvault.vault.azure.net/keys/mysqlflexiac-cmk/76999fa585955093899717080790b2a6'."
│

Expected Behaviour

Object cerated.

Actual Behaviour

It fails after creation because it cannot find the keyvault with the key.

Steps to Reproduce

resource "azurerm_mysql_flexible_server" "main" { ..... identity { type = "UserAssigned" identity_ids = [var.user_assigned_identity_id] }

customer_managed_key { key_vault_key_id = var.keyvault_vault_key_id primary_user_assigned_identity_id = var.user_assigned_identity_id } ..... }

Important Factoids

No response

References

No response

neil-yechenwei commented 1 year ago

Thanks for raising this issue. Seems I cannot repro this issue with below tf config that is similar with yours. Could you try below tf config to see if the issue still exists? If no, could you compare below tf config with yours to figure out the difference? Thanks.

provider "azurerm" {
  features {
    key_vault {
      purge_soft_delete_on_destroy       = false
      purge_soft_deleted_keys_on_destroy = false
    }
  }
}

data "azurerm_client_config" "current" {}

resource "azurerm_resource_group" "test" {
  name     = "acctestRG-mysql-test0136"
  location = "west europe"
}

resource "azurerm_virtual_network" "test" {
  name                = "acctest-vn-test0136"
  location            = azurerm_resource_group.test.location
  resource_group_name = azurerm_resource_group.test.name
  address_space       = ["10.0.0.0/16"]
}

resource "azurerm_subnet" "test" {
  name                 = "acctest-sn-test0136"
  resource_group_name  = azurerm_resource_group.test.name
  virtual_network_name = azurerm_virtual_network.test.name
  address_prefixes     = ["10.0.2.0/24"]
  service_endpoints    = ["Microsoft.Storage"]

  delegation {
    name = "fs"

    service_delegation {
      name = "Microsoft.DBforMySQL/flexibleServers"

      actions = [
        "Microsoft.Network/virtualNetworks/subnets/join/action",
      ]
    }
  }
}

resource "azurerm_private_dns_zone" "test" {
  name                = "acctest0136.mysql.database.azure.com"
  resource_group_name = azurerm_resource_group.test.name
}

resource "azurerm_private_dns_zone_virtual_network_link" "test" {
  name                  = "acctestVnetZonetest0136.com"
  private_dns_zone_name = azurerm_private_dns_zone.test.name
  virtual_network_id    = azurerm_virtual_network.test.id
  resource_group_name   = azurerm_resource_group.test.name
}

resource "azurerm_user_assigned_identity" "test" {
  name                = "acctestmitest0136"
  location            = azurerm_resource_group.test.location
  resource_group_name = azurerm_resource_group.test.name
}

resource "azurerm_key_vault" "test" {
  name                     = "acctestkvtest0136"
  location                 = azurerm_resource_group.test.location
  resource_group_name      = azurerm_resource_group.test.name
  tenant_id                = data.azurerm_client_config.current.tenant_id
  sku_name                 = "standard"
  purge_protection_enabled = true
}

resource "azurerm_key_vault_access_policy" "server" {
  key_vault_id = azurerm_key_vault.test.id
  tenant_id    = data.azurerm_client_config.current.tenant_id
  object_id    = azurerm_user_assigned_identity.test.principal_id

  key_permissions = ["Get", "List", "WrapKey", "UnwrapKey", "GetRotationPolicy", "SetRotationPolicy"]
}

resource "azurerm_key_vault_access_policy" "client" {
  key_vault_id = azurerm_key_vault.test.id
  tenant_id    = data.azurerm_client_config.current.tenant_id
  object_id    = data.azurerm_client_config.current.object_id

  key_permissions = ["Get", "Create", "Delete", "List", "Restore", "Recover", "UnwrapKey", "WrapKey", "Purge", "Encrypt", "Decrypt", "Sign", "Verify", "GetRotationPolicy", "SetRotationPolicy"]
}

resource "azurerm_key_vault_key" "test" {
  name         = "test"
  key_vault_id = azurerm_key_vault.test.id
  key_type     = "RSA"
  key_size     = 2048
  key_opts     = ["decrypt", "encrypt", "sign", "unwrapKey", "verify", "wrapKey"]

  depends_on = [
    azurerm_key_vault_access_policy.client,
    azurerm_key_vault_access_policy.server,
  ]
}

resource "azurerm_mysql_flexible_server" "test" {
  name                         = "acctest-fs-test0136"
  resource_group_name          = azurerm_resource_group.test.name
  location                     = azurerm_resource_group.test.location
  version                      = "5.7"
  delegated_subnet_id          = azurerm_subnet.test.id
  private_dns_zone_id          = azurerm_private_dns_zone.test.id
  geo_redundant_backup_enabled = false
  backup_retention_days        = 30
  administrator_login          = "dbaadmin"
  administrator_password       = "Test##00!"
  sku_name                     = "GP_Standard_D2ds_v4"

  identity {
    type         = "UserAssigned"
    identity_ids = [azurerm_user_assigned_identity.test.id]
  }

  customer_managed_key {
    key_vault_key_id                  = azurerm_key_vault_key.test.id
    primary_user_assigned_identity_id = azurerm_user_assigned_identity.test.id
  }

  storage {
    auto_grow_enabled = false
    size_gb           = 20
  }

  lifecycle {
    ignore_changes = [zone]
  }

  depends_on = [azurerm_private_dns_zone_virtual_network_link.test]
}
M4rkor commented 1 year ago

@neil-yechenwei Currently facing the same issue with a similar setup you brought up:

resource "azurerm_resource_group" "test" {
  name     = "rg-app-dev-euwest"
  location = "west europe"
  tags     = merge(var.tags, { "app" : "test" })
}

resource "azurerm_mysql_flexible_server" "test" {
  name                = lower("mysql-app-dev-euwest") # require
  location            = "west europe"
  resource_group_name = azurerm_resource_group.test.name 

  administrator_login          = "app_admin"
  administrator_password = random_password.test-password.result

  sku_name   = "GP_Standard_D2ds_v4"
  version    = 5.7

  storage {
    auto_grow_enabled = false
    size_gb = 20
  } 

  identity {
    type = "UserAssigned"
    identity_ids = [azurerm_user_assigned_identity.mysql.id]
  }

  customer_managed_key {
    key_vault_key_id = data.azurerm_key_vault_key.keyvault-key.id
    primary_user_assigned_identity_id = azurerm_user_assigned_identity.mysql.id
  }
}

resource "azurerm_key_vault_access_policy" "test" {
  key_vault_id  = data.azurerm_key_vault.keyvault.id
  tenant_id     = data.azurerm_client_config.current.tenant_id
  object_id     = azurerm_user_assigned_identity.mysql.principal_id
  key_permissions = ["Get", "UnwrapKey", "WrapKey"]
}

resource "azurerm_user_assigned_identity" "test" {
  location  = "west europe"
  name      = "id-app-dev-euwest"
  resource_group_name = azurerm_resource_group.test.name 
}

The KeyVault in that case is set in another resource Group. Maybe with that you are able to reproduce it.

Edit:

Tried it with separate KeyVault in the same resource group and worked just fine. Setup with KeyVault in different Resource Group works for flexible PostgreSQL, so it probably should for MySQL as well?

igor-marinho commented 1 year ago

@neil-yechenwei Currently facing the same issue with a similar setup you brought up:

resource "azurerm_resource_group" "test" {
  name     = "rg-app-dev-euwest"
  location = "west europe"
  tags     = merge(var.tags, { "app" : "test" })
}

resource "azurerm_mysql_flexible_server" "test" {
  name                = lower("mysql-app-dev-euwest") # require
  location            = "west europe"
  resource_group_name = azurerm_resource_group.test.name 

  administrator_login          = "app_admin"
  administrator_password = random_password.test-password.result

  sku_name   = "GP_Standard_D2ds_v4"
  version    = 5.7

  storage {
    auto_grow_enabled = false
    size_gb = 20
  } 

  identity {
    type = "UserAssigned"
    identity_ids = [azurerm_user_assigned_identity.mysql.id]
  }

  customer_managed_key {
    key_vault_key_id = data.azurerm_key_vault_key.keyvault-key.id
    primary_user_assigned_identity_id = azurerm_user_assigned_identity.mysql.id
  }
}

resource "azurerm_key_vault_access_policy" "test" {
  key_vault_id  = data.azurerm_key_vault.keyvault.id
  tenant_id     = data.azurerm_client_config.current.tenant_id
  object_id     = azurerm_user_assigned_identity.mysql.principal_id
  key_permissions = ["Get", "UnwrapKey", "WrapKey"]
}

resource "azurerm_user_assigned_identity" "test" {
  location  = "west europe"
  name      = "id-app-dev-euwest"
  resource_group_name = azurerm_resource_group.test.name 
}

The KeyVault in that case is set in another resource Group. Maybe with that you are able to reproduce it.

Edit:

Tried it with separate KeyVault in the same resource group and worked just fine. Setup with KeyVault in different Resource Group works for flexible PostgreSQL, so it probably should for MySQL as well?

The weird part for is even adding mysql-flexible in the same resource group as my keyvault it still saying it cannot find it and theyurl/ket/version are all good.

igor-marinho commented 1 year ago

Closed it by mistake ....

igor-marinho commented 1 year ago

I'm tried creating my own keyvault and when creating the mysql flexible and I got same result.

resource "azurerm_subnet" "main" {
  provider             = azurerm.current_sub
  name                 = "mysql-flexi-subnet"
  resource_group_name  = data.azurerm_virtual_network.main.resource_group_name
  virtual_network_name = data.azurerm_virtual_network.main.name
  address_prefixes     = ["${cidrsubnet(data.azurerm_virtual_network.main.address_space[0], 4, 15)}"]
  service_endpoints    = ["Microsoft.Storage"]

  delegation {
    name = "fs"

    service_delegation {
      name = "Microsoft.DBforMySQL/flexibleServers"

      actions = [
        "Microsoft.Network/virtualNetworks/subnets/join/action",
      ]
    }
  }

  lifecycle {
    ignore_changes = [address_prefixes]
  }
}

resource "azurerm_user_assigned_identity" "main" {
  name                = "test-iden"
  location            = var.resource_location
  resource_group_name = var.rg_name
}

resource "azurerm_key_vault" "main" {
  name                     = "keyvaulttes1"
  location                 = var.resource_location
  resource_group_name      = var.rg_name
  tenant_id                = data.azurerm_client_config.current.tenant_id
  sku_name                 = "standard"
  purge_protection_enabled = true
}

resource "azurerm_key_vault_access_policy" "server" {
  key_vault_id = azurerm_key_vault.main.id
  tenant_id    = data.azurerm_client_config.current.tenant_id
  object_id    = azurerm_user_assigned_identity.main.principal_id

  key_permissions = ["Get", "List", "WrapKey", "UnwrapKey", "GetRotationPolicy", "SetRotationPolicy"]
}

resource "azurerm_key_vault_access_policy" "client" {
  key_vault_id = azurerm_key_vault.main.id
  tenant_id    = data.azurerm_client_config.current.tenant_id
  object_id    = data.azurerm_client_config.current.object_id

  key_permissions = ["Get", "Create", "Delete", "List", "Restore", "Recover", "UnwrapKey", "WrapKey", "Purge", "Encrypt", "Decrypt", "Sign", "Verify", "GetRotationPolicy", "SetRotationPolicy"]
}

resource "azurerm_key_vault_key" "main" {
  name         = "main"
  key_vault_id = azurerm_key_vault.main.id
  key_type     = "RSA"
  key_size     = 2048
  key_opts     = ["decrypt", "encrypt", "sign", "unwrapKey", "verify", "wrapKey"]

  depends_on = [
    azurerm_key_vault_access_policy.client,
    azurerm_key_vault_access_policy.server,
  ]
}

resource "azurerm_mysql_flexible_server" "main" {
  name                         = "accmain-fs-main0136"
  resource_group_name          = var.rg_name
  location                     = var.resource_location
  version                      = "5.7"
  delegated_subnet_id          = azurerm_subnet.main.id
  private_dns_zone_id          = local.private_dns_zone_id
  geo_redundant_backup_enabled = false
  backup_retention_days        = 30
  administrator_login          = "dbaadmin"
  administrator_password       = "main##00!"
  sku_name                     = "GP_Standard_D2ds_v4"

  identity {
    type         = "UserAssigned"
    identity_ids = [azurerm_user_assigned_identity.main.id]
  }

  customer_managed_key {
    key_vault_key_id                  = azurerm_key_vault_key.main.id
    primary_user_assigned_identity_id = azurerm_user_assigned_identity.main.id
  }

  storage {
    auto_grow_enabled = false
    size_gb           = 20
  }

  lifecycle {
    ignore_changes = [zone]
  }
}

│ Error: creating Flexible Server (Subscription: "***" │ Resource Group Name: "na-mysql-stable-rg" │ Flexible Server Name: "accmain-fs-main0136"): polling after Create: Code="AzureKeyVaultKeyNotFound" Message="Could not find Azure Key Vault Key with key name 'https://keyvaulttes1.vault.azure.net/keys/...."

igor-marinho commented 1 year ago

Doing some tests I've create it manually without the CMK and after I tried to enable CMK as below:

 # module.root-mysql-flexi["poc"].module.mysqlflexi.azurerm_mysql_flexible_server.main will be updated in-place
  ~ resource "azurerm_mysql_flexible_server" "main" {
        id                            = "/subscriptions/*************/resourceGroups/mysqlflexiac-rg/providers/Microsoft.DBforMySQL/flexibleServers/mysqlflexiac"
        name                          = "mysqlflexiac"

        # (15 unchanged attributes hidden)

      + customer_managed_key {
          + key_vault_key_id                  = "https://na-mysql-stable********t/keys/mysqlflexiac-cmk/********8df2b4a450a022"
          + primary_user_assigned_identity_id = "/subscriptions/************/resourceGroups/mysqlflexiac-rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/mysqlflexiac-identity-server"
        }

        # (2 unchanged blocks hidden)
    }

Same error:

│ Error: updating Flexible Server (Subscription: "*********"
│ Resource Group Name: "mysqlflexiac-rg"
│ Flexible Server Name: "mysqlflexiac"): polling after Update: Code="AzureKeyVaultKeyNotFound" Message="Could not find Azure Key Vault Key with key name 'https://na-mysql-stable************/keys/mysqlflexiac-cmk/*********829b18df2b4a450a022'."    

When I do it manually it works just fine.

Screenshot 2023-03-13 at 10 01 54 AM

So, fit seems to be a bug on the azurerm provider or in Mysql Flexible server itself.

bwilczynski commented 1 year ago

@igor-marinho - I believe it's not a problem in a provider but rather the limitation of a CMK for MySQL as stated here: https://learn.microsoft.com/en-us/azure/mysql/flexible-server/concepts-customer-managed-key#limitations

This feature is only supported for key vaults, which allow public access from all networks.

I have the same issue trying to set encryption for MySQL in a portal. Despite setting the values I get the same error when I try to save the configuration. Can you check whether your AKV is set to "private networks access"?

igor-marinho commented 1 year ago

@bwilczynski Yes, I use vnet integration for mysql flexible and private connection for all keyvaults. I thought it'd work asme as flexible server but is seems that it doesn't.

sigv commented 1 year ago

Support for MySQL Flexible Server CMK with VNet integration is currently being evaluated by Microsoft in Private Preview. I would advise you to wait for it to become GA.