search

hashicorp / terraform-provider-azurerm

Terraform provider for Azure Resource Manager
https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs
Mozilla Public License 2.0
4.6k stars 4.64k forks source link

Synapse Managed Keys Cannot Be Found With Managed VNET #15976

Open TehFamine opened 2 years ago

TehFamine commented 2 years ago

Community Note

Terraform (and AzureRM Provider) Version

Terraform = 1.1.5 hashicorp/azurerm = 2.94.0 azure-cli = 2.34.1

Affected Resource(s)

Terraform Configuration Files


# Synapse Workspace

resource "azurerm_synapse_workspace" "ws_analytics" {
  provider                              = azurerm.sponsor
  name                                  = "syn-${local.service_no_network}-01"
  resource_group_name                   = var.resource_group_name
  location                              = var.location
  storage_data_lake_gen2_filesystem_id  = "https://<removed>.dfs.core.windows.net/${var.environment}-${var.client_name}"
  sql_administrator_login               = "myadmin"
  sql_administrator_login_password      = azurerm_key_vault_secret.key_vault_secret_admin_password.value
  compute_subnet_id                     = data.azurerm_subnet.SubnetSponsor.id
  public_network_access_enabled         = false
  managed_virtual_network_enabled       = true
  data_exfiltration_protection_enabled  = true
  sql_identity_control_enabled          = true

  customer_managed_key {
    key_versionless_id = azurerm_key_vault_key.encryption_cmk.versionless_id
    key_name           = azurerm_key_vault_key.encryption_cmk.name
  }

  tags = {
    environment = var.environment
    branch = var.branch_name
    service = var.sponsor_name
  }
}

# Private DNS Zone

data "azurerm_private_dns_zone" "synapse_dev" {
  name                = "privatelink.dev.azuresynapse.net"
  resource_group_name = var.resource_group_NET_HUB
}

# Private Endpoint for Synapse Dev

resource "azurerm_private_endpoint" "pe_analytics" {
  provider            = azurerm.sponsor
  name                = "pe-${local.service_network}-02"
  location            = var.location
  resource_group_name = var.resource_group_name
  subnet_id           = data.azurerm_subnet.SubnetSponsor.id

  private_service_connection {
    name                           = "psc-${local.service_network}-02"
    private_connection_resource_id = azurerm_synapse_workspace.ws_analytics.id
    is_manual_connection           = false
    subresource_names              = ["dev"]
  }
  private_dns_zone_group {
    name = "${var.environment}-${var.sponsor_name}"
    private_dns_zone_ids = [data.azurerm_private_dns_zone.synapse_dev.id]
  }
  tags = {
    environment = var.environment
    branch = var.branch_name
    service = var.sponsor_name
  }
  lifecycle {
    ignore_changes = [
      tags
    ]
  }
  depends_on = [azurerm_synapse_workspace.ws_analytics]
}

# Key Vault
# Purge Protection is On with Soft Delete

data "azurerm_key_vault" "key_vault" {
  provider            = azurerm.sponsor
  name                = "kv-vault-${var.environment}-01"
  resource_group_name = var.resource_group_name
}

# Access Policy

resource "azurerm_key_vault_access_policy" "analytics_workspace_policy" {
  provider        = azurerm.sponsor
  key_vault_id    = data.azurerm_key_vault.key_vault.id
  tenant_id       = azurerm_synapse_workspace.ws_analytics.identity[0].tenant_id
  object_id       = azurerm_synapse_workspace.ws_analytics.identity[0].principal_id

  key_permissions = [
    "Get", "WrapKey", "UnwrapKey", "List"
  ]
}

# Encryption Key 

resource "azurerm_key_vault_key" "encryption_cmk" {
  provider     = azurerm.sponsor
  name         = "kv-${local.service_no_network}-cmk"
  key_vault_id = data.azurerm_key_vault.key_vault.id
  key_type     = "RSA"
  key_size     = 2048
  key_opts = [
    "decrypt",
    "encrypt",
    "sign",
    "unwrapKey",
    "verify",
    "wrapKey"
  ]
  expiration_date = "2023-11-01T00:00:00Z"
  lifecycle {
    ignore_changes = [ expiration_date]
  }

  depends_on = [
    azurerm_key_vault_access_policy.analytics_workspace_policy
  ]
}

Debug Output

None to provide. Don't have direct access to Terraform service. Running from CI/CD commits.

Panic Output

None to provide.

Expected Behaviour

When you enable managed_virtual_network_enabled to be True and turn off public access, then you need to provide a customer_managed_key with both a key_versionless_id and key_name. You can generate this using azurerm_key_vault_key and supplying the key to the azurerm_synapse_workspace. Then you can grant the Synapse resource to that key using the azurerm_key_vault_access_policy so it can activate the workspace. If all permissions are set, private endpoints are created for both the Key Vault, Synapse Dev, and so forth, then the Synapse resource SHOULD be able to access that key over the managed VNET.

Actual Behaviour

Customer key vault base url in 'https://<removed>.vault.azure.net/keys/kv-encryption-dev-cmk' is invalid. Valid key vault endpoint base urls: https://<vault-name>.vault.azure.net, https://<hsm-name>.managedhsm.azure.net or equivalent in national cloouds.

This is the Azure error when trying to activate the workspace manually. The URL is verified as correct. The private endpoints are deployed and targeting the right subnets with the right private DNS zones, and the Synapse resource (MI) is part of the access policy with the right permissions to wrap, unwrap, read, list, purge, and more. We verified the Key Vault Private Endpoints are working as expected via the Firewall/logs.

When using other resources like azurerm_synapse_workspace_key. Similar errors occur with Terraform not able to find the key with the valid key vault URL. We also cannot use azurerm_synapse_workspace_aad_admin because the workspace cannot be activated when the key vault, the key URL cannot be found.

It looks like Synapse is not able to see the URL because it's not able to access our Private VNET via the Private Endpoint.

Steps to Reproduce

Run above.

Important Factoids

We deployed same configuration within the Azure Portal fine using the same Private Endpoint configurations, DNS Zones, and similar VNet's that are peered to a Hub and Spoke model. We cannot deploy Synapse with Public Access On and MUST use the managed_virtual_network_enabled to be True due to policy.

References

None

TehFamine commented 2 years ago

When I removed the key_name from the block to force Terraform to use the default "cmk" name instead, I see the changes applied in the plan and deployment.

~ key_name = "kv-encryption-cmk" -> "cmk"

But the actual change is not reflective in the Synapse properties. Some really weird things going on with Synapse Workspace with CMK's and Managed VNET's.

geoffmca commented 2 years ago

This is actually a MSFT issue. Sometime between 03/18-03/20 any key name with -'s doesnt work for Synapse Workspace. I have a case open and am escalating it to have this behavir fixed. As a work around you can create a key without -'s for the time being, but if you use a specific format where you need -'s I'm not sure of their ETA to address (but you could open a ticket with MSFT).

TehFamine commented 2 years ago

This is actually a MSFT issue. Sometime between 03/18-03/20 any key name with -'s doesnt work for Synapse Workspace. I have a case open and am escalating it to have this behavir fixed. As a work around you can create a key without -'s for the time being, but if you use a specific format where you need -'s I'm not sure of their ETA to address (but you could open a ticket with MSFT).

Thanks. This did fix my issue. Just to make this more clear. You must remove all hyphens key vault key name, not the URL or the key vault name itself. I was able to activate the workspace after I did those changes.