keyvault: support for creating items over a nested endpoint

cb900rr2000 commented 3 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

0.14.0 / 2.39.0

Affected Resource(s)

azurerm_key_vault_key

Terraform Configuration Files


provider "azurerm" {
  version          = "=2.39.0"
  subscription_id  = "5ab29627-1b06-47b1-bf94-251a9e6c74c7"
  tenant_id        = "90ad1900-2019-4ffa-b845-96f012d0dc5a"
  skip_provider_registration = false
  features {}

  use_msi = true
}

provider "azurerm" {
 version           = "=2.39.0"
  alias            = "dnsprod"
  subscription_id  = "ed330c6f-5b25-4d9a-8db2-13a6b612c317"
  tenant_id        = "90ad1900-2019-4ffa-b845-96f012d0dc5a"
  skip_provider_registration = true
  features {}

  use_msi = true
}

data "azurerm_client_config" "current" {}

data "azurerm_virtual_machine" "vm" {

   name                 =  "mangementvm"
   resource_group_name  = "development"
}

data "azurerm_subnet" "rg" {
  name                 = "Storage"
  virtual_network_name = "Policytest"
  resource_group_name  = "development"

}

resource "azurerm_resource_group" "rg" {
  name     = "TFtest2"
  location = "uksouth"
}

resource "azurerm_key_vault" "main" {
  name                = "newkekvault215"
  resource_group_name = azurerm_resource_group.rg.name
  location            = azurerm_resource_group.rg.location
  tenant_id           = data.azurerm_client_config.current.tenant_id
  sku_name = "premium"
  purge_protection_enabled = true
  soft_delete_enabled      = true

     network_acls {
     default_action = "Deny"
     bypass         = "AzureServices"
}
access_policy {
  tenant_id    = data.azurerm_client_config.current.tenant_id
  object_id   = data.azurerm_virtual_machine.vm.identity.0.principal_id

  key_permissions    = ["get", "create", "delete", "list", "restore", "recover", "unwrapkey", "wrapkey", "purge", "encrypt", "decrypt", "sign", "verify",]
  }
}

  data "azurerm_private_dns_zone" "key_vault_dns_private_zone" {
  name     = "privatelink.vaultcore.azure.net"
  provider = azurerm.dnsprod
  resource_group_name = "NewTestNetwork"
}

resource "azurerm_private_endpoint" "keyvault" {
   name                = "key_vault-terraform-endpoint"
   location            = azurerm_resource_group.rg.location
   resource_group_name = azurerm_resource_group.rg.name
   subnet_id           = "${data.azurerm_subnet.rg.id}"

  private_service_connection {
    name                           = "key_vault-terraform-privateserviceconnection"
    private_connection_resource_id = azurerm_key_vault.main.id
    subresource_names              = [ "vault" ]
    is_manual_connection           = false
  }

 private_dns_zone_group {
    name = data.azurerm_private_dns_zone.key_vault_dns_private_zone.name
   private_dns_zone_ids = [data.azurerm_private_dns_zone.key_vault_dns_private_zone.id]
  }
}

resource "azurerm_key_vault_key" "key" {
  name         = "Storage-KEK200"
  key_vault_id = azurerm_key_vault.main.id
  key_type     = "RSA-HSM"
  key_size     = 2048
  expiration_date = "2050-01-01T00:00:00Z"

 key_opts = [
   "decrypt",
   "encrypt",
   "sign",
   "unwrapKey",
   "verify",
   "wrapKey",
  ]
}

Debug Output

Panic Output

Expected Behaviour

The Key gets created.

Actual Behaviour

Error: Error checking for presence of existing Key "Storage-KEK200" (Key Vault "https://newkekvault215.vault.azure.net/"): keyvault.BaseClient#GetKey: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="Client address is not authorized and caller is not a trusted service.\r\nClient address: 10.0.6.5 from unknown subnet\r\nCaller: appid=feb4f312-1ff0-4b05-94b0-933b72f199d5;oid=8257275c-a014-4140-ba4f-b3b656dc9f6d;iss=https://sts.windows.net/90ad1900-2019-4ffa-b845-96f012d0dc5a/;xms_mirid=/subscriptions//resourcegroups/development/providers/Microsoft.Compute/virtualMachines/mangementvm\r\nVault: newkekvault215;location=uksouth" InnerError={"code":"ForbiddenByFirewall"}

At this point the Private End Point, KV is created correctly and from the VM with the Managed Service Identity I can create a key with az cli for example over the Private End Point because the KV has no firewalls.
If without any change I do terraform apply again the key is deployed.

I've tried to put the Key Vault access policy via a separate resource and note this as a dependency for the key Vault Key resource and put in a timer delay but this didn't help.

Steps to Reproduce

terraform apply

Important Factoids

References

0000

antanof commented 3 years ago

Need to whitelist your Public IP (server for terraform administration) in the vault

cb900rr2000 commented 3 years ago

Hi antanof we are all over Private, we orchestrate from a Private IP as you can see in the error message, we don't want to switch to a Public IP for orchestration, also we don't want a Public IP firewall rule in our KV', also note if we perform an Apply again after the initial Apply it works so fundamentally this is possible I believe, but some state related to the Private End Point is not being imported until you run terraform again so I'm sure a bug.

michalswi commented 3 years ago

@cb900rr2000 , I would ask is it possible to achieve that in one terraform iteration at all instead of running terraform apply twice. In my case I am first using public MS API to create and configure kv + create private endpoints, once created I want to create keys using PE instead of public MS API and this is what I get:

Error: Error checking for presence of existing Key "default-database-key" (Key Vault "https://miswiercdemovault.vault.azure.net/"): keyvault.BaseClient#GetKey: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="Client address is not authorized and caller is not a trusted service.\r\nClient address: <SOME_IP>\r\nCaller: appid=XXX;oid=YYY;iss=https://sts.windows.net/ZZZ/\r\nVault: miswiercdemovault;location=westeurope" InnerError={"code":"ForbiddenByFirewall"}

It works for me if I run terraform twice. Same like you I don't have FW.

antanof commented 3 years ago

You have to look at the parameters: allow or deny by default. If we deny except the Azure services, they will go through the public IP of the vault. Then you have the public ip and authorized subnets. I think you need to add the subnet of your administration vm to the keyvault firewall white list.

tombuildsstuff commented 3 years ago

👋

At this time Terraform uses the Data Plane API to interact with Azure Key Vault for Certificates, Keys and Secrets - which is available over the public internet (although can be restricted using an IP Filter as described above). In order to support connecting to Key Vault over a Private Endpoint, we'd need to switch to using the Resource Manager API to do this, which may include permission changes - but this isn't supported at this time.

As such I've updated the issue title and tagged this as an enhancement to support creating Key Vault items over a Private Endpoint.

Thanks!

amarkulis commented 3 years ago

@tombuildsstuff Can you go into more detail for the work around? We are running into the error which I believe is caused by the same behavior?

Error: retrievingcontact` for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded

Error: retrieving contact for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded

[error]Terraform command 'plan' failed with exit code '1'.: retrieving `contact` for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded | retrieving `contact` for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded | retrieving `contact` for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded

[error]

Error: retrieving contact for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded

`

rahmnstein commented 3 years ago

Have this issue as well but I don't understand. Using a private endpoint does not mean that you cannot reach it from the internet. OP choose to bypass AzureServices from network acl, so Azure Devops, where the terraform code often runs, could use Data Plane API just as if the key vault did not have a Private Endpoint.

network_acls {
     default_action = "Deny"
     bypass         = "AzureServices"
}

J-i-K commented 3 years ago

👋

At this time Terraform uses the Data Plane API to interact with Azure Key Vault for Certificates, Keys and Secrets - which is available over the public internet (although can be restricted using an IP Filter as described above). In order to support connecting to Key Vault over a Private Endpoint, we'd need to switch to using the Resource Manager API to do this, which may include permission changes - but this isn't supported at this time.

> As such I've updated the issue title and tagged this as an enhancement to support creating Key Vault items over a Private Endpoint.

Thanks!

Hi @tombuildsstuff,

Can you provide any timeline on when this support could be in place?

Thanks!

do87 commented 3 years ago

Things that helped me workaround this issue until a fix is released: assigning certificate_permissions = [ "ManageContacts" ] to the Service Principal that controls terraform i also encountered context timeout, this was because i had private DNS for keyvault linked to the vnet and it prevented getting the public IP for the vault i was trying to modify. Removing the private DNS fixed the issue.

sebader commented 3 years ago

In order to support connecting to Key Vault over a Private Endpoint, we'd need to switch to using the Resource Manager API to do this, which may include permission changes - but this isn't supported at this time.

@tombuildsstuff do you have any pointers to this? I couldnt find any API docs around that. Thanks for any pointers!

zhangweikop commented 3 years ago

Generally, I think the KeyVault implementation should have one option to turn off the data plane access (and disable those related minor feature if needed) Include: Do not ping the keyvault URL for checking existence. Do not make any API such as get contacts.

Making data plane access always cause additional trouble when we have private network endpoint.

roy-work commented 3 years ago

At this time Terraform uses the Data Plane API to interact with Azure Key Vault for Certificates, Keys and Secrets - which is available over the public internet (although can be restricted using an IP Filter as described above). In order to support connecting to Key Vault over a Private Endpoint, we'd need to switch to using the Resource Manager API to do this, which may include permission changes - but this isn't supported at this time.

As such I've updated the issue title and tagged this as an enhancement to support creating Key Vault items over a Private Endpoint.

I am also getting this error (suddenly, too; to my knowledge, nothing has changed, and I can't find anything in the activity logs in Azure to indicate it).

But the KV for which I'm getting this on is set to "Allow access from: All networks" on the Networking / Firewalls and virtual networks page, and has no connections under "Private endpoint connections"…

oliviergaumond commented 2 years ago

Multiple issues have been closed as duplicates of this one. But I am not sur the following error is the exact same context as described here. Error: retrieving contact for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded

We get this error when trying to refresh the state even if there are no data plane operations in the Terraform file, as described in may of the linked issues.

Unless you tell me that the access policy is using the data plane?

AndreasMWalter commented 2 years ago

We have a similar issue right now in our current environment the keyvault is created before peerings are created and state refresh before peering creation will fail due to the required dataplane access to set these values. We can workaround this issue by deploying the keyvault after peerings and DNS are done but that might cause other issues.

dstratman-fi commented 2 years ago

Agree with @oliviergaumond

@tombuildsstuff

Can you clarify if you are including all the problems mentioned in - https://github.com/hashicorp/terraform-provider-azurerm/issues/10501 within this issue now as well even though this issue is labeled very differently above?

Thanks!

sbugalski commented 2 years ago

Unfortunately latest Azure Terraform Provider doesn't solve the issue.

Erry91 commented 2 years ago

As of now I am still having troubles on this matter. I am pretty sure that my case has already been listed, but just in case: I had created the Azure key vault with the access policy and configured a private endpoint. Until I removed the private endpoint manually (allowing public access didn't have an effect, nor deleting the policy in my case) Terraform wasn't able to refresh its state (failing with retrieving contact for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure).

Once I removed the PE then Terraform was able to plan again and get the state for the key vault. Do we know of any workaround for the issue in the meanwhile, as the team looks for a solution?

zhangweikop commented 2 years ago

Any progress on this issue in new year?

renenielsendk commented 2 years ago

Any workaround?

jpmicrosoft commented 2 years ago

In Azure DevOps I use an Azure DevOps Private Agent and ensure that it has access to the subnet where the key vault private endpoint resides. This is my work around. It works without issues.

What happens is that once Private Endpoints are enabled in the Key Vault, it shuts off public access. Terraform reaches out to the Key Vault to confirm that it exists and/or to make a change, etc... If the access from Terraform to the Key Vault is public, it will not be able to confirm access it thus it fails. I have seen a couple of different errors that are a symptom of the same issue.

I hope this helps someone.

simonesavi commented 2 years ago

I want to have an Azure Key Vault with this configuration:

Public Access Denied (no public IP or VNet allowed)
Private endpoint connected to a specific VNet

I'm executing the terraform commands from my laptop, this means that interaction towards Azure is done with a public IP.

First time that I create the Key Vault everything works smoothly, and the Azure Key Vault is created with associated private endpoint.

If I try to execute a new terraform plan (without change nothing in terraform configuration), the execution stuck until the following error is thrown:

│ Error: retrieving `contact` for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context canceled
│ 
│   with module.infrastructure.module.key-vault.azurerm_key_vault.main,
│   on ../modules/key-vault/key-vault-main.tf line 2, in resource "azurerm_key_vault" "main":
│    2: resource "azurerm_key_vault" "main" {
│

Following the @jpmicrosoft suggestion, I have tried to whitelist my IP. But this didn't solve problem. As second attempt I have tried to re-enable public access. Also this didn't solve problem.

The only way to solve it is to remove the private endpoint from Key Vault private endpoint connections section. In this way terraform plan restart to work, but, obviously, Key Vault won't have the associated private endpoint.

jpmicrosoft commented 2 years ago

I want to have an Azure Key Vault with this configuration:

Public Access Denied (no public IP or VNet allowed)

Private endpoint connected to a specific VNet

I'm executing the terraform commands from my laptop, this means that interaction towards Azure is done with a public IP.

First time that I create the Key Vault everything works smoothly, and the Azure Key Vault is created with associated private endpoint.

If I try to execute a new terraform plan (without change nothing in terraform configuration), the execution stuck until the following error is thrown:
│ Error: retrieving `contact` for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context canceled
│ 
│   with module.infrastructure.module.key-vault.azurerm_key_vault.main,
│   on ../modules/key-vault/key-vault-main.tf line 2, in resource "azurerm_key_vault" "main":
│    2: resource "azurerm_key_vault" "main" {
│ 
Following the @jpmicrosoft suggestion, I have tried to whitelist my IP. But this didn't solve problem. As second attempt I have tried to re-enable public access. Also this didn't solve problem.

The only way to solve it is to remove the private endpoint from Key Vault private endpoint connections section. In this way terraform plan restart to work, but, obviously, Key Vault won't have the associated private endpoint.

@simonesavi The workaround is not to add IPs to the ACL but to ensure you are executing Terraform from a VM/Container/etc. that has access to the Private Endpoints subnet.

I hope this helps.

simonesavi commented 2 years ago

After a troubleshooting I can confirm that the problem was generated by a DNS misconfiguration which did not correctly resolve the key vault name.

Anyway you require access to the Private Endpoints subnet only if the key vault name is resolved with private IP.

If you are using a public DNS, asking for a name resolution, you will receive public IP and terraform plan will works without problem.

DevOpsBoondoggles commented 2 years ago

I want to have an Azure Key Vault with this configuration:

Public Access Denied (no public IP or VNet allowed)

Private endpoint connected to a specific VNet

I'm executing the terraform commands from my laptop, this means that interaction towards Azure is done with a public IP. First time that I create the Key Vault everything works smoothly, and the Azure Key Vault is created with associated private endpoint. If I try to execute a new terraform plan (without change nothing in terraform configuration), the execution stuck until the following error is thrown:
│ Error: retrieving `contact` for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context canceled
│ 
│   with module.infrastructure.module.key-vault.azurerm_key_vault.main,
│   on ../modules/key-vault/key-vault-main.tf line 2, in resource "azurerm_key_vault" "main":
│    2: resource "azurerm_key_vault" "main" {
│ 
Following the @jpmicrosoft suggestion, I have tried to whitelist my IP. But this didn't solve problem. As second attempt I have tried to re-enable public access. Also this didn't solve problem. The only way to solve it is to remove the private endpoint from Key Vault private endpoint connections section. In this way terraform plan restart to work, but, obviously, Key Vault won't have the associated private endpoint.
@simonesavi The workaround is not to add IPs to the ACL but to ensure you are executing Terraform from a VM/Container/etc. that has access to the Private Endpoints subnet.

I hope this helps.

Heya there, I have a VMSS with access to the subnets, however I can't get the private endpoint to deploy. We have azure policy's and networking in place to stop public access. So state is in a private endpoint and we can't use a public agent. The problem comes when the keyvault gets deployed and is set to deny, the terraform fails with waiting for the keyvault to become available. This then prevents the Private endpoint deploying. If I try to run it again, I get hit with the contact error mentioned because I'm trying to do dataplane access but I can't deploy the private endpoint in terraform because I can't refresh. Is this just impossible then or am I missing something obvious ?

sebader commented 2 years ago

@gabrielmccoll we solved a similar situation when using Private Build Agents with Key Vault and Private Endpoints by the use of a deployment timer:

resource "azurerm_key_vault" "stamp" {
  name                = "${local.prefix}-${local.location_short}-kv"
  location            = azurerm_resource_group.stamp.location
  resource_group_name = azurerm_resource_group.stamp.name
  tenant_id           = data.azurerm_client_config.current.tenant_id

  network_acls {
    bypass         = "None"
    default_action = "Deny" # Deny all access - except for the private endpoint connections
  }
  sku_name = "standard"
}

# Give KV secret permissions to the service principal that runs the Terraform apply itself
resource "azurerm_key_vault_access_policy" "devops_pipeline_all" {
  key_vault_id = azurerm_key_vault.stamp.id

  tenant_id = data.azurerm_client_config.current.tenant_id
  object_id = data.azurerm_client_config.current.object_id

  secret_permissions = [
    "Get", "List", "Delete", "Purge", "Set", "Backup", "Restore", "Recover"
  ]
}

resource "azurerm_private_endpoint" "buildagent_keyvault" {
  name                = "${local.prefix}-${local.location_short}-built-agent-keyvault-pe"
  location            = data.azurerm_resource_group.buildagent.location
  resource_group_name = data.azurerm_resource_group.buildagent.name
  subnet_id           = "${data.azurerm_virtual_network.buildagent.id}/subnets/private-endpoints-snet"

  private_dns_zone_group {
    name                 = "privatednskeyvault"
    private_dns_zone_ids = ["${data.azurerm_resource_group.buildagent.id}/providers/Microsoft.Network/privateDnsZones/privatelink.vaultcore.azure.net"]
  }

  private_service_connection {
    name                           = "${local.prefix}-${local.location_short}-keyvault-buildagent-privateserviceconnection"
    private_connection_resource_id = azurerm_key_vault.stamp.id
    is_manual_connection           = false
    subresource_names              = ["vault"]
  }
}

resource "time_sleep" "wait_keyvault_pe" {
  depends_on = [azurerm_private_endpoint.buildagent_keyvault]

  create_duration = "300s" # 5min should give us enough time for the Private endpoint to come online
}

resource "azurerm_key_vault_secret" "secrets" {
  # Every secret is depended on a) the access policy for the deploying service principal being created and b) - only when running in private mode - on the build agent private endpoint being up and running
  depends_on = [azurerm_key_vault_access_policy.devops_pipeline_all, time_sleep.wait_keyvault_pe]
  name         = "foo"
  value        = "bar"
  key_vault_id = azurerm_key_vault.stamp.id
}

DevOpsBoondoggles commented 2 years ago

@gabrielmccoll we solved a similar situation when using Private Build Agents with Key Vault and Private Endpoints by the use of a deployment timer:

resource "azurerm_key_vault" "stamp" {
  name                = "${local.prefix}-${local.location_short}-kv"
  location            = azurerm_resource_group.stamp.location
  resource_group_name = azurerm_resource_group.stamp.name
  tenant_id           = data.azurerm_client_config.current.tenant_id

  network_acls {
    bypass         = "None"
    default_action = "Deny" # Deny all access - except for the private endpoint connections
  }
  sku_name = "standard"
}

# Give KV secret permissions to the service principal that runs the Terraform apply itself
resource "azurerm_key_vault_access_policy" "devops_pipeline_all" {
  key_vault_id = azurerm_key_vault.stamp.id

  tenant_id = data.azurerm_client_config.current.tenant_id
  object_id = data.azurerm_client_config.current.object_id

  secret_permissions = [
    "Get", "List", "Delete", "Purge", "Set", "Backup", "Restore", "Recover"
  ]
}

resource "azurerm_private_endpoint" "buildagent_keyvault" {
  name                = "${local.prefix}-${local.location_short}-built-agent-keyvault-pe"
  location            = data.azurerm_resource_group.buildagent.location
  resource_group_name = data.azurerm_resource_group.buildagent.name
  subnet_id           = "${data.azurerm_virtual_network.buildagent.id}/subnets/private-endpoints-snet"

  private_dns_zone_group {
    name                 = "privatednskeyvault"
    private_dns_zone_ids = ["${data.azurerm_resource_group.buildagent.id}/providers/Microsoft.Network/privateDnsZones/privatelink.vaultcore.azure.net"]
  }

  private_service_connection {
    name                           = "${local.prefix}-${local.location_short}-keyvault-buildagent-privateserviceconnection"
    private_connection_resource_id = azurerm_key_vault.stamp.id
    is_manual_connection           = false
    subresource_names              = ["vault"]
  }
}

resource "time_sleep" "wait_keyvault_pe" {
  depends_on = [azurerm_private_endpoint.buildagent_keyvault]

  create_duration = "300s" # 5min should give us enough time for the Private endpoint to come online
}

resource "azurerm_key_vault_secret" "secrets" {
  # Every secret is depended on a) the access policy for the deploying service principal being created and b) - only when running in private mode - on the build agent private endpoint being up and running
  depends_on = [azurerm_key_vault_access_policy.devops_pipeline_all, time_sleep.wait_keyvault_pe]
  name         = "foo"
  value        = "bar"
  key_vault_id = azurerm_key_vault.stamp.id
}

hey there, thank you for this, I've given it a try but I get same error "##[error]╷ │ Error: retrieving contact for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded"
Basically I can't get public routing to the data planeaccess due to policies and networking so I have to do it all private routing but that seems impossible since I can't get the private endpoint to deploy due to the above error, soon as the keyvault deploys it's a closed book it seems.

sebader commented 2 years ago

make sure the key vault gets newly deployed. The timer only applies the first time when the KV is actually getting created. On the next runs the timer should not be required anymore as then the Private Endpoint is already there and is used by your private build agent

DevOpsBoondoggles commented 2 years ago

make sure the key vault gets newly deployed. The timer only applies the first time when the KV is actually getting created. On the next runs the timer should not be required anymore as then the Private Endpoint is already there and is used by your private build agent

thanks again for the reply and help, the key vault is getting freshly deployed but doesn't complete as it failed with the above error. The private endpoint doesn't deploy because the initial deployment of the keyvault doesn't get all the way there it seems. So the PE never exists to be used for the next run.

sebader commented 2 years ago

looks like somewhere in your TF state you still have references to some KV-child resources (certificates?!). TF tries to check their state I guess. Try to start from a completely fresh (clean) state file

ncmuthu commented 2 years ago

I am able to pull the secrets through public interface(coresvcazvault01.vault.azure.net). I have created a private endpoint and configured the private dns zone and hostname with "coresvcazvault01.privatelink.vaultcore.azure.net". Anyone please help me on how to configure terraform resource to use this private endpoint to pull the secrets from the Azure Key vault. Would like to avoid using public interface for security reason.

data "azurerm_key_vault" "vault" {
  name                = "coresvcazvault01"
  resource_group_name = "azkeyvaulttest01"
}

data "azurerm_key_vault_secret" "secret1" {
  name         = "secret1"
  key_vault_id = data.azurerm_key_vault.vault.id
}

output "secret_value" {
  value = data.azurerm_key_vault_secret.secret1.value
  sensitive = true
}

output "vault_url1" {
  value = data.azurerm_key_vault.vault.vault_uri
  sensitive = false
}
root@terraform# terraform output
secret_value = <sensitive>
vault_url1 = "https://coresvcazvault01.vault.azure.net/"

jpmicrosoft commented 2 years ago

@Ncmuthu You must run terraform from a location that has network connectivity to the Key Vault via the Private Endpoint. The code used will be the same as any other method you would use to call a Key Vault. The difference will be the route in which the data travels because of the connectivity between the source (where terraform runs) and the Key Vault Private Endpoint. In Azure Devops we accomplish this by using Self Hosted Agents (Private Agents). The normal Microsoft Hosted Agent will not be able to access anything behind a Private Endpoint.

I hope this helps.

Lddeiva commented 2 years ago

As Microsoft.KeyVault/Vaults/Certificates/* is a data plane operation, the following code (L684-L692) needs to be updated. These lines must only run after verifying key vault network settings. Running these lines in a network protected key vault without checking the dependencies such as private endpoint and private DNS A records will result in error.

https://github.com/hashicorp/terraform-provider-azurerm/blob/7110726415ebb8f2faeb0fe8a2bfdb590f6c39a7/internal/services/keyvault/key_vault_resource.go#L684

sebader commented 2 years ago

@Lddeiva the problem is that just adding a dependency to the Private Endpoint resource is mostly not enough. I found that it takes some time for the private endpoint connection to become effective for the machine on which you are running Terraform on. That's why I had to add the timer resource as shown in my example above.

Lddeiva commented 2 years ago

Hi @sebader , My response was for #10501.

chadcarlton commented 2 years ago

Terraform v1.0.2 on windows_amd64

provider registry.terraform.io/hashicorp/azuread v2.19.1
provider registry.terraform.io/hashicorp/azurerm v3.0.1
provider registry.terraform.io/hashicorp/random v3.1.2

I am recieving this error│ Error: retrievingcontactfor KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded when working with Key vaults that do NOT use private endpoints or any Firewall or virtual network settings.

I have seen other people mention this as well, in issues that have been closed and marked as duplicate. I believe there are multiple issues here, or there is a bug that affects both KV terraform operations regardless of networking/dataplane status.

silvesterc commented 2 years ago

I had this issue and it was caused by the proxy. I had to put in a proxy bypass for the vault and vaultcore URLs.

silvesterc commented 2 years ago

I had this issue and it was caused by the proxy. I had to put in a proxy bypass for the vault and vaultcore URLs.

Okay, this opened a can of worms. I think I've narrowed down the issue and have a workaround but it's not a good one.

The issue appears to be the fact that when used with a private endpoint, both public and private IPs are used on the same URL. When a fresh build is started, it goes to the public interface for Azure and creates the vault. TF checks the vault has been created on it's url i.e. https://examplekv.vault.azure.net, once it's been created it will continue until the private endpoint is present; at this point, connectivity via the public interface is lost. Adding a proxy bypass for the vault's address will allow connectivity again and future plans will work however; you cannot re-build the vault as the proxy bypass will block TF from checking the URL on it's public interface to see if it's been built.

The workaround is to check if a private endpoint is present. If not, make sure there is no proxy bypass, if so add one. When rebuilding a vault, you must start from scratch; delete the PE, private DNS record etc.

natilik-mikeguy commented 1 year ago

I was going to open a new bug, but this looks to be the same thing, so I haven't. When creating a key vault using a privatelink (and no public access), plus all the associated resources - it is fine. However, if you try and create a key (or cert etc.) in the same config during the same run, it fails as Terraform tries to access the Vault public IP. Obviously Terraform is running from a VM with VNet connectivity to the resource and ability to resolve the URLs against private DNS zones.

I presume what is happening is DNS for the Vault URI get's resolved before the CNAME has automatically been added by Azure to point to .privatelink.vaultcore.azure.net. The subsequent azurerm_key_vault_key resource then fails as it is hitting the public endpoint.

╷
│ Error: checking for presence of existing Key "tfex-des-key" (Key Vault "https://kv-vault-test-mike.vault.azure.net/"): keyvault.BaseClient#GetKey: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="Public network access is disabled and request is not from a trusted service nor via an approved private link.\r\nCaller: appid=f38b65f5-7e0c-4639-9d9c-11eb1a8ee409;oid=89252736-c950-4a5f-be6b-4ff64f4ebf6d;iss=https://sts.windows.net/f8667506-a537-4c81-842a-41fd0e547e43/\r\nVault: kv-vault-test-mike;location=uksouth" InnerError={"code":"ForbiddenByConnection"}
│
│   with azurerm_key_vault_key.tf_kv_std_deskey,
│   on main.tf line 129, in resource "azurerm_key_vault_key" "tf_kv_std_deskey":
│  129: resource "azurerm_key_vault_key" "tf_kv_std_deskey" {

If you re-run it seems to work the second time. I did try adding a provisioner to my azurerm_private_endpoint resource to force a cache flush, but didn't seem to work (I guess the provider is caching it somewhere).

  provisioner "local-exec" {
    interpreter = ["/bin/bash", "-c"]
    command     = "sleep 60 && systemd-resolve --flush-caches"
  }

I've not tried some of the other workarounds yet, but a bit of a pain!

jpmicrosoft commented 1 year ago

My work around is to deploy the KeyVault, add the secrets, etc. then enable Private Endpoints at the end. Next time things run you should be in the clear as long as the device running the code is in a vNet that can resolve the PEs.

sebader commented 1 year ago

@natilik-mikeguy have you tried the solution I posted above using a deployment timer? that works quite well for me

lieberlois commented 1 year ago

Any updates on this? I get the same issue even without a private endpoint. Key Vault is stuck in "Still creating..." state. When cancelled, terraform logs Error: retrieving contact for KeyVault: keyvault.BaseClient#GetCertificateContacts: Failure sending request: StatusCode=0 -- Original Error: context deadline exceeded

masbed commented 1 year ago

I have the same issue, even without private endpoints. What worked for me was to downgrade azurerm from 3.* to 2.99.

# azurerm_key_vault.apim_keyvault:
resource "azurerm_key_vault" "apim_keyvault" {
    name                = local.apim_keyvault_name
    location            = local.location
    resource_group_name = local.resource_group_name
    tenant_id           = data.azurerm_client_config.current.tenant_id
    sku_name            = "standard"
    tags                = {}

    enabled_for_disk_encryption     = true
    enabled_for_template_deployment = false
    enabled_for_deployment          = false
    soft_delete_retention_days      = 90
    purge_protection_enabled        = false

    network_acls {
        default_action = "Allow"
        bypass         = "AzureServices"
    }
}

#azurerm_key_vault_access_policy.default_policy
resource "azurerm_key_vault_access_policy" "default_policy" {
    key_vault_id = azurerm_key_vault.apim_keyvault.id
    tenant_id    = data.azurerm_client_config.current.tenant_id
    object_id    = data.azurerm_client_config.current.object_id

    lifecycle {
        create_before_destroy = true
    }

    key_permissions = var.kv-key-permissions-full
    secret_permissions = var.kv-secret-permissions-full
    certificate_permissions = var.kv-certificate-permissions-full
    storage_permissions = var.kv-storage-permissions-full
}

lieberlois commented 1 year ago

@masbed I also fixed this :) appears to be an issue with azurerm=3.0.0 I changed the version to 3.29.1 (latest stable patch) and got it working 😄

resource "azurerm_key_vault" "demo" {
  name                        = "kv-${local.settings.projectName}-${terraform.workspace}"
  location                    = azurerm_resource_group.demo.location
  resource_group_name         = azurerm_resource_group.demo.name
  enabled_for_disk_encryption = true
  tenant_id                   = local.settings.tenantId
  soft_delete_retention_days  = 7
  purge_protection_enabled    = false

  sku_name = local.settings.keyVaultSku
}

resource "azurerm_key_vault_access_policy" "demo" {
  depends_on = [
    azurerm_key_vault.demo
  ]

  key_vault_id = azurerm_key_vault.demo.id

  for_each = toset(local.settings.devOpsAdmins)

  tenant_id = local.settings.tenantId
  object_id = each.key

  secret_permissions = [
    "Get",
    "Backup",
    "Delete",
    "List",
    "Purge",
    "Recover",
    "Restore",
    "Set",
  ]
}

natilik-mikeguy commented 1 year ago

@natilik-mikeguy have you tried the solution I posted above using a deployment timer? that works quite well for me

@sebader - it seemed to work for the Key Vault private endpoint, but not for Azure Storage (which I appreciate this issue isn't about - but it is all the same issue). I have to re-run the apply to create tables/blob containers after the initial failed with a 403

│ Error: containers.Client#GetProperties: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="AuthorizationFailure" Message="This request is not authorized to perform this operation.\nRequestId:c745f5d5-201e-003d-7d09-032d5d000000\nTime:2022-11-28T09:10:14.6282172Z"
│

sebader commented 1 year ago

yes, the exact same solution works for storage for me, too. See complete example here: https://github.com/Azure/Mission-Critical-Connected/blob/main/src/infra/workload/releaseunit/modules/stamp/storage.tf#L38

pwolthausen commented 1 year ago

I'm experiencing the same problem, trying to use airgaped environment.

The key vault resource is created, but does not report ready
the network acls I configured in terraform are not getting set in the key vault resource so the api request will time out.

My runner does have public access, so it can resolve the public DNS name (though I'd rather block it). But it can't interact with the endpoint because there are no allowed rules

I can't use purely private access because the private endpoint can't be created (and thus the private DNS entry) can't be created until after the key vault is created.

pwolthausen commented 1 year ago

If I try a wget from the runner where terraform is being run:

azureuser@GitLabRunner-UbuntuServer22-VM-01:~$ wget  https://myvault.vault.azure.net
--2022-12-21 00:20:59--  https://myvault.vault.azure.net/
Resolving myvault.vault.azure.net (myvault.vault.azure.net)... 20.48.197.105
Connecting to myvault.vault.azure.net (myvault.vault.azure.net)|20.48.197.105|:443... connected.
OpenSSL: error:0A000126:SSL routines::unexpected eof while reading
Unable to establish SSL connection.

magodo commented 1 year ago

Providing a workaround using azapi provider (so that we can avoid calling the contact list data plane API), e.g.:

resource "azapi_resource" "vault" {
  type      = "Microsoft.KeyVault/vaults@2021-10-01"
  parent_id = azurerm_resource_group.test.id
  name      = "vault230119153000994165"
  location  = azapi_resource.resourceGroup.location
  body = jsonencode({
    properties = {
      accessPolicies = [
        {
          objectId = "xxxx"
          permissions = {
            certificates = [
              "ManageContacts",
            ]
            keys = [
              "Create",
            ]
            secrets = [
              "Set",
            ]
            storage = [
            ]
          }
          tenantId = "xxxx"
        },
      ]
      enableRbacAuthorization      = false
      enableSoftDelete             = true
      enabledForDeployment         = false
      enabledForDiskEncryption     = false
      enabledForTemplateDeployment = false
      publicNetworkAccess          = "Enabled"
      sku = {
        family = "A"
        name   = "standard"
      }
      softDeleteRetentionInDays = 7
      tenantId                  = "xxxx"
    }
  })
}

johhess40 commented 1 year ago

Ran into this same issue @tombuildsstuff and was able to add the contact field to the ignore_changes field inside of the lifecycle block for the key vault resource:

  lifecycle {
    ignore_changes = [
      tags["CreationDate"],
      tags["creationDate"],
      tags["Project"],
      contact
    ]
  }

I think that the code mentioned above by @Lddeiva is the root of the issue as the vault's uri is used to query this property, and an addition to that function which can check the status of networking/private endpoints for the vault may be useful. Either way this was my workaround so that my pipelines could proceed when checking against state. I also don't usually use the ignore_changes functionality, but in this instance it is inconsequential since the field can only be set once and cannot be change per the documentation here.

reshmav18 commented 1 year ago

Hi @tombuildsstuff , I'm reaching out w.r.t to https://github.com/hashicorp/terraform-provider-azurerm/issues/21876

My customer whitelisted IP address as per the suggestion. however, issue still persist. Looking for an alternative.

Thanks

hashicorp / terraform-provider-azurerm