Private Endpoint for Web App Slot fails in Azurerm >= 3.108.0

donjuanmon commented 1 month ago

Is there an existing issue for this?

[X] I have searched the existing issues

Community Note

Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave comments along the lines of "+1", "me too" or "any updates", they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Terraform Version

1.8.5

AzureRM Provider Version

>=3.108.0

Affected Resource(s)/Data Source(s)

azurerm_private_endpoint

Terraform Configuration Files

resource "azurerm_private_endpoint" "web_app_slot" {
  name                = azurecaf_name.web_app_slot_private_endpoint[0].result
  location            = var.location
  resource_group_name = var.resource_group_name
  subnet_id           = var.private_endpoint_subnet_id

  private_service_connection {
    name                           = azurecaf_name.web_app_slot_private_endpoint[0].results["azurerm_private_service_connection"]
    subresource_names              = ["sites-${var.slot_name}"]
    is_manual_connection           = false
  }

  private_dns_zone_group {
    name = azurecaf_name.web_app_slot_private_endpoint[0].results["azurerm_private_dns_zone_group"]
    private_dns_zone_ids = [
      var.privatelink_web_site_private_dns_zone_id
    ]
  }
}

Debug Output/Panic Output

{"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Web/sites/lxwapp-webapp-terratest/slots/slottest' under resource group 'rg-webapp-terratest' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}: timestamp=2024-07-05T15:21:22.214-0500
2024-07-05T15:21:22.215-0500 [DEBUG] provider.terraform-provider-azurerm_v3.108.0_x5: AzureRM Request: 
PUT /subscriptions/18a0edd3-31af-414f-ab0c-f5edc746cf6a/resourceGroups/rg-webapp-terratest/providers/Microsoft.Web/sites/lxwapp-webapp-terratest/slots/slottest?api-version=2023-01-01 HTTP/1.1
Host: management.azure.com
User-Agent: HashiCorp/go-azure-sdk (Go-http-Client/1.1 webapps/2023-01-01) HashiCorp Terraform/1.8.5 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/3.108.0 VSTS_af0c2b0e-4e05-48b9-9d3e-96a4575992dc_build_7_0 pid-222c6c49-1b0a-5959-a213-6608f9eb8820
Content-Length: 1298
Content-Type: application/json; charset=utf-8
X-Ms-Correlation-Request-Id: 8a55a010-5a01-99a7-6ec1-c93aea755323
Accept-Encoding: gzip

{"identity":{"type":"SystemAssigned","userAssignedIdentities":null},"location":"northcentralus","properties":{"clientAffinityEnabled":false,"clientCertEnabled":false,"clientCertMode":"Required","enabled":true,"httpsOnly":false,"publicNetworkAccess":"Enabled","serverFarmId":"/subscriptions/18a0edd3-31af-414f-ab0c-f5edc746cf6a/resourceGroups/rg-webapp-terratest/providers/Microsoft.Web/serverFarms/plan-webapp-terratest","siteConfig":{"acrUseManagedIdentityCreds":false,"alwaysOn":true,"appSettings":[{"name":"EXAMPLE_STICKY","value":"example"},{"name":"EXAMPLE_PERSISTENT","value":"example"}],"autoHealEnabled":false,"ftpsState":"Disabled","http20Enabled":false,"ipSecurityRestrictionsDefaultAction":"Allow","linuxFxVersion":"DOCKER|registry.doit.wisc.edu/smph/smph-it/informatics/apps/px-redcap/13.8.4/px-redcap-web:latest","loadBalancing":"LeastRequests","localMySqlEnabled":false,"managedPipelineMode":"Integrated","minTlsVersion":"1.2","publicNetworkAccess":"Enabled","remoteDebuggingEnabled":false,"scmIpSecurityRestrictionsDefaultAction":"Allow","scmIpSecurityRestrictionsUseMain":true,"scmMinTlsVersion":"1.2","use32BitWorkerProcess":true,"vnetRouteAllEnabled":false,"webSocketsEnabled":false},"vnetRouteAllEnabled":false},"tags":{"project":"webapp","uw_msn_udds":"A000000","zone":"green"}}: timestamp=2024-07-05T15:21:22.215-0500
2024-07-05T15:21:22.215-0500 [DEBUG] provider.terraform-provider-azurerm_v3.108.0_x5: PUT https://management.azure.com/subscriptions/18a0edd3-31af-414f-ab0c-f5edc746cf6a/resourceGroups/rg-webapp-terratest/providers/Microsoft.Web/sites/lxwapp-webapp-terratest/slots/slottest?api-version=2023-01-01: timestamp=2024-07-05T15:21:22.215-0500
2024-07-05T15:21:22.942-0500 [DEBUG] provider.terraform-provider-azurerm_v3.108.0_x5: AzureRM Response for https://management.azure.com/subscriptions/18a0edd3-31af-414f-ab0c-f5edc746cf6a/resourceGroups/rg-webapp-terratest/providers/Microsoft.Network/privateEndpoints/pe-slottest-terratest?api-version=2023-11-01: 
HTTP/2.0 400 Bad Request
Content-Length: 127
Cache-Control: no-cache
Content-Type: application/json; charset=utf-8
Date: Fri, 05 Jul 2024 20:21:22 GMT
Expires: -1
Pragma: no-cache
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Cache: CONFIG_NOCACHE
X-Content-Type-Options: nosniff
X-Ms-Arm-Service-Request-Id: 83c82507-e924-48df-ae1f-9ca2185a7836
X-Ms-Correlation-Request-Id: 8a55a010-5a01-99a7-6ec1-c93aea755323
X-Ms-Ratelimit-Remaining-Subscription-Global-Writes: 2999
X-Ms-Ratelimit-Remaining-Subscription-Writes: 199
X-Ms-Request-Id: 11475149-333f-4b01-865e-81260d4441e7
X-Ms-Routing-Request-Id: NORTHCENTRALUS:20240705T202122Z:e719ee98-9593-4529-a02e-dcbf01673077
X-Msedge-Ref: Ref A: 5EEAB7C37D574289ABF5CCEF97C34D56 Ref B: CH1AA2020610035 Ref C: 2024-07-05T20:21:22Z

{"error":{"code":"BadRequest","message":"Call to Microsoft.Web/sites failed. Error message: GroupId is invalid.","details":[]}}: timestamp=2024-07-05T15:21:22.942-0500
2024-07-05T15:21:22.942-0500 [ERROR] provider.terraform-provider-azurerm_v3.108.0_x5: Response contains error diagnostic: tf_provider_addr=provider tf_req_id=1c8c3068-48a1-787a-d0b4-a1f48f3f16eb diagnostic_detail="" diagnostic_severity=ERROR
  diagnostic_summary=
  | creating Private Endpoint (Subscription: "18a0edd3-31af-414f-ab0c-f5edc746cf6a"
  | Resource Group Name: "rg-webapp-terratest"
  | Private Endpoint Name: "pe-slottest-terratest"): performing CreateOrUpdate: unexpected status 400 (400 Bad Request) with error: BadRequest: Call to Microsoft.Web/sites failed. Error message: GroupId is invalid.
   tf_proto_version=5.4 tf_rpc=ApplyResourceChange @caller=github.com/hashicorp/terraform-plugin-go@v0.19.0/tfprotov5/internal/diag/diagnostics.go:58 @module=sdk.proto tf_resource_type=azurerm_private_endpoint timestamp=2024-07-05T15:21:22.942-0500
2024-07-05T15:21:22.960-0500 [ERROR] vertex "module.web_app_deployment_slot.azurerm_private_endpoint.web_app_slot[0]" error: creating Private Endpoint (Subscription: "18a0edd3-31af-414f-ab0c-f5edc746cf6a"
Resource Group Name: "rg-webapp-terratest"
Private Endpoint Name: "pe-slottest-terratest"): performing CreateOrUpdate: unexpected status 400 (400 Bad Request) with error: BadRequest: Call to Microsoft.Web/sites failed. Error message: GroupId is invalid.

Expected Behaviour

In <=3.107.0, the working solution was to use subresource_names = ["sites-slotname"] as pointed out by https://github.com/hashicorp/terraform-provider-azurerm/issues/17551 and Microsofts own documentation: https://learn.microsoft.com/en-us/azure/app-service/overview-private-endpoint#conceptual-overview

Each slot of an app is configured separately. You can plug up to 100 private endpoints per slot. You can't share a private endpoint between slots. The sub-resource name of a slot is sites-.

Actual Behaviour

In AzureRM >=3.108.0, terraform apply fails with:

CreateOrUpdate: unexpected status 400 (400 Bad Request) with error: BadRequest: Call to Microsoft.Web/sites failed. Error message: GroupId is invalid.

Changing subresource_names to singulare ["sites"] fixes the problem, but I don't see this documented anywhere. Wondering if this was changed with serviceconnector PR?

Steps to Reproduce

Use subresource_names = ["sites-<slot_name>"] Pin azurerm provider to >=3.108.0 and try to create a Private Endpoint for a web app slot. Fails with:

CreateOrUpdate: unexpected status 400 (400 Bad Request) with error: BadRequest: Call to Microsoft.Web/sites failed. Error message: GroupId is invalid.

Setting subresource_names = ["sites"] fixes issue, but this is not documented anywhere.

Important Factoids

N/A

References

https://github.com/hashicorp/terraform-provider-azurerm/issues/17551

https://learn.microsoft.com/en-us/azure/app-service/overview-private-endpoint#conceptual-overview

xiaxyi commented 1 month ago

Thanks @donjuanmon for raising this issue, I don't see the required field private_connection_resource_id or private_connection_resource_alias is specified in your private_service_connection block doc reference:https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/private_endpoint#private_connection_resource_id

Besides of the required field, have you tried to use app service id as the private_connection_resource_id as below example? The PE can be created without any issue by specifying the private_connection_resource_id + subresource_names=["sites-xiaxintestWAS-app-pe"]

 private_service_connection {
    name                           = "xiaxintest-pe-connection"
    subresource_names              = ["sites-xiaxintestWAS-app-pe"]
    is_manual_connection           = false
    private_connection_resource_id = azurerm_linux_web_app.test.id

  }

Let me know if you have any further questions.

donjuanmon commented 1 month ago

Hey @xiaxyi,

Apologies, I must have removed private_connection_serouce_id when cleaning up extra comments. Here is the full code block I have been using in a web_app_slot module:

resource "azurerm_private_endpoint" "web_app_slot" {
  count               = var.enable_private_endpoint ? 1 : 0
  name                = azurecaf_name.web_app_slot_private_endpoint[0].result
  location            = var.location
  resource_group_name = var.resource_group_name
  subnet_id           = var.private_endpoint_subnet_id

  private_service_connection {
    name                           = azurecaf_name.web_app_slot_private_endpoint[0].results["azurerm_private_service_connection"]
    private_connection_resource_id = var.app_service_id
    subresource_names              = ["sites-${var.slot_name}"]

    is_manual_connection           = false
  }

  private_dns_zone_group {
    name = azurecaf_name.web_app_slot_private_endpoint[0].results["azurerm_private_dns_zone_group"]
    private_dns_zone_ids = [
      var.privatelink_web_site_private_dns_zone_id
    ]
  }
}

I have confirmed this fails on azurerm provider versions 3.108.0 and greater repeatedly. Can you share what version of the provider you are using?

xiaxyi commented 1 month ago

@donjuanmon I used 3.111.0, but the PE can still be created even if I switched to 3.108.0

Is the failure always happening or intermittently?

donjuanmon commented 1 month ago

Hey @xiaxyi, failure is always happening. We use Terratest to validate our Azure modules and the only solution that works 100% of the time is pinning to 3.107 or lower. Happy to share more debug information with you. Can you confirm which version of Terraform you are using?

GerardLarwa commented 2 weeks ago

Hi @donjuanmon. Try to add dependency (dependsOn) on slot to private endpoint. Seems Terraform wants to create PE when the slot is not built yet.

resource "azurerm_private_endpoint" "web_app_slot" {
  [....]

  depends_on = [
    azurerm_linux_function_app_slot.slot
  ]
}

hashicorp / terraform-provider-azurerm