azurerm_app_service_virtual_network_swift_connection - Error supporting cross subscription app -> subnet

darren-mcdonald commented 2 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform 1.0.7 AzureRM 2.86.0

Affected Resource(s)

azurerm_app_service_virtual_network_swift_connection

Terraform Configuration Files

SCENARIO 1

main.tf

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = ">= 2.73.0"
    }
  }
}

provider "azurerm" { #app subscription
  subscription_id = "<APP_SUB_ID>"
  tenant_id       = "<APP_TENANT_ID>"
  client_id       = "<APP_CLIENT_ID>"
  client_secret   = "<APP_CLIENT_SECRET>"
  features {}
}

provider "azurerm" {
  alias = "vnetsubscription"
  subscription_id = "<VNET_SUB_ID>"
  tenant_id       = "<VNET_TENANT_ID>"
  client_id       = "<VNET_CLIENT_ID>"
  client_secret   = "<VNET_CLIENT_SECRET>"
  features {}
}

appsub.tf

resource "azurerm_resource_group" "example" {
  name     = "test"
  location = "North Europe"
}

resource "azurerm_app_service_plan" "example" {
  name                = "example-appserviceplan"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name

  sku {
    tier = "Standard"
    size = "S1"
  }
}

resource "azurerm_app_service" "example" {
  name                = "my-example-app-service"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
  app_service_plan_id = azurerm_app_service_plan.example.id
}

vnetsub.tf

data "azurerm_resource_group" "vnet_rg" {
  provider = azurerm.vnetsubscription
  name = "vnet-rg"
}

data "azurerm_virtual_network" "example" {
  provider = azurerm.vnetsubscription
  name                = "my-vnet"
  resource_group_name = data.azurerm_resource_group.vnet_rg.name
}

resource "azurerm_subnet" "example" {
  provider = azurerm.vnetsubscription
  name                 = "example-subnet"
  resource_group_name  = data.azurerm_resource_group.vnet_rg.name
  virtual_network_name = data.azurerm_virtual_network.example.name
  address_prefixes     = ["10.0.1.0/28"]
  delegation {
    name = "example-delegation"

    service_delegation {
      name    = "Microsoft.Web/serverFarms"
      actions = ["Microsoft.Network/virtualNetworks/subnets/action"]
    }
  }
}

resource "azurerm_app_service_virtual_network_swift_connection" "example" {
  provider = azurerm.vnetsubscription
  app_service_id = azurerm_app_service.example.id
  subnet_id      = azurerm_subnet.example.id
}

SCENARIO 2

As above but "azurerm_app_service_virtual_network_swift_connection.example" is created from the default (app service subscription) provider.

resource "azurerm_app_service_virtual_network_swift_connection" "example" {
  app_service_id = azurerm_app_service.example.id
  subnet_id      = azurerm_subnet.example.id
}

Debug Output

SCENARIO 1

  # azurerm_app_service_virtual_network_swift_connection.example will be created
  + resource "azurerm_app_service_virtual_network_swift_connection" "example" {
      + app_service_id = "/subscriptions/<appsubscription>/resourceGroups/test/providers/Microsoft.Web/sites/my-example-app-service"
      + id             = (known after apply)
      + subnet_id      = "/subscriptions/<vnetsubscription>/resourceGroups/vnet-rg/providers/Microsoft.Network/virtualNetworks/my-vnet/subnets/example-subnet"
    }

azurerm_app_service_virtual_network_swift_connection.example: Creating...
╷
│ Error: retrieving existing App Service "my-example-app-service" (Resource Group "test"): App Service not found in resource group
│ 
│   with azurerm_app_service_virtual_network_swift_connection.example,
│   on shared.tf line 28, in resource "azurerm_app_service_virtual_network_swift_connection" "example":
│   28: resource "azurerm_app_service_virtual_network_swift_connection" "example" {
│

SCENARIO 2

│ Error: waiting for provisioning state of subnet for App Service VNet association between "my-example-app-service" (Resource Group "test") and Virtual Network "my-vnet": polling for Subnet: (Name "example-subnet" / Virtual Network Name "my-vnet" / Resource Group "vnet-rg"): network.SubnetsClient#Get: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceGroupNotFound" Message="Resource group 'vnet-rg' could not be found."
│ 
│   with azurerm_app_service_virtual_network_swift_connection.example,
│   on aod.tf line 31, in resource "azurerm_app_service_virtual_network_swift_connection" "example":
│   31: resource "azurerm_app_service_virtual_network_swift_connection" "example" {

Expected Behaviour

SCENARIO 1 & 2 : Terraform does not respond with an error and joins the application to the subnet

Actual Behaviour

SCENARIO 1: Error response saying it cannot find the app service even though the provider is using a service principal with contributor permissions to the apps resource group. I believe it might be looking for the app resource group under the context of the vnet provider subscription rather than the subscription id in the app resource id.

SCENARIO 2 : The app is successfully connected to the subnet but it seems terraform wants to do a refresh on the vnet resource group and cant find it so throws an error. I believe it might be looking for the vnet resource group under the context of the app provider subscription rather than the subscription id in the subnet resource id.

Steps to Reproduce

terraform apply

References

darren-mcdonald commented 2 years ago

Link to demo to recreate the issue - https://github.com/darren-mcdonald/azurerm_app_service_virtual_network_swift_connection_issue

darren-mcdonald commented 2 years ago

I'm very fresh with Go but I believe the issue stems from here: https://github.com/Azure/azure-sdk-for-go/blob/main/services/network/mgmt/2021-02-01/network/subnets.go#L242

It's using the clients subscriptionID which I presume in this context is that of the app service subscription and not that of the vnet subscription. It feels like a subscriptionId parameter needs to be passed into the function instead. As I say I'm new to Go so I'd welcome any input.

haydnchapman-pg commented 2 years ago

Are there any updates on this?

ghost commented 2 years ago

I am also having this issue, please let us know an update. I'm happy to engage on any level I can to assist resolution.

chadcarlton commented 2 years ago

This is still an issue/occurring using: hashicorp/azurerm v2.91.0

I can confirm that Vnet integration is successfully setup, even though terraform throws an error during the apply.

it looks like the state file doesn't get updated, so later apply runs get a message about needing an import.

Please provide status on this issue.

ghost commented 2 years ago

For me, the apply fails with the error that it can't find find the right RG. But actually it applies the link to the app service successfully anyway so the next apply shows that the connection already exists if we could pass in two providers that'd be the ticket.

pregress commented 2 years ago

Still applies to: Terraform: 1.1.9 AzureRM: 3.5.0

After the first apply you can import it and the error is gone for next applies. terraform import azurerm_app_service_virtual_network_swift_connection.example /subscriptions/.../...

dtotopus commented 2 years ago

Yep, this is legitimate issue. Currently using hashicorp/azurerm 3.12.0 and receiving `azurerm_app_service_virtual_network_swift_connection.vnet_integration: Creating... ╷ │ Error: retrieving existing App Service "app-test" (Resource Group "rg-test"): App Service not found in resource group │ │ with azurerm_app_service_virtual_network_swift_connection.vnet_integration, │ on main.tf line 7, in resource "azurerm_app_service_virtual_network_swift_connection" "vnet_integration": │ 7: resource "azurerm_app_service_virtual_network_swift_connection" "vnet_integration" { │ ╵ ERRO[0072] 1 error occurred:

exit status 1` Integration does work when created over Azure Portal

BigFrog-coding commented 1 year ago

Hitting this exact same issue now and it is blocking deployments. Please help

dtotopus commented 1 year ago

Hitting this exact same issue now and it is blocking deployments. Please help

Microsoft doesn't care. Simply enable it manually, that's it :) I'm using trrragrunt so I'm just skipping the resource till the better day comes :)

hashicorp / terraform-provider-azurerm