bug: Subscription with vnet fails on first run as Microsoft.Network provider is not yet registered

earldata commented 1 year ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Versions

Please paste the output of terraform version command from within the initialized directory:

Terraform v1.5.4
on linux_amd64

Please enter the module version that you are using:

v3.3.0

Description

When creating a subscription with virtual network enabled it fails on the first run with:

│ Error: creating/updating "Resource: (ResourceId \"/subscriptions/***/resourceGroups/***/providers/Microsoft.Network/virtualNetworks/***\" / Api Version \"2021-08-01\")": PUT https://management.azure.com/subscriptions/***/resourceGroups/***/providers/Microsoft.Network/virtualNetworks/***
│ --------------------------------------------------------------------------------
│ RESPONSE 409: 409 Conflict
│ ERROR CODE: SubscriptionNotRegistered
│ --------------------------------------------------------------------------------
│ {
│   "error": {
│     "code": "SubscriptionNotRegistered",
│     "message": "Subscription *** is not registered with NRP.",
│     "details": []
│   }
│ }
│ --------------------------------------------------------------------------------
│ 
│ 
│   with module.subscription.module.virtualnetwork[0].azapi_resource.vnet["***"],
│   on .terraform/modules/subscription/modules/virtualnetwork/main.tf line 37, in resource "azapi_resource" "vnet":
│   37: resource "azapi_resource" "vnet" {

I believe this is because the Microsoft.Network provider has not been registered yet. This can be seen in the terraform output:

module.subscription.module.virtualnetwork[0].azapi_resource.vnet["***"]: Creating...
module.subscription.module.subscription[0].azapi_resource_action.resource_provider_registration["Microsoft.GuestConfiguration"]: Creation complete after 1s [id=/subscriptions/***/providers/Microsoft.GuestConfiguration/register]
module.subscription.module.subscription[0].azapi_resource_action.resource_provider_registration["Microsoft.PowerBIDedicated"]: Creation complete after 1s [id=/subscriptions/***/providers/Microsoft.PowerBIDedicated/register]
module.subscription.module.subscription[0].azapi_resource_action.resource_provider_registration["Microsoft.Network"]: Creation complete after 1s [id=/subscriptions/***/providers/Microsoft.Network/register]

As above you can see that the vnet is created before the provider is registered.

Steps to Reproduce

The following config was used:

module "subscription" {
  source  = "Azure/lz-vending/azurerm"
  version = "v3.3.0"

  location = "northeurope"

  subscription_alias_enabled = true
  subscription_billing_scope = var.billing_scope
  subscription_display_name  = var.name
  subscription_alias_name    = var.name
  subscription_workload      = "Production"

  network_watcher_resource_group_enabled = true

  subscription_management_group_association_enabled = true
  subscription_management_group_id                  = var.management_group_id
  subscription_tags                                 = var.subscription_tags

  virtual_network_enabled = true
  virtual_networks = {
    main = {
      name                                   = "main"
      address_space                          = var.address_space
      dns_servers                            = var.dns_servers
      resource_group_name                    = var.resource_group_name
      vwan_associated_routetable_resource_id = ""
      vwan_security_configuration = {
        secure_internet_traffic = true
        secure_private_traffic  = true
      }
      vwan_connection_enabled = true
      vwan_hub_resource_id    = var.vwan_hub_id
      resource_group_tags     = var.subscription_tags
      tags                    = var.subscription_tags
    }
  }

  role_assignment_enabled = true
  role_assignments ={
    app_sub_owner = {
      principal_id   = var.owner_object_id
      definition     = "Owner"
      relative_scope = ""
    }
  }

  disable_telemetry = true
}

Screenshots

Additional context

matt-FFFFFF commented 1 year ago

Hi! Thanks for raising. This was fixed in 3.4, please update to the latest (3.4.1)

VenkateshJayachandra commented 11 months ago

Hi! Thanks for raising. This was fixed in 3.4, please update to the latest (3.4.1)

Strangely with the latest 3.4.1 we are facing this issue.

However, when running it locally it works fine as the module creates all the resources as expected on the first run and this happens only on the runners (self-hosted agent - GitHub).

Initially we thought just the azapi_action is causing the issue but there are few other azapi_action calls which are working fine.

We fixed with the below workaround which is creating the subscription first and then registering the providers finally using the vending module.

locals {
  providerList = "Microsoft.Compute, Microsoft.Network"
}
resource "null_resource" "es_register_provider_and_wait_for_state" {
  triggers = {
    providerList = sha1(local.providerList)
  }

  provisioner "local-exec" {

    command     = <<-EOT

        az login --service-principal --username $env:ARM_CLIENT_ID --password $env:ARM_CLIENT_SECRET --tenant $env:ARM_TENANT_ID --output none

        az account set -s "${azurerm_subscription.workload.subscription_id}"        

        $providersToRegister = '${local.providerList}' -split ',' | ForEach-Object { $_.Trim() }

        foreach ($provider in $providersToRegister) {

            $registrationState = (az provider show --namespace $provider | ConvertFrom-Json).registrationState 

            if ($registrationState -eq "Registered") {
                Write-Host "##[section] ($provider) already registered..." -ForegroundColor Green
            }
            else {
                az provider register --namespace $provider
                Write-Host "##[section] ($provider) registering..." -ForegroundColor Green

                $found = "Registering.."
                $breakafterMultipleRetry = 1 

                while ("Registered" -ne $found) {
                    Write-Host  "##[section]  Waiting for the ($provider) registration to complete"
                    Start-Sleep 30

                    $found = (az provider show --namespace $provider | ConvertFrom-Json).registrationState 
                    if ($breakafterMultipleRetry -eq 10) { 
                        Write-Host "##[error] Unable to register provider ($provider) with the newly vended subscription"
                        break
                    }
                    $breakafterMultipleRetry++
                }

                Write-Host "##[section] ($provider) $found" -ForegroundColor Green
            }
        }

    EOT
    interpreter = ["pwsh", "-Command"]
  }

  depends_on = [azurerm_subscription.workload
  ]
}

Below are the plugins and their versions.

Downloading registry.terraform.io/Azure/lz-vending/azurerm 3.4.1 for lz_vending...
- lz_vending in .terraform/modules/lz_vending
- lz_vending.resourcegroup in .terraform/modules/lz_vending/modules/resourcegroup
- lz_vending.resourcegroup_networkwatcherrg in .terraform/modules/lz_vending/modules/resourcegroup
- lz_vending.resourceproviders in .terraform/modules/lz_vending/modules/resourceprovider
- lz_vending.roleassignment in .terraform/modules/lz_vending/modules/roleassignment
- lz_vending.roleassignment_umi in .terraform/modules/lz_vending/modules/roleassignment
- lz_vending.subscription in .terraform/modules/lz_vending/modules/subscription
- lz_vending.usermanagedidentity in .terraform/modules/lz_vending/modules/usermanagedidentity
- lz_vending.virtualnetwork in .terraform/modules/lz_vending/modules/virtualnetwork

Initializing provider plugins...
- Finding hashicorp/null versions matching ">= 3.0.0"...
- Finding hashicorp/azurerm versions matching ">= 3.7.0, >= 3.62.0"...
- Finding hashicorp/time versions matching ">= 0.9.0"...
- Finding azure/azapi versions matching ">= 1.3.0, >= 1.4.0, >= 1.8.0"...
- Installing hashicorp/null v3.2.1...
- Installed hashicorp/null v3.2.1 (signed by HashiCorp)
- Installing hashicorp/azurerm v3.79.0...
- Installed hashicorp/azurerm v3.79.0 (signed by HashiCorp)
- Installing hashicorp/time v0.9.1...
- Installed hashicorp/time v0.9.1 (signed by HashiCorp)
- Installing azure/azapi v1.10.0...
- Installed azure/azapi v1.10.0 (signed by a HashiCorp partner, key ID *****)

Any insights would be helpful.

Azure / terraform-azurerm-lz-vending