aztfmod / terraform-azurerm-caf

Terraform supermodule for the Terraform platform engineering for Azure
https://aztfmod.github.io/documentation/
MIT License
551 stars 692 forks source link

Bug report-example vng 101 (BGP config peering_address invalid) #1736

Open sjackson0109 opened 1 year ago

sjackson0109 commented 1 year ago

Objective

Migrate my customers LZ configuration from IaC to CaC, using the terraform, with the aztfmod module.

Background

The entire enterprise scale "wheel" has been built. Vnet peering is rejected when advertising 'gateway' across to the peer (guess MS's API checks for the presence of a VNG appliance). So I need to include the VNG appliance, including all it's desired configuration.

Issue

Using the VNG appliance, with the AZ SKU (2x public vpn endpoints), I cannot get the bgp_settings block to work.

This link provides an example to follow; but even that code is incorrect. Specifically peering_addresses block is not found.

Doing my own reading

I have read the code block that actually creates the azurerm_virtual_network_gateway resource, looking at the bgp_settings block between lines 137-151 of this file. Note the presence of peering_addresses, not peering address, confirming the example is definitely invalid! image

Products:

Terraform Configuration Files

virtual_network_gateways = {
  uks_vng_test = {
    name                       = "ukw-hub-test"
    resource_group_key         = "hub"
    type                       = "Vpn"
    sku                        = "VpnGw2AZ"
    generation                 = "Generation2"
    vpn_type                   = "RouteBased"
    active_active              = true
    private_ip_address_enabled = true
    ip_configuration = {
      ipconfig1 = {
        ipconfig_name                 = "ukw-hub-vng-pip-primary-config"
        public_ip_address_key         = "uks_hub_vng_pip_primary"
        vnet_key                      = "uks_hub"
        private_ip_address_allocation = "Dynamic"
      }
      ipconfig2 = { 
        ipconfig_name                 = "ukw-hub-vng-pip-secondary-config"
        public_ip_address_key         = "uks_hub_vng_pip_secondary"
        vnet_key                      = "ukw_hub"
        private_ip_address_allocation = "Dynamic"
      }
    }
    ## P2P VPN CONFIG
    enable_bgp = true
    bgp_settings = {
      bpgsettings1 = {
        asn         = 65099
        peer_weight = 100
        peering_addresses = { 
          vti_primary = {
            ip_configuration_name = "ukw-hub-vng-vti-primary"
            apipa_addresses       = ["169.254.21.1", "169.254.21.5", "169.254.21.9"]
          }
          vti_secondary = {
            ip_configuration_name = "ukw-hub-vng-vti-secondary"
            apipa_addresses       = ["169.254.22.1", "169.254.22.5", "169.254.22.9"]
          }
        }
      }
    }
  }
}

Expected Behaviour

Expect a VNG appliance to build with BGP configuation applied.

Actual Behaviour

No BGP configuration was saved, as there needs to be a minimum of one peering_address for the configuration to save.

│ Error: Creating/Updating Virtual Network Gateway: (Name "vgw-ukw-hub-test" / Resource Group "HUB"): network.VirtualNetworkGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="IpconfigurationIdMustbeInGatewayIpconfigurations" Message="All IpconfigurationIds in BgpPeeringAddresses must belong to gateway /subscriptions/<REDACTED>/resourceGroups/HUB/providers/Microsoft.Network/virtualNetworkGateways/vgw-ukw-hub-test IpConfigurations." Details=[]
│
│   with module.aztfmod.module.virtual_network_gateways["uks_hub_test"].azurerm_virtual_network_gateway.vngw,
│   on .terraform\modules\aztfmod\modules\networking\virtual_network_gateways\module.tf line 11, in resource "azurerm_virtual_network_gateway" "vngw":
│   11: resource "azurerm_virtual_network_gateway" "vngw" {
sjackson0109 commented 1 year ago

OK, so I can build the VNG now... without BGP APIPA addressing..

virtual_network_gateways = {
  uks_hub_test = {
    name                       = "ukw-hub-test"
    resource_group_key         = "hub"
    type                       = "Vpn"
    sku                        = "VpnGw2AZ"
    generation                 = "Generation2"
    vpn_type                   = "RouteBased"
    active_active              = true
    private_ip_address_enabled = false #tried false and true here, no difference.
    ip_configuration = {
      # NOTE: number of public IPs depends:
      #  No matter what SKU you have, you NEED 1x PIP address
      #  Is it an Active/Active appliance? Yes: 1x additional PIP required for the second unit
      #  Is this an SSLVPN Endpoint? Yes: 1x additional PIP required (ha)
      ipconfig1 = {
        ipconfig_name                 = "ukw-hub-vng2-pip-primary-config"
        public_ip_address_key         = "ukw_hub_vng2_pip_primary"
        vnet_key                      = "ukw_hub"
        private_ip_address_allocation = "Dynamic"
      }
      ipconfig2 = {
        ipconfig_name                 = "ukw-hub-vng2-pip-secondary-config"
        public_ip_address_key         = "ukw_hub_vng2_pip_secondary"
        vnet_key                      = "ukw_hub"
        private_ip_address_allocation = "Dynamic"
      }
    }
    ## P2P VPN CONFIG
    enable_bgp = true
    bgp_settings = {
      config = {
        asn         = 65099
        peer_weight = 100
        ## Virtual-Tunnel-Interfaces (VTIs)
        # peering_addresses = { 
        #   vti_primary = {
        #     ip_configuration_name = "ukw-hub-vng-vti-primary"
        #     apipa_addresses       = ["169.254.21.1"]
        #   }
        #   vti_secondary = {
        #     ip_configuration_name = "ukw-hub-vng-vti-secondary"
        #     apipa_addresses       = ["169.254.22.1"]
        #   }
        # }
      }
    }
  }
}

image

But when I un-comment by peering_addresses map, it fails to build with another error:

│ Error: Creating/Updating Virtual Network Gateway: (Name "vgw-ukw-hub-test" / Resource Group "HUB"): 
│  network.VirtualNetworkGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: 
│  Code="BgpPeeringAddressesCountNotCorrect" Message="The BgpPeeringAddresses count is incorrect for gateway 
│  /subscriptions/REDACTED/resourceGroups/HUB/providers/Microsoft.Network/virtualNetworkGateways/vgw-ukw-hub-test.
│  It should be 1 for ActivePassive and 2 for ActiveActive gateway." Details=[]
sjackson0109 commented 1 year ago

@LaurentLesle - Sorry for the tag. I see you were the last person to contribute to the VNG module bgp_settings dynamic block.

Could you possibly offer some advice on getting past this problem? Is it a bug in the aztfmod example? and can you support me getting across the fence with working bgp config?

sjackson0109 commented 1 year ago

Now with this block:

    enable_bgp = true
    bgp_settings = {
      config = {
        asn         = 65099
        peer_weight = 100

        # Virtual-Tunnel-Interfaces (VTIs)
        peering_addresses = {
          primary = {
            ip_configuration_name = "vti-0"
            apipa_addresses       = ["169.254.21.1","169.254.21.5"]
          }
          secondary = {
            ip_configuration_name = "vti-1"
            apipa_addresses       = ["169.254.22.1","169.254.22.5"]
          }
        }
    }

I now get this error:

│ Error: Creating/Updating Virtual Network Gateway: (Name "vgw-ukw-hub-test" / Resource Group "HUB"): 
│  network.VirtualNetworkGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=400
│  -- Original Error: Code="IpconfigurationIdMustbeInGatewayIpconfigurations" 
│     Message="All IpconfigurationIds in BgpPeeringAddresses must belong to gateway
│     /subscriptions/REDACTED/resourceGroups/HUB/providers/Microsoft.Network/virtualNetworkGateways/vgw-ukw-hub-test IpConfigurations."
│     Details=[]
sjackson0109 commented 1 year ago

OK, so having MANUALLY added BGP APIPA addresses via the azure portal, and then un-commenting this the bgp_settings block inside my variables - terraform plan tells me the changes it's intending on making:

~ bgp_settings {
            # (2 unchanged attributes hidden)

          ~ peering_addresses {
              ~ apipa_addresses       = [
                  - "169.254.21.65"
                  - "169.254.21.69",
                  + "169.254.21.1",
                  + "169.254.21.5",
                ]
                # (3 unchanged attributes hidden)
            }
          ~ peering_addresses {
              ~ apipa_addresses       = [
                  - "169.254.22.65",
                  - "169.254.22.69",
                  + "169.254.22.1",
                  + "169.254.22.5",
                ]
                # (3 unchanged attributes hidden)
            }
        } 

Terraform apply > passes fine.

Terraform destroy > cleans up perfectly

Terraform apply again > same error: "The BgpPeeringAddresses count is incorrect for gateway . It should be 1 for ActivePassive and 2 for ActiveActive gateway."

sjackson0109 commented 1 year ago

I found the terraform DEBUG entity:

PUT /subscriptions/{SUBSCRIPTION_GUID}/resourceGroups/{RG_NAME}/providers/Microsoft.Network/virtualNetworkGateways/gs-vgw-ukw-hub?api-version=2022-07-01 HTTP/1.1
Host: management.azure.com
User-Agent: Go/go1.19.3 (amd64-windows) go-autorest/v14.2.1 tombuildsstuff/kermit/v0.20230424.1090808 network/2022-07-01 HashiCorp Terraform/1.4.6 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/dev pid-222c6c49-1b0a-5959-a213-6608f9eb8820
Content-Length: 2819
Content-Type: application/json; charset=utf-8
X-Ms-Correlation-Request-Id: bb030160-555b-44dd-11ea-a996c8855c4e
Accept-Encoding: gzip

{"location":"ukwest","properties":{"activeActive":true,"bgpSettings":{"asn":65030,"peerWeight":200,"bgpPeeringAddresses":[{"customBgpIpAddresses":["169.254.21.65","169.254.21.69","169.254.21.73","169.254.21.77"],"ipconfigurationId":"/subscriptions/{SUBSCRIPTION_GUID}/resourceGroups/{RG_NAME}/providers/Microsoft.Network/virtualNetworkGateways/{VNG_NAME}/ipConfigurations/gatewayIp1"},{"customBgpIpAddresses":["169.254.22.65","169.254.22.69","169.254.22.73","169.254.22.77"],"ipconfigurationId":"/subscriptions/{SUBSCRIPTION_GUID}/resourceGroups/{RG_NAME}/providers/Microsoft.Network/virtualNetworkGateways/{VNG_NAME}/ipConfigurations/gatewayIp2"}]},"customRoutes":{"addressPrefixes":["10.101.28.0/24","10.20.240.0/23","10.30.240.0/23"]},"enableBgp":false,"enablePrivateIpAddress":false,"gatewayType":"Vpn","ipConfigurations":[{"name":"ukw-hub-vng-pip-primary-config","properties":{"privateIPAllocationMethod":"Dynamic","publicIPAddress":{"id":"/subscriptions/{SUBSCRIPTION_GUID}/resourceGroups/{RG_NAME}/providers/Microsoft.Network/publicIPAddresses/gs-pip-ukw-hub-vng-pri"},"subnet":{"id":"/subscriptions/{SUBSCRIPTION_GUID}/resourceGroups/{RG_NAME}/providers/Microsoft.Network/virtualNetworks/gs-vnet-ukw-hub/subnets/GatewaySubnet"}}},{"name":"ukw-hub-vng-pip-secondary-config","properties":{"privateIPAllocationMethod":"Dynamic","publicIPAddress":{"id":"/subscriptions/{SUBSCRIPTION_GUID}/resourceGroups/{RG_NAME}/providers/Microsoft.Network/publicIPAddresses/gs-pip-ukw-hub-vng-sec"},"subnet":{"id":"/subscriptions/{SUBSCRIPTION_GUID}/resourceGroups/{RG_NAME}/providers/Microsoft.Network/virtualNetworks/gs-vnet-ukw-hub/subnets/GatewaySubnet"}}},{"name":"ukw-hub-vng-pip-ha-config","properties":{"privateIPAllocationMethod":"Dynamic","publicIPAddress":{"id":"/subscriptions/{SUBSCRIPTION_GUID}/resourceGroups/{RG_NAME}/providers/Microsoft.Network/publicIPAddresses/gs-pip-ukw-hub-vng-ha"},"subnet":{"id":"/subscriptions/{SUBSCRIPTION_GUID}/resourceGroups/{RG_NAME}/providers/Microsoft.Network/virtualNetworks/gs-vnet-ukw-hub/subnets/GatewaySubnet"}}}],"sku":{"name":"VpnGw2","tier":"VpnGw2"},"vpnClientConfiguration":{"vpnClientAddressPool":{"addressPrefixes":["10.30.240.0/23"]},"vpnClientRootCertificates":null,"vpnClientRevokedCertificates":null,"vpnClientProtocols":["OpenVPN"],"vpnAuthenticationTypes":["AAD"],"radiusServerAddress":"","radiusServerSecret":"","aadTenant":"https://login.microsoftonline.com/1167dbde-5cbd-4f60-8d9e-a6c81c8e8db8/","aadAudience":"41b23e61-6c1e-4545-b367-cd054e0ed4b4","aadIssuer":"https://sts.windows.net/1167dbde-5cbd-4f60-8d9e-a6c81c8e8db8/"},"vpnGatewayGeneration":"Generation2","vpnType":"RouteBased"},"tags":{"module":"virtual_network_gateways"}}
2023-08-14T14:05:50.852+0100 [DEBUG] provider.terraform-provider-azurerm_v3.56.0_x5.exe: AzureRM Response for https://management.azure.com/subscriptions/{SUBSCRIPTION_GUID}/resourceGroups/{RG_NAME}/providers/Microsoft.Network/virtualNetworkGateways/{VNG_NAME}?api-version=2022-07-01: 
HTTP/2.0 400 Bad Request
Content-Length: 374
Cache-Control: no-cache
Content-Type: application/json; charset=utf-8
Date: Mon, 14 Aug 2023 13:05:50 GMT
Expires: -1
Pragma: no-cache
Server: Microsoft-HTTPAPI/2.0
Server: Microsoft-HTTPAPI/2.0
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-Ms-Arm-Service-Request-Id: 1a7fff65-8d37-47d0-afc4-f99aa8b897c1
X-Ms-Correlation-Request-Id: bb030160-555b-44dd-11ea-a996c8855c4e
X-Ms-Ratelimit-Remaining-Subscription-Writes: 1198
X-Ms-Request-Id: 7744dbca-ab9c-4718-9fff-02221d7c0c08
X-Ms-Routing-Request-Id: UKSOUTH:20230814T130550Z:93264bb0-cea7-4e84-85e0-b19967ed4fa5

{
  "error": {
    "code": "BgpPeeringAddressesCountNotCorrect",
    "message": "The BgpPeeringAddresses count is incorrect for gateway /subscriptions/{SUBSCRIPTION_GUID}/resourceGroups/{RG_NAME}/providers/Microsoft.Network/virtualNetworkGateways/{VNG_NAME}. It should be 1 for ActivePassive and 2 for ActiveActive gateway.",
    "details": []
  }
}: timestamp=2023-08-14T14:05:50.852+0100

Starting to look like it's Mirosoft's API that is causing this issue... image

sjackson0109 commented 1 year ago

OK, with varying degrees of success. I've managed SUCCESSFULLY the following config. Note the peer_weight value resolved the issue with Microsoft's API's accepted values for peering_addresses blocks.

virtual_network_gateways = {
  ukw_hub = {
    name                       = "ukw-hub"
    resource_group_key         = "hub"
    region                     = "ukw"
    type                       = "Vpn"
    sku                        = "VpnGw2"
    generation                 = "Generation2"
    vpn_type                   = "RouteBased"
    active_active              = true
    zones                      = null  #Unsupported in UKWEST
    private_ip_address_enabled = false #Use only APIPA Addressing for VTI interfaces.  Unsupported without AZ SKU too!
    ip_configuration = {
      # NOTE: number of public IPs depends:
      #  No matter what SKU you have, you NEED 1x PIP address
      #  Is it an Active/Active appliance? Yes: 1x additional PIP required for the second unit
      #  Is this an SSLVPN Endpoint? Yes: 1x additional PIP required (ha)
      ipconfig1 = {
        ipconfig_name                    = "ukw-vng-pip-primary-config"
        public_ip_address_key        = "ukw_vng_pip_primary"
        vnet_key                             = "ukw_hub"
        private_ip_address_allocation = "Dynamic"
      }
      ipconfig2 = {
        ipconfig_name                    = "ukw-vng-pip-secondary-config"
        public_ip_address_key        = "ukw_vng_pip_secondary"
        vnet_key                             = "ukw_hub"
        private_ip_address_allocation = "Dynamic"
      }
      ipconfig3 = {
        ipconfig_name                    = "ukw-vng-pip-ha-config"
        public_ip_address_key        = "ukw_vng_pip_ha"
        vnet_key                             = "ukw_hub"
        private_ip_address_allocation = "Dynamic"
      }
    }
    # SSL VPN CLIENT CONFIG
    vpn_client_configuration = {
      openvpn_aad = {
        address_space        = ["10.1.10.0/24"]
        vpn_auth_types       = ["AAD"]
        vpn_client_protocols = ["OpenVPN"]
        aad_audience         = "{GUID REMOVED}"                                    #Azure VPN (Application ID)
        aad_issuer           = "https://sts.windows.net/${data.azurerm_client_config.default.tenant_id}/"           #Tenant STS Endpoint
        aad_tenant           = "https://login.microsoftonline.com/${data.azurerm_client_config.default.tenant_id}/" #Tenant Authentication Endpoint
      }
    }
    custom_route = {
      vpn_routes = {
        address_prefixes = ["10.20.240.0/23", "10.30.240.0/23", "10.101.28.0/24"]
      }
    }
enable_bgp = true
    bgp_settings = {
      routing = {
        asn         = 65001
        peer_weight = 0 #When specifying multiple peering_addresses, this value MUST be 0.
        # Virtual-Tunnel-Interfaces (VTIs)
        # Active/Passive => only a vti_primary is required
        # Active/Active => both vti_primary and vti_secondary are required
        peering_addresses = {
          vti_primary = {
            ip_configuration_name = "gatewayIp1"
            apipa_addresses       = ["169.254.21.1", "169.254.21.5", "169.254.21.9"]
          }
          vti_secondary = {
            ip_configuration_name = "gatewayIp2"
            apipa_addresses       = ["169.254.22.1", "169.254.22.5", "169.254.22.9"]
          }
        }
      }
    }
  }
 }