Open sjackson0109 opened 1 year ago
OK, so I can build the VNG now... without BGP APIPA addressing..
virtual_network_gateways = {
uks_hub_test = {
name = "ukw-hub-test"
resource_group_key = "hub"
type = "Vpn"
sku = "VpnGw2AZ"
generation = "Generation2"
vpn_type = "RouteBased"
active_active = true
private_ip_address_enabled = false #tried false and true here, no difference.
ip_configuration = {
# NOTE: number of public IPs depends:
# No matter what SKU you have, you NEED 1x PIP address
# Is it an Active/Active appliance? Yes: 1x additional PIP required for the second unit
# Is this an SSLVPN Endpoint? Yes: 1x additional PIP required (ha)
ipconfig1 = {
ipconfig_name = "ukw-hub-vng2-pip-primary-config"
public_ip_address_key = "ukw_hub_vng2_pip_primary"
vnet_key = "ukw_hub"
private_ip_address_allocation = "Dynamic"
}
ipconfig2 = {
ipconfig_name = "ukw-hub-vng2-pip-secondary-config"
public_ip_address_key = "ukw_hub_vng2_pip_secondary"
vnet_key = "ukw_hub"
private_ip_address_allocation = "Dynamic"
}
}
## P2P VPN CONFIG
enable_bgp = true
bgp_settings = {
config = {
asn = 65099
peer_weight = 100
## Virtual-Tunnel-Interfaces (VTIs)
# peering_addresses = {
# vti_primary = {
# ip_configuration_name = "ukw-hub-vng-vti-primary"
# apipa_addresses = ["169.254.21.1"]
# }
# vti_secondary = {
# ip_configuration_name = "ukw-hub-vng-vti-secondary"
# apipa_addresses = ["169.254.22.1"]
# }
# }
}
}
}
}
But when I un-comment by peering_addresses
map, it fails to build with another error:
│ Error: Creating/Updating Virtual Network Gateway: (Name "vgw-ukw-hub-test" / Resource Group "HUB"):
│ network.VirtualNetworkGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error:
│ Code="BgpPeeringAddressesCountNotCorrect" Message="The BgpPeeringAddresses count is incorrect for gateway
│ /subscriptions/REDACTED/resourceGroups/HUB/providers/Microsoft.Network/virtualNetworkGateways/vgw-ukw-hub-test.
│ It should be 1 for ActivePassive and 2 for ActiveActive gateway." Details=[]
@LaurentLesle - Sorry for the tag. I see you were the last person to contribute to the VNG module bgp_settings dynamic block.
Could you possibly offer some advice on getting past this problem? Is it a bug in the aztfmod example? and can you support me getting across the fence with working bgp config?
Now with this block:
enable_bgp = true
bgp_settings = {
config = {
asn = 65099
peer_weight = 100
# Virtual-Tunnel-Interfaces (VTIs)
peering_addresses = {
primary = {
ip_configuration_name = "vti-0"
apipa_addresses = ["169.254.21.1","169.254.21.5"]
}
secondary = {
ip_configuration_name = "vti-1"
apipa_addresses = ["169.254.22.1","169.254.22.5"]
}
}
}
I now get this error:
│ Error: Creating/Updating Virtual Network Gateway: (Name "vgw-ukw-hub-test" / Resource Group "HUB"):
│ network.VirtualNetworkGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=400
│ -- Original Error: Code="IpconfigurationIdMustbeInGatewayIpconfigurations"
│ Message="All IpconfigurationIds in BgpPeeringAddresses must belong to gateway
│ /subscriptions/REDACTED/resourceGroups/HUB/providers/Microsoft.Network/virtualNetworkGateways/vgw-ukw-hub-test IpConfigurations."
│ Details=[]
OK,
so having MANUALLY added BGP APIPA addresses via the azure portal, and then un-commenting this the bgp_settings
block inside my variables - terraform plan tells me the changes it's intending on making:
~ bgp_settings {
# (2 unchanged attributes hidden)
~ peering_addresses {
~ apipa_addresses = [
- "169.254.21.65"
- "169.254.21.69",
+ "169.254.21.1",
+ "169.254.21.5",
]
# (3 unchanged attributes hidden)
}
~ peering_addresses {
~ apipa_addresses = [
- "169.254.22.65",
- "169.254.22.69",
+ "169.254.22.1",
+ "169.254.22.5",
]
# (3 unchanged attributes hidden)
}
}
Terraform apply > passes fine.
Terraform destroy > cleans up perfectly
Terraform apply again > same error:
"The BgpPeeringAddresses count is incorrect for gateway
I found the terraform DEBUG entity:
PUT /subscriptions/{SUBSCRIPTION_GUID}/resourceGroups/{RG_NAME}/providers/Microsoft.Network/virtualNetworkGateways/gs-vgw-ukw-hub?api-version=2022-07-01 HTTP/1.1
Host: management.azure.com
User-Agent: Go/go1.19.3 (amd64-windows) go-autorest/v14.2.1 tombuildsstuff/kermit/v0.20230424.1090808 network/2022-07-01 HashiCorp Terraform/1.4.6 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/dev pid-222c6c49-1b0a-5959-a213-6608f9eb8820
Content-Length: 2819
Content-Type: application/json; charset=utf-8
X-Ms-Correlation-Request-Id: bb030160-555b-44dd-11ea-a996c8855c4e
Accept-Encoding: gzip
{"location":"ukwest","properties":{"activeActive":true,"bgpSettings":{"asn":65030,"peerWeight":200,"bgpPeeringAddresses":[{"customBgpIpAddresses":["169.254.21.65","169.254.21.69","169.254.21.73","169.254.21.77"],"ipconfigurationId":"/subscriptions/{SUBSCRIPTION_GUID}/resourceGroups/{RG_NAME}/providers/Microsoft.Network/virtualNetworkGateways/{VNG_NAME}/ipConfigurations/gatewayIp1"},{"customBgpIpAddresses":["169.254.22.65","169.254.22.69","169.254.22.73","169.254.22.77"],"ipconfigurationId":"/subscriptions/{SUBSCRIPTION_GUID}/resourceGroups/{RG_NAME}/providers/Microsoft.Network/virtualNetworkGateways/{VNG_NAME}/ipConfigurations/gatewayIp2"}]},"customRoutes":{"addressPrefixes":["10.101.28.0/24","10.20.240.0/23","10.30.240.0/23"]},"enableBgp":false,"enablePrivateIpAddress":false,"gatewayType":"Vpn","ipConfigurations":[{"name":"ukw-hub-vng-pip-primary-config","properties":{"privateIPAllocationMethod":"Dynamic","publicIPAddress":{"id":"/subscriptions/{SUBSCRIPTION_GUID}/resourceGroups/{RG_NAME}/providers/Microsoft.Network/publicIPAddresses/gs-pip-ukw-hub-vng-pri"},"subnet":{"id":"/subscriptions/{SUBSCRIPTION_GUID}/resourceGroups/{RG_NAME}/providers/Microsoft.Network/virtualNetworks/gs-vnet-ukw-hub/subnets/GatewaySubnet"}}},{"name":"ukw-hub-vng-pip-secondary-config","properties":{"privateIPAllocationMethod":"Dynamic","publicIPAddress":{"id":"/subscriptions/{SUBSCRIPTION_GUID}/resourceGroups/{RG_NAME}/providers/Microsoft.Network/publicIPAddresses/gs-pip-ukw-hub-vng-sec"},"subnet":{"id":"/subscriptions/{SUBSCRIPTION_GUID}/resourceGroups/{RG_NAME}/providers/Microsoft.Network/virtualNetworks/gs-vnet-ukw-hub/subnets/GatewaySubnet"}}},{"name":"ukw-hub-vng-pip-ha-config","properties":{"privateIPAllocationMethod":"Dynamic","publicIPAddress":{"id":"/subscriptions/{SUBSCRIPTION_GUID}/resourceGroups/{RG_NAME}/providers/Microsoft.Network/publicIPAddresses/gs-pip-ukw-hub-vng-ha"},"subnet":{"id":"/subscriptions/{SUBSCRIPTION_GUID}/resourceGroups/{RG_NAME}/providers/Microsoft.Network/virtualNetworks/gs-vnet-ukw-hub/subnets/GatewaySubnet"}}}],"sku":{"name":"VpnGw2","tier":"VpnGw2"},"vpnClientConfiguration":{"vpnClientAddressPool":{"addressPrefixes":["10.30.240.0/23"]},"vpnClientRootCertificates":null,"vpnClientRevokedCertificates":null,"vpnClientProtocols":["OpenVPN"],"vpnAuthenticationTypes":["AAD"],"radiusServerAddress":"","radiusServerSecret":"","aadTenant":"https://login.microsoftonline.com/1167dbde-5cbd-4f60-8d9e-a6c81c8e8db8/","aadAudience":"41b23e61-6c1e-4545-b367-cd054e0ed4b4","aadIssuer":"https://sts.windows.net/1167dbde-5cbd-4f60-8d9e-a6c81c8e8db8/"},"vpnGatewayGeneration":"Generation2","vpnType":"RouteBased"},"tags":{"module":"virtual_network_gateways"}}
2023-08-14T14:05:50.852+0100 [DEBUG] provider.terraform-provider-azurerm_v3.56.0_x5.exe: AzureRM Response for https://management.azure.com/subscriptions/{SUBSCRIPTION_GUID}/resourceGroups/{RG_NAME}/providers/Microsoft.Network/virtualNetworkGateways/{VNG_NAME}?api-version=2022-07-01:
HTTP/2.0 400 Bad Request
Content-Length: 374
Cache-Control: no-cache
Content-Type: application/json; charset=utf-8
Date: Mon, 14 Aug 2023 13:05:50 GMT
Expires: -1
Pragma: no-cache
Server: Microsoft-HTTPAPI/2.0
Server: Microsoft-HTTPAPI/2.0
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-Ms-Arm-Service-Request-Id: 1a7fff65-8d37-47d0-afc4-f99aa8b897c1
X-Ms-Correlation-Request-Id: bb030160-555b-44dd-11ea-a996c8855c4e
X-Ms-Ratelimit-Remaining-Subscription-Writes: 1198
X-Ms-Request-Id: 7744dbca-ab9c-4718-9fff-02221d7c0c08
X-Ms-Routing-Request-Id: UKSOUTH:20230814T130550Z:93264bb0-cea7-4e84-85e0-b19967ed4fa5
{
"error": {
"code": "BgpPeeringAddressesCountNotCorrect",
"message": "The BgpPeeringAddresses count is incorrect for gateway /subscriptions/{SUBSCRIPTION_GUID}/resourceGroups/{RG_NAME}/providers/Microsoft.Network/virtualNetworkGateways/{VNG_NAME}. It should be 1 for ActivePassive and 2 for ActiveActive gateway.",
"details": []
}
}: timestamp=2023-08-14T14:05:50.852+0100
Starting to look like it's Mirosoft's API that is causing this issue...
OK, with varying degrees of success. I've managed SUCCESSFULLY the following config. Note the peer_weight
value resolved the issue with Microsoft's API's accepted values for peering_addresses blocks.
virtual_network_gateways = {
ukw_hub = {
name = "ukw-hub"
resource_group_key = "hub"
region = "ukw"
type = "Vpn"
sku = "VpnGw2"
generation = "Generation2"
vpn_type = "RouteBased"
active_active = true
zones = null #Unsupported in UKWEST
private_ip_address_enabled = false #Use only APIPA Addressing for VTI interfaces. Unsupported without AZ SKU too!
ip_configuration = {
# NOTE: number of public IPs depends:
# No matter what SKU you have, you NEED 1x PIP address
# Is it an Active/Active appliance? Yes: 1x additional PIP required for the second unit
# Is this an SSLVPN Endpoint? Yes: 1x additional PIP required (ha)
ipconfig1 = {
ipconfig_name = "ukw-vng-pip-primary-config"
public_ip_address_key = "ukw_vng_pip_primary"
vnet_key = "ukw_hub"
private_ip_address_allocation = "Dynamic"
}
ipconfig2 = {
ipconfig_name = "ukw-vng-pip-secondary-config"
public_ip_address_key = "ukw_vng_pip_secondary"
vnet_key = "ukw_hub"
private_ip_address_allocation = "Dynamic"
}
ipconfig3 = {
ipconfig_name = "ukw-vng-pip-ha-config"
public_ip_address_key = "ukw_vng_pip_ha"
vnet_key = "ukw_hub"
private_ip_address_allocation = "Dynamic"
}
}
# SSL VPN CLIENT CONFIG
vpn_client_configuration = {
openvpn_aad = {
address_space = ["10.1.10.0/24"]
vpn_auth_types = ["AAD"]
vpn_client_protocols = ["OpenVPN"]
aad_audience = "{GUID REMOVED}" #Azure VPN (Application ID)
aad_issuer = "https://sts.windows.net/${data.azurerm_client_config.default.tenant_id}/" #Tenant STS Endpoint
aad_tenant = "https://login.microsoftonline.com/${data.azurerm_client_config.default.tenant_id}/" #Tenant Authentication Endpoint
}
}
custom_route = {
vpn_routes = {
address_prefixes = ["10.20.240.0/23", "10.30.240.0/23", "10.101.28.0/24"]
}
}
enable_bgp = true
bgp_settings = {
routing = {
asn = 65001
peer_weight = 0 #When specifying multiple peering_addresses, this value MUST be 0.
# Virtual-Tunnel-Interfaces (VTIs)
# Active/Passive => only a vti_primary is required
# Active/Active => both vti_primary and vti_secondary are required
peering_addresses = {
vti_primary = {
ip_configuration_name = "gatewayIp1"
apipa_addresses = ["169.254.21.1", "169.254.21.5", "169.254.21.9"]
}
vti_secondary = {
ip_configuration_name = "gatewayIp2"
apipa_addresses = ["169.254.22.1", "169.254.22.5", "169.254.22.9"]
}
}
}
}
}
}
Objective
Migrate my customers LZ configuration from IaC to CaC, using the terraform, with the aztfmod module.
Background
The entire enterprise scale "wheel" has been built. Vnet peering is rejected when advertising 'gateway' across to the peer (guess MS's API checks for the presence of a VNG appliance). So I need to include the VNG appliance, including all it's desired configuration.
Issue
Using the VNG appliance, with the AZ SKU (2x public vpn endpoints), I cannot get the
bgp_settings
block to work.This link provides an example to follow; but even that code is incorrect. Specifically
peering_addresses block is not found
.Doing my own reading
I have read the code block that actually creates the azurerm_virtual_network_gateway resource, looking at the![image](https://user-images.githubusercontent.com/38080190/258368937-1df66db7-a66f-49be-aa10-9080ffc6ae5a.png)
bgp_settings
block between lines 137-151 of this file. Note the presence ofpeering_addresses
, notpeering address
, confirming the example is definitely invalid!Products:
Terraform Configuration Files
Expected Behaviour
Expect a VNG appliance to build with BGP configuation applied.
Actual Behaviour
No BGP configuration was saved, as there needs to be a minimum of one peering_address for the configuration to save.