search

hashicorp / terraform-provider-azurerm

Terraform provider for Azure Resource Manager
https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs
Mozilla Public License 2.0
4.53k stars 4.6k forks source link

Support for Custom BGP Addresses within VPN connection #15854

Closed miroslavngena closed 9 months ago

miroslavngena commented 2 years ago

Community Note

Description

Azure VPN connection supports Custom BGP Addresses. For this in Connection configuration "Enable Custom BGP Addresses" must be checked and primary and secondary BGP addresses can be configured. Example is described within guide:

https://docs.microsoft.com/sk-sk/azure/vpn-gateway/vpn-gateway-howto-aws-bgp

New or Affected Resource(s)

Potential Terraform Configuration

resource "azurerm_virtual_network_gateway_connection" "azure_connection" {
  name                = "azure_connection"
  location            = "France Central"
  resource_group_name = "rg_name"

  type                       = "IPsec"
  virtual_network_gateway_id = azurerm_virtual_network_gateway.ha_gateway.id
  local_network_gateway_id   = azurerm_local_network_gateway.azure_local_network_gw.id

  shared_key = "shared_key"

  enable_bgp = true

  #enable_custom_bgp_addresses = true

}

References

DevOpsFu commented 2 years ago

@miroslavngena I agree that this support should be added. I have been working from the same MS documentation as yourself recently, trying to get AWS to Azure connectivity with BGP and an redundant connections.

FYI though - I found that the connections come up successfully with BGP even if you do not specify the custom BGP addresses on the azurerm_virtual_network_gateway_connection resources. I'm not sure why this is the case and have not done enough testing to verify if there are any problems with this configuration, but it does seem to be working fine so far. Maybe this will help you too.

miroslavngena commented 2 years ago

@DevOpsFu Yes, BGP is coming UP. The issue is AWS side is not receiving routes only from two neighbors. Two additional neighbors are sending to AWS routes with wrong next-hop value. This can be fixed with "enable_custom_bgp_addresses" and specifying the correct neighbors. Currently we are using REST API as a workaround.

DevOpsFu commented 2 years ago

@miroslavngena That's interesting, thanks for the info! Could you tell me how you found these two additional incorrect routes? I'm looking at the Transit Gateway route table and I see two routes for my Azure range, each route pointing at the VPN attachments as I would expect. AWS is not my strong subject though, so I may be missing something here.

miroslavngena commented 2 years ago

On AWS side there were learning 0 routes on that neighbors (BGP neighbors are up). On Azure side you can see it in BGP advertised routes output (VPN GW - BGP peers - neighbor IP - . . . - view advertised routes). Normally next-hop value should be from same subnet as peering. Affected peerings are sending routes with different next-hop.

ljtill commented 2 years ago

This configuration setting was added as part of a recent PR (#16631).

sjackson0109 commented 1 year ago

I think this open issue was resolved.

Last year I successfully got this built:

resource "azurerm_virtual_network_gateway" "vngw" {
  name                = var.gateway_name
  location            = azurerm_resource_group.uks_hub_rg.location
  resource_group_name = azurerm_resource_group.uks_hub_rg.name
  type       = "Vpn"
  sku        = "VpnGw2AZ"
  generation = "Generation2"
  active_active = true

  ## ETHERNET-INTERFACES
  ## TODO: Refactor to use an array of Public IPs (azurerm_public_ip.uks_vng_pip)
  ip_configuration {
    name                          = "${var.gateway_name}-IPConfig1"
    public_ip_address_id          = azurerm_public_ip.uks_vnet_gateway_pip_01.id
    private_ip_address_allocation = "Dynamic"
    subnet_id                     = azurerm_subnet.subnets["GatewaySubnet"].id
  }
  ip_configuration {
    name                          = "${var.gateway_name}-IPConfig2"
    public_ip_address_id          = azurerm_public_ip.uks_vnet_gateway_pip_02.id
    private_ip_address_allocation = "Dynamic"
    subnet_id                     = azurerm_subnet.subnets["GatewaySubnet"].id
  }
  ip_configuration {
    name                          = "${var.gateway_name}-IPConfig3"
    public_ip_address_id          = azurerm_public_ip.uks_vnet_gateway_pip_03.id
    private_ip_address_allocation = "Dynamic"
    subnet_id                     = azurerm_subnet.subnets["GatewaySubnet"].id
  }
  ## P2S VPN Config
  vpn_type = "RouteBased"
  vpn_client_configuration {
    vpn_client_protocols = ["OpenVPN"]
    vpn_auth_types       = ["AAD"]
    aad_audience = "REDACTED" #Azure VPN (Application ID)
    aad_issuer   = "https://sts.windows.net/REDACTED/"
    aad_tenant   = "https://login.microsoftonline.com/REDACTED/"
    address_space = ["${var.vng_pool}"]
  }
  #SSL VPN clients need to reach non-bgp propagated routes (aka reach inside the provider's VPN)
  custom_route {
    address_prefixes = ["${var.vng_pool}", "10.1.18.0/24"]
  }
  enable_bgp = true
  bgp_settings {
    asn         = var.gateway_asn
    peer_weight = 100

    ## TUNNEL-INTERFACES
    # - Group 1 = SITE 1
    # - Group 2 = SITE 2
    # - Group 3 = RESERVED: SITE 3
    # - Group 4 = PARTNER SITE
    peering_addresses {
      apipa_addresses       = ["169.254.21.33", "169.254.21.37", "169.254.21.41", "169.254.21.45"]
      ip_configuration_name = "${var.gateway_name}-IPConfig1"
    }
    peering_addresses {
      apipa_addresses       = ["169.254.22.33", "169.254.22.37", "169.254.22.41", "169.254.22.45"]
      ip_configuration_name = "${var.gateway_name}-IPConfig2"
    }
  }
}

Are we sure this issue is still OPEN?

eddiewhho commented 10 months ago

I think this open issue was resolved.

Last year I successfully got this built:

resource "azurerm_virtual_network_gateway" "vngw" {
  name                = var.gateway_name
  location            = azurerm_resource_group.uks_hub_rg.location
  resource_group_name = azurerm_resource_group.uks_hub_rg.name
  type       = "Vpn"
  sku        = "VpnGw2AZ"
  generation = "Generation2"
  active_active = true

  ## ETHERNET-INTERFACES
  ## TODO: Refactor to use an array of Public IPs (azurerm_public_ip.uks_vng_pip)
  ip_configuration {
    name                          = "${var.gateway_name}-IPConfig1"
    public_ip_address_id          = azurerm_public_ip.uks_vnet_gateway_pip_01.id
    private_ip_address_allocation = "Dynamic"
    subnet_id                     = azurerm_subnet.subnets["GatewaySubnet"].id
  }
  ip_configuration {
    name                          = "${var.gateway_name}-IPConfig2"
    public_ip_address_id          = azurerm_public_ip.uks_vnet_gateway_pip_02.id
    private_ip_address_allocation = "Dynamic"
    subnet_id                     = azurerm_subnet.subnets["GatewaySubnet"].id
  }
  ip_configuration {
    name                          = "${var.gateway_name}-IPConfig3"
    public_ip_address_id          = azurerm_public_ip.uks_vnet_gateway_pip_03.id
    private_ip_address_allocation = "Dynamic"
    subnet_id                     = azurerm_subnet.subnets["GatewaySubnet"].id
  }
  ## P2S VPN Config
  vpn_type = "RouteBased"
  vpn_client_configuration {
    vpn_client_protocols = ["OpenVPN"]
    vpn_auth_types       = ["AAD"]
    aad_audience = "REDACTED" #Azure VPN (Application ID)
    aad_issuer   = "https://sts.windows.net/REDACTED/"
    aad_tenant   = "https://login.microsoftonline.com/REDACTED/"
    address_space = ["${var.vng_pool}"]
  }
  #SSL VPN clients need to reach non-bgp propagated routes (aka reach inside the provider's VPN)
  custom_route {
    address_prefixes = ["${var.vng_pool}", "10.1.18.0/24"]
  }
  enable_bgp = true
  bgp_settings {
    asn         = var.gateway_asn
    peer_weight = 100

    ## TUNNEL-INTERFACES
    # - Group 1 = SITE 1
    # - Group 2 = SITE 2
    # - Group 3 = RESERVED: SITE 3
    # - Group 4 = PARTNER SITE
    peering_addresses {
      apipa_addresses       = ["169.254.21.33", "169.254.21.37", "169.254.21.41", "169.254.21.45"]
      ip_configuration_name = "${var.gateway_name}-IPConfig1"
    }
    peering_addresses {
      apipa_addresses       = ["169.254.22.33", "169.254.22.37", "169.254.22.41", "169.254.22.45"]
      ip_configuration_name = "${var.gateway_name}-IPConfig2"
    }
  }
}

Are we sure this issue is still OPEN?

that example you have is a different resource, you quoted "azurerm_virtual_network_gateway" but the current issue is for azurerm_virtual_network_gateway_connection

eddiewhho commented 10 months ago

``> This configuration setting was added as part of a recent PR (#16631).

there is a bug with that PR

Original Error: Code="GatewayCustomBgpAddressesMustHaveAllIpConfigurations" Message="GatewayCustomBgpAddresses must have all VirtualNetworkGateway <retracted> **IpConfigurations** in the virtual network gateway Connection <retracted>" Details=[]

looks like IpConfigurations is missing when trying to apply the terraform with those extra custom bgp custom_bgp_addresses { primary = "169.254.22.2" secondary = "169.254.21.2" }

eddiewhho commented 10 months ago

Ive created a bug related to custom BGP

sjackson0109 commented 10 months ago

If it helps, I already fixed this in my offline copy On 17 Nov 2023, at 15:17, Eddie @.***> wrote: Ive created a bug related to custom BGP

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

eddiewhho commented 10 months ago

looks like my version of azurerm v3.57.0 didnt support that functionality yet, its only release in v3.6.0 (my bad)

rcskosir commented 9 months ago

Marking this issue as closed, based on feedback in the comments including this successful build: 

resource "azurerm_virtual_network_gateway" "vngw" {
  name                = var.gateway_name
  location            = azurerm_resource_group.uks_hub_rg.location
  resource_group_name = azurerm_resource_group.uks_hub_rg.name
  type       = "Vpn"
  sku        = "VpnGw2AZ"
  generation = "Generation2"
  active_active = true

  ## ETHERNET-INTERFACES
  ## TODO: Refactor to use an array of Public IPs (azurerm_public_ip.uks_vng_pip)
  ip_configuration {
    name                          = "${var.gateway_name}-IPConfig1"
    public_ip_address_id          = azurerm_public_ip.uks_vnet_gateway_pip_01.id
    private_ip_address_allocation = "Dynamic"
    subnet_id                     = azurerm_subnet.subnets["GatewaySubnet"].id
  }
  ip_configuration {
    name                          = "${var.gateway_name}-IPConfig2"
    public_ip_address_id          = azurerm_public_ip.uks_vnet_gateway_pip_02.id
    private_ip_address_allocation = "Dynamic"
    subnet_id                     = azurerm_subnet.subnets["GatewaySubnet"].id
  }
  ip_configuration {
    name                          = "${var.gateway_name}-IPConfig3"
    public_ip_address_id          = azurerm_public_ip.uks_vnet_gateway_pip_03.id
    private_ip_address_allocation = "Dynamic"
    subnet_id                     = azurerm_subnet.subnets["GatewaySubnet"].id
  }
  ## P2S VPN Config
  vpn_type = "RouteBased"
  vpn_client_configuration {
    vpn_client_protocols = ["OpenVPN"]
    vpn_auth_types       = ["AAD"]
    aad_audience = "REDACTED" #Azure VPN (Application ID)
    aad_issuer   = "https://sts.windows.net/REDACTED/"
    aad_tenant   = "https://login.microsoftonline.com/REDACTED/"
    address_space = ["${var.vng_pool}"]
  }
  #SSL VPN clients need to reach non-bgp propagated routes (aka reach inside the provider's VPN)
  custom_route {
    address_prefixes = ["${var.vng_pool}", "10.1.18.0/24"]
  }
  enable_bgp = true
  bgp_settings {
    asn         = var.gateway_asn
    peer_weight = 100

    ## TUNNEL-INTERFACES
    # - Group 1 = SITE 1
    # - Group 2 = SITE 2
    # - Group 3 = RESERVED: SITE 3
    # - Group 4 = PARTNER SITE
    peering_addresses {
      apipa_addresses       = ["169.254.21.33", "169.254.21.37", "169.254.21.41", "169.254.21.45"]
      ip_configuration_name = "${var.gateway_name}-IPConfig1"
    }
    peering_addresses {
      apipa_addresses       = ["169.254.22.33", "169.254.22.37", "169.254.22.41", "169.254.22.45"]
      ip_configuration_name = "${var.gateway_name}-IPConfig2"
    }
  }
}
github-actions[bot] commented 4 months ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.