[Feature] Fully support Private AKS clusters with AGIC in a VWAN environment

WaitingForGuacamole commented 1 year ago

Is your feature request related to a problem? Please describe. If I configure my AKS cluster in a Virtual Hub spoke VNet, configure it privately (both AGIC and load balancer), then in order to get health probes working, I MUST install a UDR default route of 0.0.0.0/0 -> Internet. Otherwise all probes result in Unknown status.

If I install that default route, then outbound access from my cluster does not work, most likely because my NSGs prohibit inbound access from the internet - WHICH I WANT, because I want all traffic to be inspected by the firewall

The problem here, to me, is that Azure Firewall in a Virtual Hub setting does not allow BYOIP.

I think this problem might go away if I could do one of two things:

Configure Application Gateway to use a public IP address rather than a resource - in which case I could provide a public IP allocated by by Virtual Hub/Firewall combination.
Allow Public IP resources to be attached as public IPs to a Virtual Hub/Firewall

If one of these were to happen, then Azure sees the App Gateway at a public IP that has to traverse the firewall, THEN get DNATted to the App Gateway internally.

Describe the solution you'd like

VWAN/Virtual Hub environment, in which an AKS cluster is installed in a spoke VNet attached to the Virtual Hub.
Private AKS cluster with Private AGIC (v2 SKU exposes a public IP for the App Gateway, but all AKS services reference private IP, so no public listeners attach to it)
userDefinedRouting network_profile option for the AKS cluster, so that I can use an internal load balancer
Health probes in the Application Gateway work.
No wonky routing/firewall/NSG tricks

Describe alternatives you've considered nginx or Traefik as my ingress controller

Additional context Add any other context or screenshots about the feature request here.