[Feature] Allow to configure per-nodepool property to select outbound LB backenpool

Tazmaniac commented 1 year ago

Is your feature request related to a problem? Please describe. We would like to be able to select which outbound IP(s) are used inside our distributed application.

Describe the solution you'd like One very simple, elegant and effective possibility is to use one IP/list_of_IP/subnet per K8S nodepool. The application schedule with IP(s) to use for witch workload/usage by scheduling pods on the proper nodepool, or pods could select tasks requiring a specific IP address to use in a message bus by filtering messages based on the pool they are running on. Or many other possibilities.

To achieve that, you simply need

a (or many) user provided LB on which you configure as many backendpool as you want with proper frontend and outbound rules
a per nodepool property to configure which backendpool will be injected in the corresponding vmss "virtualMachineProfile.networkProfile.networkInterfaceConfigurations[0].ipConfigurations[0].loadBalancerBackendAddressPools"

The usage of these new nodepool properties could be subject/conditioned by a new outbound mode "userAssignedLoadbalancer", but it already work today and is just missing this trivial property plumbing between AKS orchestrator engine and the VMSS one. See below. The functionality could be restricted to apiserver-vnet-integrated cluster to simplify things as the outbound path is not need by the nodes to contact their API servers in this case.

Describe alternatives you've considered I have this setup running, but build/maintained in a very inelegant/fragile way:

My cluster is configured with "userDefinedRouting" outbound mode (with an empty routing table) and is apiserver-vnet-integrated
I deployed a NATGateway assigned to the cluster subnet for the deployment of the cluster
I manually set "virtualMachineProfile.networkProfile.networkInterfaceConfigurations[0].ipConfigurations[0].loadBalancerBackendAddressPools" of all the underlying vmss with the wanted LB backendpools after the AKS deployment.
Disassociate the NATgateway of the subnet : You now have per nodepool outbound IP(s) setup/full control.

All scaling up/down operations on the nodepools work without disruption. Start/Stop of the K8S cluster work as intended too.

But for all upgrade/re-imaging/... of K8S (and so VMSS) you need to stop your workload, re-assing the NATgateway to your K8S subnet, do the upgrade, re-set the VMSS loadBalancerBackendAddressPools, disassociate the NAT gateway, restore the workload. Automatic upgrades are prohibited.

Tazmaniac commented 10 months ago

For those daring enough to test my setup, your cluster must use load-balancer-backend-pool-type nodeIP for fully functional managed aks internal load balancer. Otherwise, the VMSS manual setup will conflict with it.

In the proposed "userAssignedLoadbalancer" outbound mode, if the AKS engine use the LB nodeIP mode to inject the configuration directly in the nodepool provided "loadBalancerBackendAddressPools" properties (a list of LB backend pools to inject the nodeIP to) the VMSS engine will not need to be involved. load-balancer-backend-pool-type nodeIP could be a hard requirement for the functionality to simplify it implementation.

Tazmaniac commented 10 months ago

https://github.com/Azure/AKS/issues/4023 is the generalization of this feature at the K8S Service CRD level for specifying backendpools used in this use case in inbound nat or LB rule. But it require the creation of an cluster "inbound mode" property similar to the outbound mode one with possible values "userAssignedLoadbalancer" or "managedLoadBalancer"