Kubernetes API concurrent private and public load balancers

[alex-dabija]

User Story

- As a cluster admin, I want to have two Kubernets API loadbalancers, one public and one private, in order to use them for either public or private use-cases.

Details, Background

Cluster API allows for only one Kubernetes API loadbalancer to be created, either private or public, but not both. In contrast, Giant Swarm clusters have two Kubernetes API loadbalancers, one public and one private, to be used for different use-cases.

The gap between Cluster API and Giant Swarm clusters can be reduced by supporting two loadbalancers for Cluster API clusters.

UPDATE: With this PR: https://github.com/kubernetes-sigs/cluster-api-provider-aws/pull/4733 we could satisfy customer requirements to use the internal LB. We have to test this properly.

Questions

• [ ] Do we have customers which need both private and public Kubernetes API loadbalancers?
• [ ] What would be the requirements from Customers? We can assume that the customers would use it in same way as on Vintage

Blocked by / depends on

None

[AverageMarcus]

Related upstream issues: Support multiple endpoints for API (private + internet-facing) Support for multiple ControlPlaneEndpoints Load Balancer Provider Upstream Load Balance Provider proposal: https://docs.google.com/document/d/1wJrtd3hgVrUnZsdHDXQLXmZE3cbXVB5KChqmNusBGpE/edit

Semi-related issue (though not about supporting multiple in its current form) - Bring your own control plane ELB

[AverageMarcus]

I'm going to park this for now until there's some feedback from upstream on how/if they want to include this functionality.

For reference: I've suggested updating the Scheme property to include a new both value (https://github.com/kubernetes-sigs/cluster-api-provider-aws/pull/2852)

We could back-port this via the use of Kyverno or similar to generate a new "APILoadBalancer" resource if Scheme is set to both and mutate the Scheme to be internal. We'd then need an operator to handle creation of the additional ELB.

[AverageMarcus]

After looking at the proposed Load Balancer Provider more closely I've noticed that the current draft doesn't actually support multiple load balancers.

Instead it makes use of an object reference, e.g.

apiVersion: controlplane.cluster.x-k8s.io/v1alpha4
kind: KubeadmControlPlane
metadata:
 name: my-cluster-control-plane
spec:
 loadBalancerRef:
     apiGroup: cluster.x-k8s.io
     kind: LoadBalancer
     name: my-cluster-control-plane-lb
---

apiVersion: cluster.x-k8s.io
kind: LoadBalancer
metadata:
 name: my-cluster-control-plane-lb
spec:
 infrastructureRef:
   apiGroup: infrastructure.cluster.x-k8s.io
   kind: AWSELBClassicLoadBalancer
   name: my-cluster-control-plane-celb

This does open up the possibility to create a AWSDualELBClassicLoadBalancer or similar to handle the case where both internal and internet-facing are needed.

[T-Kukawka]

should be possible now with: https://github.com/kubernetes-sigs/cluster-api-provider-aws/pull/4733