Network Load Balancer dual-stack support for AWS

[T-Kukawka]

Request:

We have received a request to enable dual-stack support on the AWS NLB.

The requirements are to enable the IPv6 support on NLB side only, not involving changes for internal LBs nor the cluster itself. Dual-stack NLB handles the network address translation to IPv4 out of the box.

Investigation/Initial Implementation details:

To satisfy the requirements, the setup would have to involve following steps:

• Adding IPv6 CIDR to VPC which will have to be different per cluster due to the peering.
• Creating IPv6 Subnet for each availability zone
• Adding the route tables for the VPCs
• Deploying the NLB to IPv6 subnet for the IPv6 NLB support.

This does not involve only adding and exposing the annotation on the aws-load-balancer-controller-app. The implementation will also have to consist of creating an operator and logic handling the points above. The assignments have to happen automatically per MC and WCs.

Another points to consider is that the implementation will differ between AWS Vintage and CAPA, which will also affect migration.

• In Vintage we will have to implement the logic within aws-operator (and have a WC release)
• Plain CAPA does not support IPv6 just yet: https://github.com/kubernetes-sigs/cluster-api-provider-aws/issues/2420. To have any IPv6 features full upstream implementation has to be provided, otherwise it will be rejected.
• Migration from Vintage to CAPA with IPv6 setup (if implemented) will be very problematic as is still unclear due to no implementation on CAPA.
• Alternatively Managed EKS CAPA implementation supports IPv6, however it has several limitations: https://github.com/kubernetes-sigs/cluster-api-provider-aws/blob/82b93149354660c0e54988e11a87c3287d92cc29/docs/book/src/topics/eks/ipv6-enabled-cluster.md. It would involve running Cilium ENI for ipv6 (currently we are only having Cilium ENI mode for IPv4, the IPv6 details have never been reviewed nor tested) or you can't define custom POD CIDRs on EKS with IPv6. Also important note is that The use of a Nitro enabled instance is required.

All and all this topic is an uncharted territory, both in CAPA and Vintage. It has never been tested, implementation upstream might take a very long time specially for plain CAPA. We also have to take into consideration the efforts for the actual migration from Vintage to CAPA for existing clusters if implemented. The 'combined' efforts can't be easily estimated and we will have to discuss within Product which direction we want to take with all providers in general.

### Tasks
- [ ] https://github.com/giantswarm/giantswarm/issues/29159

[T-Kukawka]

Based on the internal spike, the dual stack support for the NLB can be achieved initially with set of terraform scripts that can be applied on CAPA clusters.