Investigate if we can assign EIP to nginx Network Load Balancer

[timckt]

Background

We would like to add Network Load Balancer to AWS Shield Advance for protection. However, according to AWS document, we cannot directly add NLB to Shield and instead we need to add it through associations to Amazon EC2 Elastic IP addresses.

Currently our cluster's network load balancer is created by ingress-controller, and the public IP is auto assigned by AWS and we cannot add this public IP to AWS Shield.

Relates to #6001

Proposed user journey

• Investigate if we can assign EIP to nginx Network Load Balancer to enable Shield for NLB.

• If this is not possible to configure on the ingress controller config, can we request AWS make the NLB dynamic EIPs available to us for assigning Shield rules

• Approach

Which part of the user docs does this impact

Communicate changes

• [ ] post for #cloud-platform-update
• [ ] Weeknotes item
• [ ] Show the Thing/P&A All Hands/User CoP
• [ ] Announcements channel

Questions / Assumptions

Definition of done

• [ ] Assign EIP to the test cluster nginx ingress
• [ ] Verify if the EIP can be added to AWS Shield
• [ ] Prepare demo for the team
• [ ] Pass integration test
• [ ] Complete Communication Changes

Reference

How to write good user stories

[timckt]

Below is the update from aws support ticket

In the case where the NLB already has a IP that is assigned by AWS associated with it, you would need to add a new availability zone(AZ), if available, and specify the IPv4 address as an EIP(Shield Advanced Protected one). NLB does not support changing an existing AZ from AWS assigned IP address to EIP. It is important to note that the original AZ's will have non Shield Advanced Protected IP. The recommended option would be to create a new NLB with EIPs in all the AZ's. These EIPs will be Shield Advanced protected. Once this NLB is provisioned the customer can shift traffic from the old NLB to the new one. Once the EIPs are associated with your NLB, go to the AWS Shield Advanced console and add those EIPs as protected resources. After completing these steps, your NLB will be protected by AWS Shield Advanced against DDoS attacks

Reference https://github.com/kubernetes/kubernetes/issues/63959

[timckt]

Update:

• Our current IP of NLB are assigned by AWS, and we cannot add the assigned IP to AWS Shield Advanced Resources.

• Have created a new set of NLB with EIP attached, working branch here

  • default nginx ingress

  • modsec nginx ingress

• However, once the NLB is created, we cannot change its IP. We cannot directly change the live cluster NLB IP from assigned by aws to EIP. AWS Document here

  When you create an internet-facing load balancer, you can optionally specify one Elastic IP address per subnet. If you do not choose one of your own Elastic IP addresses, Elastic Load Balancing provides one Elastic IP address per subnet for you. These Elastic IP addresses provide your load balancer with static IP addresses that will not change during the life of the load balancer. You can't change these Elastic IP addresses after you create the load balancer.

• Also, as confirmed from AWS support, we have already utilised 3 AZ for current NLB, and we cannot add additional subnet with EIP to it

  your NLB is already utilizing all the available subnets across the three AZs. Unfortunately, it is not possible to add additional subnets to an existing NLB once it has been created.

• If we would like to switch our live NLB IP from assigned by AWS to EIP with minimum downtime , we need to create a new set of NLB with EIP and then switch over.

  To address your requirement of attaching EIPs to your NLB while ensuring high availability and fault tolerance, I recommend creating a new NLB with EIPs provisioned in all the AZs.

[timckt]

Next Step:

• Update ingress module to create NLB with EIP (Completed with this working branch)
• Investigate the way to implement the change in live cluster with minimal downtime