[EKS] [Pod Eviction Order]: Pod Eviction during upgrades

[mohitanchlia]

When using ALB ingress controller with clusterIP it's important that alb ingress controller is not evicted along with the applications pods, this avoids any impact to the ongoing traffic. Either ingress controller needs to be evicted first or the last but not together, this is because alb ingress controller drains the pod ip from AWS ALB and when it's not running at the time of eviction causes disruption in service. It would be very useful to allow users to specify pod eviction order to deal with such scenarios.

[M00nF1sh]

In our new helm charts for AWSLoadBalancerController(v2 of ALB ingress controller):

• we support priorityClassName(by default it's system-cluster-critical). You can customize the priorityClass if needed.

• we support podDisruptionBudget as well.

  you can achieve desired behavior with the above two settings.