[Feature Request] Need mechanism to re-trigger subnet auto-discovery during reconcile

[leoxlin]

Is your feature request related to a problem?

The current reconcile system does not go through subnet auto-discovery after ALB creation. Due to an internal issue we had some subnets run out of IP addresses. According to the docs we saw that the controller only considers subnets with at least eight available IP addresses. This caused ALBs that were created during our internal issue to be missing AZs.

We tried to restart the aws-load-balancer-controller pod to trigger a reconcile but we found that the controller only ran auto-discovery when the the ALB is created.

We couldn't find a way to trigger a reconcile to fix this. Fortunately all of the ALBs created were in our staging environment so we were able to delete the ALB and have the controller recreate them. However we are concerned about the process of restoring this in production.

Describe the solution you'd like

• We would like to see this behavior (subnet auto-discovery only happens on ALB create) documented in the subnet auto-discovery documentation.
• We would like to see a native way to force auto-discovery to run on every reconcile. Perhaps, an annotation like alb.ingress.kubernetes.io/subnet-discovery-strategy=always|create

Describe alternatives you've considered

• We deleted the ALB to get them to reconcile but we cannot use this option in production as it will result in downtime
• We also considered modifying the created ALB manually but decided against that option

Versions

We run aws-load-balancer-controller 2.7.1 on our EKS cluster with K8S version 1.28

[M00nF1sh]

@leoxlin We have deliberately made a change to only trigger subnet discovery for new ALB/NLB only due to the fact that switch subnets for existing LBs will result in downtimes.

For use cases that need to change subnets for existing LBs, you can use alb.ingress.kubernetes.io/subnets annotation to explicitly denote the subnets you want to use.

[leoxlin]

Hey @M00nF1sh thanks for getting back to us so quickly on this! Appreciate it :)

It will still be nice to clarify this in the subnet auto-discovery documentation, we ended up needing to dig through the code base to confirm this behavior.

We had considered using the alb.ingress.kubernetes.io/subnets annotation as well. It is good to know that changing the subnets will result in ALB downtime. Even so, it would still be nice to rely on the other checks built into the auto-discovery process to refresh our subnet list instead of manually changing things via alb.ingress.kubernetes.io/subnets. We manage quite a number of VPCs and it would be a lot of manual work to run through those checks ourselves to determine the subnet list across the board.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale