Open egg-r opened 1 year ago
GilchristT
commented
1 year ago +1 having this issue with trying to deploy a new HAProxy ingress on AKS. Was beating my head against a wall for days before thinking to check the AKS load balancer health checks.
evandam
commented
1 year ago Faced a similar issue here on AWS. We had ALB health checks pointing to /healthz that was removed when the default-backend service was no longer deployed with the ingress controller so target groups went unhealthy.
We're now pointing ALB health checks to the stats port :1042/healthz which seems to work - this is what the haproxy-kubernetes-ingress deployment is using for its readiness probes as well.
I'm wondering if it's worth adding a /healthz route on the controller by default to avoid situations like this? I also tried adding an ingress to route /healthz to the stats port of the ingress controller, but it seems to expose the entire HAProxy stats page, so definitely not ideal.
stale[bot]
commented
1 year ago This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
egg-r
commented
1 year ago By following the AKS sigs documentation I was able to set the health-probe protocol back to TCP. I tested using port 1042 as mentioned previously but that was only working for HTTP checks, and not HTTPS. At least without more modifications to the helm deployment setup we have put together... don't have time to review that now. Rather have checks on both. I do agree that adding /healthz route would avoid this situation.
Sig docs for AKS: https://cloud-provider-azure.sigs.k8s.io/topics/loadbalancer/#custom-load-balancer-health-probe
--set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-internal"=true `
--set controller.service.annotations."service\.beta\.kubernetes\.io/port_443_health-probe_protocol"=tcp `
--set controller.service.annotations."service\.beta\.kubernetes\.io/port_80_health-probe_protocol"=tcp `
I upgraded one of our AKS clusters today to K8s version 1.24.9 and this caused our HAProxy Ingress to start failing probe checks. The protocol for probe paths were updated from TCP to HTTP/HTTPS. I was able to get the ingress back online by setting the controller to "externalTrafficPolicy=Local" . I tried specifying the probe path the "/healthz" , but that did not work. Does HAProxy ingress support HTTP/HTTPS probe protocol for the service ports? I upgraded via Helm to the latest chart "kubernetes-ingress-1.29.2" in my troubleshooting but that didn't resolve my issues trying to get the new probe protocols to work.
This comment helped me get the Ingress restored and gives more context.
https://github.com/Azure/AKS/issues/2907#issuecomment-1115721052