search

Azure / application-gateway-kubernetes-ingress

This is an ingress controller that can be run on Azure Kubernetes Service (AKS) to allow an Azure Application Gateway to act as the ingress for an AKS cluster.
https://azure.github.io/application-gateway-kubernetes-ingress
MIT License
678 stars 422 forks source link

Supporting more than 100 ingresses #1299

Open alxy opened 3 years ago

alxy commented 3 years ago

Situation We have a microservice application consisting of ~20 services that we want to deploy for multiple customers. To save some costs, we use one AKS (with one namespace per customer) and one Application gateway (with a dedicated subdomain per customer) resource. That has proven to work for the first few customers.

Problem Now, with more customer it looks like we are approaching some limitations of Application gateway. Mainly we are struggling with the fact, that AppGW only allows to create 100 backend pools per instance. As most of the services do expose an ingress currently, we get also one additional backend pool for each such ingress, leaving is roughly with #customers x #services backend pools. As there seems to be a 1:1 relation between ingresses and backend pools created by AGIC, that currently limits us to having 5 customers on one AKS and AppGW, and severely limits our future capacity to add new services.

Question As the limits can't be raised for AppGW via support ticket and seem to be rather fixed, Im asking if there is an alternative solution on the AGIC level here? I'm not too familiar with all the concepts and would just be open to any kind of solutions to remedy the situation.

lbargery commented 2 years ago

Also massively hitting this issue!

arodriguezdlc commented 2 years ago

We have a similar problem too. This 100 limit for most of application gateway configuration (backend pools, listeners, etc) is so low if you want to offer a multitenant service using AKS and the Application gateway. And if you want to use WAF is worse, because the limit is reduced to 40.

One thing we use to mitigate (only a little bit) the problem is the undocumented (https://github.com/Azure/application-gateway-kubernetes-ingress/issues/1263#issuecomment-919668462) annotation appgw.ingress.kubernetes.io/hostname-extension, which allows the use of multiple hostnames for one listeners (up to 5). If you use multiple hostnames for the same backend, it could be useful to you.

I don't know how we can push and make Microsoft realize how necessary is to increase this limit.

gitflo1 commented 2 years ago

We have the same issue. We already talked with Microsoft for a possible solution here. They kindly asked us to create Microsoft Support tickets in the Azure portal to create visibility and transparency. Other than this I did not really find a solution other than scaling the App Gateway instances and the AGIC charts per cluster. Maybe you could host a dedicated ingress controller in each application's namespace and do a sort of cascading load balancing? So the App Gateway only needs one route (e.g. my-app-hostname.com/) pointing to the namespace's load balancer, which is capable of doing the path-based routing per application (e.g. /api -> backend-service, /billing -> billing-service, ...)?