Azure Platform: Only 5 Frontend IPConfigurations are allowed

[colemickens]

Currently, Azure ARM limits you to 5 frontend ipconfigurations per load balancer, and allows you to raise that limit to 30 by contacting support.

These limits seems low.

@theobolo (and any other users): Can you please comment on your deployment metrics - avg cluster size (number of machines, vm size) as well as the number of external-facing Services you have deployed.

---

This issue is relevant to:

• Kubernetes 1.4 users using the native Azure cloudprovider support
• Users who are manually reconfiguring the Azure loadbalancer in front of a VMSS

[theobolo]

In my case :

• 5 nodes : Standard_DS3_v2
• Master : Standard_DS4_v2

I tried to deploy more than 5 external services but on the 6th service Kubernetes throw that error :

(will retry): failed to create load balancer for service default/test-svc-lb2: network.LoadBalancersClient#CreateOrUpdate: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="LoadBalancerFrontendIPConfigurationCountLimitReached" Message="A load balancer cannot have more than 5 FrontendIPConfigurations." Details=[]

[colemickens]

How many Services do you need to be able to deploy on a cluster of this size? In general, is that an average-sized cluster? Do you envision having like a larger Prod cluster, or no?

[theobolo]

Hummm if i think a little about it,

That's not really about the cluster size but more than the expectation in term of applications. In my case i need 5 frontend IP for my different API / Dashboard / Backend / etc ....

Then i think about the Logging and monitoring stuff : grafana / weave-scope / kubernetes dashboard / Kibana / some others ...

If i do the math i reach easily more than 10 Frontend IP.

I can also have a PREPROD Environnement on the same cluster than the Production one and need 2x more frontend IP for that PreProd PODs.

[xynova]

Yep 5 is very low indeed.. you can hit that limit pretty fast just by having a 2 env and a couple of apps. Having said this, exposing every service on a different IP might not be ideal once you start getting more services exposed.

[theobolo]

Any news about this Cole ?

[colemickens]

Unfortunately, there's nothing much to say. For now, users will need to simply request the limit be raised to 30. If you need more than 30, please let me know why (if you don't want to discuss it here, you can email me at colemick@microsoft.com).

[colemickens]

It should be fairly straightforward to ask for the limit to be moved from 5 to 30. If you run into friction here, please let me know.

[theobolo]

@colemickens Ok Cole i asked about increasing the limitation, should be enough for the moment.

[colemickens]

Another note, the limit is per-subscription, so once it's lifted, it will be lifted for all resource groups / load balancers.

[theobolo]

As i wondered, thanks Cole !

@colemickens

[djsly]

Hey @colemickens I hit the same limitation today.

2h      3m      32  {service-controller }           Warning     CreatingLoadBalancerFailed  Error creating load balancer (will retry): Failed to create load balancer for service ntg6ms1/gateway: network.LoadBalancersClient#CreateOrUpdate: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="LoadBalancerFrontendIPConfigurationCountLimitReached" Message="A load balancer cannot have more than 10 FrontendIPConfigurations." Details=[]

I will request to have this opened up to 30 but I'm scared that this won't be enough.

We are expecting to have multiple users deploying the same stack for Dev Testing / QA Testing, etc. Setting the service type to Load Balancer is too easy for users...

Any plans on creating multiple LB in the future?

Note that we are not yet using Ingress Controller, which could indeed reduce the number of frontendIP Pool.

Thanks!

[colemickens]

I mean, I would've made multiple LBs if it were possible originally. Having to put all the ip configs on one is non-trivial. :(

Unfortunately, this is just how Azure LBs work. There is a 1:1 relationship between the LB objects and a given backend pool. And a node can only be in one LB backend pool.

Ingress is the answer for now.

[colemickens]

@djsly Looking through my email again, I think there's a possibility of getting it raised higher in exceptional cases.

If you or anyone else reading this has requirements for more than 30, please drop me an email with more details (how many, why you need them all in one cluster, did you consider Ingress, etc) at colemick@microsoft.com and I can escalate it.

[djsly]

@colemickens thanks. with 30 we are good for now, The major issue getting us close to 30 is the IP leak that I will be going back to debug this week.... when we create/recreate the same objects... this fills up the LB quite fast as well.