Azure / AKS

Azure Kubernetes Service
https://azure.github.io/AKS/
1.97k stars 307 forks source link

SyncLoadBalancerFailed Tag value too large when creating several LoadBalancer services with the same IP #2160

Closed kortatu closed 3 years ago

kortatu commented 3 years ago

What happened: When creating multiple LoadBalancer services with the same IP (just a preview feature from February 2021), you can't get more than 10/15 (depending on the services names) because you get a Tag value too large error:

Warning SyncLoadBalancerFailed 5m22s (x7 over 14m) service-controller Error syncing load balancer: failed to ensure load balancer: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 400, RawError: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 400, RawError: {"error":{"code":"InvalidTagValueLength","message":"Tag value too large. Following tag value 'default/public-ip-07d,default/s-mmv8b,default/s-khwwp,default/s-m2dwz,default/s-vqqwm,default/s-7shb4,default/s-wkxn8,default/s-jtbj4,default/s-28ncs,default/s-lvr87,default/s-qvnjd,default/s-q4c7g,default/s-mchg2,default/s-5rjmj,default/s-wc8v2,default/s-zkgct' exceeded the maximum length. Maximum allowed length for tag value - '256' characters."}}

It seems that the Public IP in Azure is tagged with the names of the services concatenated.

What you expected to happen: I expected the name of the services not being a limitation on the number of services sharing the same IP as long as ports are available and not conflicting

How to reproduce it (as minimally and precisely as possible):

Create several LoadBalancer services with the same IP, each with different ports and after 10 /15 you start getting errors of SyncLoadBalancerFailed (I attached an example bash script)

Anything else we need to know?:

Environment:

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.17", GitCommit:"f3abc15296f3a3f54e4ee42e830c61047b13895f", GitTreeState:"clean", BuildDate:"2021-01-15T02:40:30Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-21T15:16:40Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

A bunch of HTTP microservices, a small database, a Kafka cluster and a lot of small short-lived containers processing streams (each one with an external facing port)

lb-service-gen.txt

ghost commented 3 years ago

Hi kortatu, AKS bot here :wave: Thank you for posting on the AKS Repo, I'll do my best to get a kind human from the AKS team to assist you.

I might be just a bot, but I'm told my suggestions are normally quite good, as such: 1) If this case is urgent, please open a Support Request so that our 24/7 support team may help you faster. 2) Please abide by the AKS repo Guidelines and Code of Conduct. 3) If you're having an issue, could it be described on the AKS Troubleshooting guides or AKS Diagnostics? 4) Make sure your subscribed to the AKS Release Notes to keep up to date with all that's new on AKS. 5) Make sure there isn't a duplicate of this issue already reported. If there is, feel free to close this one and '+1' the existing issue. 6) If you have a question, do take a look at our AKS FAQ. We place the most common ones there!

ghost commented 3 years ago

Triage required from @Azure/aks-pm

ghost commented 3 years ago

Action required from @Azure/aks-pm

ghost commented 3 years ago

Issue needing attention of @Azure/aks-leads

ghost commented 3 years ago

Issue needing attention of @Azure/aks-leads

feiskyer commented 3 years ago

This is known issue from upstream cloud-provider-azure and would be tracked by https://github.com/kubernetes-sigs/cloud-provider-azure/issues/510.