search

kubernetes-sigs / cloud-provider-azure

Cloud provider for Azure
https://cloud-provider-azure.sigs.k8s.io/
Apache License 2.0
260 stars 273 forks source link

Internal LoadBalancers do not release allocated IP addresses when service is deleted #3226

Closed fr6nco closed 7 months ago

fr6nco commented 1 year ago

What happened:

Several services were exposed as an internal load balancer (annotated with service.beta.kubernetes.io/azure-load-balancer-internal: "true"). When these services were updated to type ClusterIP, private IP addresses were not released but present in the fontend IP configuration.

What you expected to happen:

Private IPs to be released and attachment removed from the Frontent IP configuration.

How to reproduce it (as minimally and precisely as possible):

Create a LoadBalancer Service with annotation service.beta.kubernetes.io/azure-load-balancer-internal: "true". Wait until fully provisioned (private IP assigned in azure console). Patch this Service, set type to ClusterIP and remove annotation. IP address is not released from the loadbalancer service. Perhaps try multiple loadbalancers with this annotation and change only one of them to ClusterIP and see if the updated resource deleted the IP address allocation.

Anything else we need to know?:

Environment:

feiskyer commented 1 year ago

Do you share more details on "IP address is not released"? Do you mean the IP address is still on service status, or the LoadBalancer rule/frontend configurations are not deleted?

fr6nco commented 1 year ago

The frontend configuration is not deleted. The IP address was still attached to the Load Balancer in the azure console.

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

fr6nco commented 1 year ago

Issue still present on 1.25.

Attaching some screenshots.

Initially the loadbalancer was created with annotation

    service.beta.kubernetes.io/azure-load-balancer-internal: "true"

Which created a loadbalancer and allocated an IP address from the internal range

engine-ws-api-prod01-engine-ws-api-server            LoadBalancer   10.235.26.112   172.31.0.15                              80:30678/TCP,18291:31834/TCP                                                                                                 137d

which got changed to

[root@azure-ooredoo-prod-k8s-master-0 ~]# kubectl get svc -n prod engine-ws-api-prod01-engine-ws-api-server
NAME                                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)            AGE
engine-ws-api-prod01-engine-ws-api-server   ClusterIP   10.235.26.112   <none>        80/TCP,18291/TCP   137d
image

Associated IP address was not removed from the azure console tho.

image

fr6nco commented 1 year ago

/remove-lifecycle stale

nilo19 commented 1 year ago

Can you provide the logs of cloud-controller-manager during the service creation and deletion?

k8s-triage-robot commented 8 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

fr6nco commented 7 months ago

Seems to be fixed on 1.27+