Orphaned IPs remain in IP Pool (/w nsx-t based Load Balancer)

moonek commented 2 years ago

What happened?

When using NSX-T based LoadBalancer, I encountered a problem where the number of IPs allocated to the IP Pool and the number of VIPs actually used did not match. This results in the following issues that I checked through the NSX-T Web Console and REST API.

There are orphaned IPs in the IP Pool.
Duplicate IPs exist in the IP Pool. (Duplicate IPs cannot be deleted with REST API)
The number of IPs that exceeds the IP range set in the IP Pool appears to be allocated IPs.

We experienced this issue twice a year in the staging environment.

I wonder if it is a cloud provider issue or an NSX-T issue, and I want to know a solution or workaround.

What did you expect to happen?

I hope that the IP assigned to the IP Pool matches the number of k8s services(w/ type: LoadBalancer) created.

How can we reproduce it (as minimally and precisely as possible)?

In the staging environment, k8s services(w/ type: LoadBalancer) were frequently created/deleted.

Kubernetes version

v1.20.0

Cloud provider or hardware configuration

v1.20.0

Others

NSX-T version: 3.1.2.1.0.17975795 vSphere version: 7.0.2.00200

lubronzhan commented 2 years ago

Sounds like a bug to me in NSX. CPV should just be request ip for the service, and NSX-T manager should be allocating the IP based on availability, from your description, NSX-T allocates IP and it failed to clean up all of them

lubronzhan commented 2 years ago

Let me ask my NSX-T foks

moonek commented 2 years ago

@lubronzhan This is an important issue for us. Thanks for checking further.

lubronzhan commented 2 years ago

Hi @moonek I talked to them, they said you should talk to NSX-T GSS, the support guy. Collect bundle and file issue to NSX-T

moonek commented 2 years ago

It is unclear whether CPV can solve it, so I will close it. I will reopen if there is any update in the future.

kubernetes / cloud-provider-vsphere