Service `type: LoadBalancer` IPs re-assigned when services are updated

Setup Details

CIS Version: 2.18.0 Build: 0.289.5 BIGIP Version: Big IP 17.1.1.3 AS3 Version: 3.46.2 Agent Mode: AS3/CCCL
Orchestration: K8S/OSCP
Orchestration Version: 1.28.9 / 4.15.12 Pool Mode: Cluster/Nodeport
Additional Setup details: <Platform/CNI Plugins/ cluster nodes/ etc>

Description

When a service of type: LoadBalancer is updated after a service with a lexicographically lower external IP is deleted, the external IP address assigned to the updated svc changes. The originally assigned external IP address is put back into the IP pool and can be assigned to another service. This is extremely problematic for services where the external IP is expected to stay the same with dependencies that rely on the IP address not changing.

Steps To Reproduce

1) Create two services of type: LoadBalancer and check that they have been successfully assigned external IPs. Save these IPs somewhere where you can compare the service names and IPs after the following steps.

kubectl get svc -A | grep -i loadbalancer > lb.before

2) Delete the service of type: LoadBalancer with the lexicographically lower IP.

3) Edit the service with the lexicographically higher IP in some way. For example, add a fake label to the service and save the changes.

labels:
   app: test

labels:
   app: test
   testing: new-label

4) Diff the output from step 1 with the output containing the services and IP addresses after the above edit in step 3 is made

kubectl get svc -A | grep -i loadbalancer > lb.after

diff lb.before lb.after

The above diff should show the IP address was changed.

Expected Result

The IP address assigned to a svc (after another svc is deleted) should not change. It should maintain the same IP even if the existence of other services change.

Actual Result

The external IP address assigned to a svc (after a svc with a lexicographically lower IP is deleted) changes.

Diagnostic Information

kubectl get svc -A | grep -i loadbalancer > lb.before

test-ns      service1     LoadBalancer       X.X.X.12          
test-ns      service2     LoadBalancer       X.X.X.20                                 
---
Delete service1
---
Edit service2
---
kubectl get svc -A | grep -i loadbalancer > lb.after

test-ns      service2     LoadBalancer       X.X.X.12

X.X.X.20  becomes available when it should have stayed assigned to service2. X.X.X.12 should have become available

Note: since this follows lexicographical ordering, this behavior is observed to be the same when X.X.X.20 is replaced with X.X.X.2

Observations (if any)

It seems that when a service is updated (after a svc with lexicographically lower IP is deleted), it is being treated like a POST request instead of a PUT request. When the contents of the service object are updated, the external IP is re-assigned (like a create) instead of kept the same (like an update).
The order that external IPs are assigned and values are retrieved follows lexicographical ordering, not numerical ordering. https://stackoverflow.com/questions/45950646/what-is-lexicographical-order

F5Networks / k8s-bigip-ctlr