IPAM-assigned IP addr does not persist across CIS restart if another CR has same IP manually assigned.

mikeoleary commented 2 years ago

Setup Details

CIS Version : 2.7.1 FIC Version: 0.1.6 Build: f5networks/k8s-bigip-ctlr:latest
BIGIP Version: Big IP 15.1 AS3 Version: 3.x
Agent Mode: AS3 Orchestration: K8S
Pool Mode: Cluster/Nodeport

Description

Customer has several applications that need both a HTTP virtual server and a TCP virtual server, where the HTTP vs listens on port 80/443 and the TCP vs listens on additional port(s). They all need to have the same IP address.

You can achieve this with a VS and TS CRD with same IP configured and unique ports, but this customer uses IPAM ctlr so that dev teams are not required to know any IP ranges.

CIS & IPAM controller give out a new IP address for each CR that is labeled, which does not allow for multiple CR's with different ports to use IPAM and get the same IP addr. This is the driver behind feature request #1902

To work within that limitation, customer automated a process with Ansible like this: 1) deploy HTTP VS CRD with IPAM label 2) run kubectl get vs -n <namespace> foo-vs and get the IP addr that IPAM assigned 3) edit the YAML manifest for the TS CRD. Add the IP address learned from step 2 4) deploy TS CRD

This works, but when CIS is restarted, all of the IPs that were assigned to VS resources tend to get different IPs as compared to the IP each had before the CIS controller restart. Note that the FIC was never restarted in this scenario, only the CIS controller.

Steps To Reproduce

1) Deploy VS CRD with IPAM label, get dynamic IP assigned 2) Deploy TS CRD on unique port with the same IP addr that was assigned to your VS 3) Restart CIS. Customer reports that VS usually gets new IP but TS keeps statically-assigned old IP.

Expected Result

IPAM-assigned IP addresses persist across CIS restarts, even if there is a duplicate manually-assigned IP on another resource,

OR

Have a way to create a TS to dynamically "follow" the IP address of an associated VS.

Actual Result

CIS restart causes new IP addr to be assigned to VS, but TS now has a different IP address from the VS. Application functionality is broken for clients.

Observations (if any)

Not related to this behavior, but customer also uses the EDNS resource to create WideIP's on GTM.
Customer has about 36 apps that need multiple CRD's on same IP / unique ports, and these are replicated in 4 env's, so there's 120+ apps that this affects.

trinaths commented 2 years ago

Created [CONTCNTR-3247] for internal tracking.

trinaths commented 7 months ago

Closing as completed. Updated documentation.

F5Networks / k8s-bigip-ctlr