Controller deletes services and recreates during bigip-ctlr pod restart

[ssurenr]

Description

Whenever the CC Pod crashes (or during scheduled upgrades) - the entire partition it manages gets wiped out - then it a couple of minutes to restore all the services configured in K8s to get configured on the BIG-IP again. This is occurring when a large number of services configured in k8s.

Kubernetes Version

1.10

Controller Version

1.7.0

BIG-IP Version

13.1

[mdditt2000]

Fix coming in release 1.8 end of Feb timeframe

[kudipudi]

@ssurenr I am using version 1.9.1 and this issue still persists.

[ssurenr]

@kudipudi May I know how many services are exposed to BIG-IP controller?

[kudipudi]

@ssurenr I have 351 services.

[sfudeus]

We are on 1.8.1 and observe that too, with just 40 services exposed via configmap to F5.

[ssurenr]

We tested this issue specifically against the release images. Please create a service request with F5 Support and let's see what's going on.

[juliantaylor]

We have opened a F5 Support case for the issue, Serial Number f5-nwwc-abxb

[sfudeus]

fyi: We just observed this once again in one of our production cluster, with only 14 virtual servers being managed there via iApp configs. Quite exactly 2 minutes after pod start, deletions started and took 130s for deleting those 14 services. A minute later, all nodes started being deleted (that took around 70s for 9 nodes) 90s later, services started to be recreated, that took around 90s as well. Overall outage this caused on the services was 7.5 minutes. As @juliantaylor commented, a case is open. If possible, I'd appreciate raising the priority again.

[mdditt2000]

@sfudeus @juliantaylor is it possible to start moving away from the legacy iApp config to using AS3. The stability has dramatically improved. Unlike a iApp AS3 declares the configuration via JSON using a validated schema. Its very predictable. You know what you are going to get. I can help you this week with creating some AS3 declarations from your current configuration. Are you just using the http iApp?

[sfudeus]

@mdditt2000 We did not look into AS3 yet, but just heard rumors about it. We do not use a standard iApp but some custom thing. Reasons for using a custom iApp have been so far missing support for snatpools, route advertisement and some minor things. Is that doable with AS3? I need to check with our balancer team if AS3 is usable in our instances. The documentation states a requirement to install the iControl extension RPM, don't know if that is acceptable for us.

[mdditt2000]

@sfudeus CIS will be moving away from iApp support as we transition over to AS3. AS3 supports snatpools, route advertisement etc by default. This will prevent using a custom iApp. Can we please chat next week so i can understand your concerns/rumors regarding AS3. I can also help with the communication to the balancer team. AS3 is a fully supported extension.

[sfudeus]

Case number is C3037553, I just added the requested data and logs. This is reproducible on our sandbox-cluster.

@mdditt2000 Thanks for the offer, if AS3 will be the way for the future I am eager to learn more details. Another thing we currently do in the iApp is abstracting some details away from our users, like the snatpool and vlan selection, which are platform specific, not user specific, and we are setting some defaults there. Is there a way in AS3 or could that be done in the bigip-ctlr (in the future?)

[juliantaylor]

I have looked into the issue a bit and am relatively certain of the cause. It seems to be due to missing synchronization between the component monitoring the kubernetes api (this repository, written in go) and the component performing the f5 updates which is in python.

The information is transferred via json file written by the k8s controller, this file is watched via inotify by the python bigipconfigdriver. The k8s information seems to be updated one event at a time so the f5 update may happen before the json configfile contains the complete current cluster state leading to the configdriver to sync with f5 based on an incomplete desired state and delete existing valid entries from the f5.

By moving the ordering of how things start in the k8s controller and adding an appropriate sleep to the python process start (about 0.7s per configmap in kubernetes ...) I could avoid the controller deleting entries.

This is of course just a workaround, a better fix could be to start listing the current state of the cluster and write the json configuration file completely before triggering the first sync with the f5, only then start a watch at the resource version of the complete cluster listing to handle updates since it was made (basically what Reflector.ListAndWatch does)

If the AS3 handling is done in the same way as the iapp handling (via the python library to sync with the f5) changing to it would probably not change anything.

[juliantaylor]

After some more tests there seems to be another more significant race than the race between initial configmap addition events and configuration write.

The virtualserver sync and the namespace sync are not synchronized with each other, so the controller starts writing the configuration to be placed into the F5 before the namespace sync process has handled all namespaces. Thus it only sees an incomplete cluster state when starting the virtual server sync.

We have around 70 namespaces and the namespace sync takes about 0.6-1 second per namespace.

quick and dirty workaround which seems to fix the problem in our clusters with about 70 namespaces and 70 virtual servers:

--- a/pkg/appmanager/appManager.go
+++ b/pkg/appmanager/appManager.go
@@ -709,6 +709,12 @@ func (appMgr *Manager) runImpl(stopCh <-chan struct{}) {
        }

        appMgr.startAndSyncAppInformers()
+       for appMgr.nsQueue.Len() > 0 {
+               log.Infof("waiting for namespace sync to complete, remaining %d", appMgr.nsQueue.Len())
+               time.Sleep(5 * time.Second)
+       }
+       time.Sleep(5 * time.Second)
+       log.Infof("done waiting for namespace sync")

        // Using only one virtual server worker currently.
        go wait.Until(appMgr.virtualServerWorker, time.Second, stopCh)

[mdditt2000]

PM Jira tracking CONTCNTR-862 for resolution in CIS 1.10 schedule mid August.

[mdditt2000]

Fix coming into CIS 1.10 scheduled for 8/14 +- and if you need an urgent fix then we can provide a hot fix. Please let us know. TY

[sfudeus]

@mdditt2000 Great to know, thanks. We are doing fine so far with out own hotfix, so we can wait for 1.10. If you would like us to test your fix, we are on standby.

[mdditt2000]

Fix is coming in CIS release 1.9.2 schedule for July 26th.

[ssurenr]

Fixed in v1.9.2