Optimize sleep in test/systemtests

[vhosakot]

The Contiv system tests in netplugin/test/systemtests call time.Sleep() 151 times in code, many of them in loops. Due to this, the time needed to run these system tests is high. Netplugin CI needs 2.5 hours to run them.

This PR optimizes and reduces the sleeps in system tests' code in the following areas:

1. Add polling code in contivModelClient.go to poll for the status of the objects AppProfile, EndpointGroup, Global, Network, Rule after creating (httpPost) and deleting (httpDelete) them.
2. Remove sleep in the tests and poll for the status of objects AppProfile, EndpointGroup, Global, Network, Rule after creating and deleting them.
3. Add execRetry to retry a command if it fails for a specific duration.
4. Call execRetry instead of execBG in startListener and checkConnection so that they will be retried if they fail in the tests.
5. Replace RunCommandBackground with RunCommand so that the test blocks until the command completes.
6. Replace podExecBG with podExec so that the test blocks until the kubectl exec command completes.
7. Remove unwanted sleep in code.
8. Reduce sleep in code wherever needed.
9. Optimize polling interval in code when checking things like:
   • Node is reloaded
   • Etcd and consul are restarted
   • Interface is flapped
   • VTEPs are verified
   • ACI gateway is restarted
   • Checking for the status of Netplugin after starting and stopping it
   • k8s pods are deleted

Signed-off-by: Vikram Hosakote vhosakot@cisco.com

[contivbot]

Can one of the admins verify this patch?

[vhosakot]

build PR

[unclejack]

build PR

[unclejack]

build PR

[unclejack]

build PR

[unclejack]

build PR

[unclejack]

build PR

[unclejack]

build PR

[vhosakot]

build PR

[unclejack]

There's an issue with the CI. A new PR needs to be opened to get rid of this issue.