Remove coupling with Kubernetes API

[munnerz]

Right now, in order to determine what actions to take or events have been fired (see hooks and events packages), we directly query the Kubernetes API.

In order to support alternative environments, we need to identify where we query this API, why we query it, and then provide an abstraction for that functionality via the TBC 'navigator-apiserver'.

Once we've refactored out all queries to the Kubernetes API, we should be able to run Pilots in any environment supported by the navigator-apiserver.

[munnerz]

I think the first steps for this issue are to identify what each of these points are -

1) pkg/manager/hooks/events/scale_down.go - here we query the Kubernetes API for the current number of replicas in the owning 'set' for this Pod.

2) pkg/manager/manager.go - here we have a KubeClient field that returns a Kubernetes client reference

3) pkg/util/kube.go - provides some utility functions to create a Kubernetes Clientset

4) pkg/util/es.go - here we rely upon the 'name' of this 'pod' (or instance in the general case?) to identify if this is the 'first' (ie. 0th index) of the statefulset. This is effectively a rudimentary 'leader election' (where the 0th instance is leader). This should be replaced by some leader election primitive in the navigator-apiserver.

From this, there are two requirements of navigator-apiserver:

• a leader election primitive, to help us replace (4)
• some way to determine if this instance is 'going away forever' (ie. a scale down is happening and the 'current' node has been selected to be deleted). This addresses (1). This may be a complex thing between different types of environments, and will require some special consideration as to how it's implemented.