Added support for preflight checks for cluster upgrades

Added preflight checks for checking

All masters are in ready state
All control plane pods are in ready state
Kubernetes version on every machine is equal or higher than minimum

For querying K8s object used kubeclient created from kubeconfig as running kubectl and greping could have got ugly.

Control plane pods not running

Simulated a pod failure by removing the required image

coreos-puneet-339-10-105-16-77platform9 ~ # kubectl get pods  --all-namespaces
NAMESPACE     NAME                                   READY     STATUS         RESTARTS   AGE
kube-system   kube-apiserver-10.105.16.43            1/1       Running        27         2h
kube-system   kube-apiserver-10.105.16.77            1/1       Running        1          1h
kube-system   kube-apiserver-10.105.16.9             0/1       ErrImagePull   28         2h
kube-system   kube-controller-manager-10.105.16.43   1/1       Running        0          2h

2018/08/31 04:29:32 Running preflight checks for cluster upgrade
2018/08/31 04:29:32 Not able to verify machine SSH identity: No public keys given. Continuing...
2018/08/31 04:29:32 Checking if all masters are in ready state
2018/08/31 04:29:32 Checking if all control plane pods are in ready state
2018/08/31 04:29:32 Preflight check failed with error: there are NotReady control plane pods in the cluster: [kube-apiserver-10.105.16.9]

Master node not ready

Simulated by stopping kubelet on a master

coreos-puneet-339-10-105-16-77platform9 ~ # kubectl get nodes
NAME           STATUS     ROLES     AGE       VERSION
10.105.16.34   Ready      <none>    20h       v1.10.4
10.105.16.43   NotReady   master    21h       v1.10.4
10.105.16.48   Ready      <none>    21h       v1.10.4
10.105.16.77   Ready      master    21h       v1.10.4
10.105.16.9    Ready      master    21h       v1.10.4

coreos-puneet-339-10-105-16-77platform9 ~ # ./cctl upgrade cluster
2018/08/31 22:52:37 Running preflight checks for cluster upgrade
2018/08/31 22:52:37 Not able to verify machine SSH identity: No public keys given. Continuing...
2018/08/31 22:52:37 Checking if all masters are in ready state
2018/08/31 22:52:37 Preflight check failed with error: there are NotReady masters in the cluster: [10.105.16.43]

Version skew in kubernetes version

Manually edited state file to change kubernetes version on a host

coreos-puneet-339-10-105-16-77platform9 ~ # ./cctl upgrade cluster
2018/08/31 20:37:06 Running preflight checks for cluster upgrade
2018/08/31 20:37:06 Preflight check failed with error: minimum version check failed for machine 10.105.16.77. Kubernetes version on the machine is 1.9.4 minimum supported version for upgrade 1.10.0

platform9 / cctl

Added support for preflight checks for cluster upgrades #110

Control plane pods not running

Master node not ready

Version skew in kubernetes version