Upgrade K8S Version

[kahkhang]

Hopefully solves #52. Having some certificate error with kubelet.service which I was unable to debug.

Nov 08 23:20:35 localhost kubelet-wrapper[696]: E1108 23:20:35.705796     696 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:413: Failed to list *v1.Service: Get https://45.33.92.201:443/api/v1/services?resourceVersion=0: x509: certificate has expired or is not yet valid
Nov 08 23:20:35 localhost kubelet-wrapper[696]: E1108 23:20:35.714885     696 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:422: Failed to list *v1.Node: Get https://45.33.92.201:443/api/v1/nodes?fieldSelector=metadata.name%3D45.33.92.201&resourceVersion=0: x509: certificate has expired or is not yet valid
Nov 08 23:20:36 localhost kubelet-wrapper[696]: E1108 23:20:36.198351     696 kubelet_node_status.go:390] Error updating node status, will retry: error getting node "45.33.92.201": Get https://45.33.92.201:443/api/v1/nodes/45.33.92.201?resourceVersion=0: x509: certificate has expired or is not yet valid
Nov 08 23:20:36 localhost kubelet-wrapper[696]: E1108 23:20:36.207936     696 kubelet_node_status.go:390] Error updating node status, will retry: error getting node "45.33.92.201": Get https://45.33.92.201:443/api/v1/nodes/45.33.92.201: x509: certificate has expired or is not yet valid
Nov 08 23:20:36 localhost kubelet-wrapper[696]: E1108 23:20:36.216213     696 kubelet_node_status.go:390] Error updating node status, will retry: error getting node "45.33.92.201": Get https://45.33.92.201:443/api/v1/nodes/45.33.92.201: x509: certificate has expired or is not yet valid

[displague]

@kahkhang Is there a way to apply an upgrade/update like this to an existing k8s cluster? A broader version ofkubectl replace perhaps? (If so -- that would be a nice README.md addition)

[kahkhang]

Hi @displague, unfortunately there is no such tool (unless you use an enterprise one such as the CoreOS tectonic installer which does rolling upgrades), and the process is more complicated than I would like it to be. The cluster comprises of several services, and I am using the design here https://github.com/kubernetes/community/blob/master/contributors/design-proposals/cluster-lifecycle/self-hosted-kubernetes.md which makes it slightly easier to upgrade. In short, the kubernetes manifests for the core services need to have their version bumped up, then the kubelet.service systemd service needs to have it's version bumped up too. This is assuming there's no backward incompatibility with other services, such as etcd. This README describes how you can go about doing it https://github.com/kubernetes-incubator/bootkube/blob/master/Documentation/upgrading.md (I'm using bootkube to render all the initial assets)

[displague]

I walked through those steps and now I'm pretty sure I have randomly updated parts of the cluster.. I'm just going to redeploy as I didn't get past the Rook problems completely on the first pass.

Does this command offer salvation? (In terms of future updates to this repo including a means to update the deployed cluster with the lastest values)

kubectl rolling-update -h
# Examples:
# Update pods of frontend-v1 using new replication controller data in frontend-v2.json.
#  kubectl rolling-update frontend-v1 -f frontend-v2.json

[kahkhang]

Hmm if I'm not wrong kubectl deploy is a newer command that does a rolling upgrade too. Unfortunately rook uses a kubernetes operator so some of the cluster components (such as monitors) might not be backward compatible: https://github.com/rook/rook/blob/master/design/upgrade.md.

I think a rolling update can possibly be supported in the future when rook stops releases breaking changes, though I think such updates might be breaking and be outside of the cluster as well. For example, the latest update involved changing the kubelet.service and etcd.service, which is outside of the kubernetes cluster. I suppose a bash script to manually do an upgrade is still possible (probably what they are doing in enterprise kubernetes upgrades), but such efforts seem time consuming, and difficult to support in this repo.