search

DataONEorg / api-entrypoint

The DataONE Kubernetes cluster's API ingress controller component that is shared between microservices
0 stars 1 forks source link

Upgrade k8s on dev, then production #6

Closed gothub closed 3 years ago

gothub commented 3 years ago

Upgrade k8s on the dev k8s cluster then the production k8s to the current version which is v1.19.0 (https://kubernetes.io/docs/setup/release/notes/)

Currently the k8s version running on production is:

metadig@docker-ucsb-4:~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:14:22Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:07:13Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}

and the version running on dev is:

metadig@docker-dev-ucsb-1:~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:14:22Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89", GitTreeState:"clean", BuildDate:"2020-01-18T23:22:30Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}

Note that it may be necessary to upgrade first to 1.18.x, then to 1.19.0. This process will be fully tested on the dev k8s, then production, so that downtime is minimized. In the past, these upgrades take 10-15 minutes, unless complications are encountered.

gothub commented 3 years ago

Note that as part of the upgrade, dependent ubuntu components should be updated first:

$ apt-mark showhold
containerd.io
docker-ce
docker-ce-cli
kubeadm
kubectl
kubelet
kubernetes-cni
gothub commented 3 years ago

For the dev upgrade, continue to k8s v1.20.x, then try a rolling ugrade to 1.21.x.

It will be necessary to have 2+ worker nodes to perform the rolling update, however, currently there is only one.

gothub commented 3 years ago

The dev k8s cluster has been upgraded to k8s v1.20.6 on 2021 04 19.

gothub commented 3 years ago

The production k8s cluster has been updated to k8s v1.20.6 as well. Calico 3.19.1 (most current version) has also been installed on dev and prod k8s.