Incorrect software repo config on k8s prod

[gothub]

@nickatnceas has indicated that the Linux software package repo configuration on the docker-ucsb-4,5,6,7 contain older repo definitions (Ubuntu 16.04, 16.10) than the current os version (18.04):

An abbreviation convo from Slack, that summarizes the issue:

I’m noticing that docker-ucsb-6 and 7 have K8s apt package repos enabled for Ubuntu xenial (16.04) and yakkety (16.10), but we’re running bionic (18.04)
... sometimes that make a difference, if the packages are built against different libraries than what the OS provides
... docker-ucsb-5 has the two incorrect versions as well
... same with docker-ucsb-4
... [ to resolve, possibly ] delete the duplicate entry, change the remaining entry to bionic, then upgrade the packages with apt dist-upgrade or something similar. Definitely want to test on dev first, in case something breaks
... you’re telling apt to only usage packages built for 16.04 and 16.10, but we’re running 18.04

This is confirmed with:

metadig@docker-ucsb-4:~$ apt list | grep kube

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

cri-tools/kubernetes-xenial,now 1.13.0-01 amd64 [installed,automatic]
docker-engine/kubernetes-xenial 1.11.2-0~xenial amd64
golang-github-kubernetes-gengo-dev/bionic,bionic,bionic,bionic 0.0~git20170531.0.c79c13d-1 all
kubeadm/kubernetes-xenial 1.22.1-00 amd64 [upgradable from: 1.22.0-00]
kubectl/kubernetes-xenial 1.22.1-00 amd64 [upgradable from: 1.22.0-00]
kubelet/kubernetes-xenial 1.22.1-00 amd64 [upgradable from: 1.22.0-00]
kubernetes-cni/kubernetes-xenial,now 0.8.7-00 amd64 [installed,automatic]
python-kubernetes/bionic,bionic,bionic,bionic 2.0.0-2ubuntu1 all
python3-kubernetes/bionic,bionic,bionic,bionic 2.0.0-2ubuntu1 all
ruby-kubeclient/bionic,bionic,bionic,bionic 3.0.0-1 all
salt-formula-kubernetes/bionic,bionic,bionic,bionic 2016.12.1-1ubuntu1 all

Notice the xenial packages, that should be bionic.

[gothub]

The 'xenial' repo is actually the most recent one available for kubernetes distributions. The repo name 'bionic' (Ubuntu 18.04) should be used, but for whatever reason it is not available, and 'xenial' is the most recent repo. See https://packages.cloud.google.com/apt/dists and search for 'kubernetes-'.

Looking in https://packages.cloud.google.com/apt/dists/kubernetes-xenial/Release shows that this version has been built very recently - "Date: Wed, 15 Sep 2021 23:58:22 UTC".

So, I'll keep a lookout for a newer repo the next time k8s is updated, but for now 'xenial' will be used.

[nickatnceas]

It looks like they are using "Xenial" as a generic deb repository: https://github.com/kubernetes/kubernetes/issues/66300

Some of our servers have both xenial (16.04) and yakkety (16.10) repos enabled, while others only have xenial enabled. We should probably get rid of the Yakkety repos.

outin@docker-ucsb-5:~$ sudo apt update | grep kubernetes
Hit:6 https://packages.cloud.google.com/apt kubernetes-yakkety InRelease
Hit:7 https://packages.cloud.google.com/apt kubernetes-xenial InRelease

[gothub]

@nickatnceas yes, good idea. I'll go through all the docker-ucsb- and docker-dev-ucsb- nodes and remove the yakkety references.

[gothub]

The following nodes have been update to use only 'deb https://apt.kubernetes.io/ kubernetes-xenial main' as the kubernetes distribution:

• docker-ucsb-4
• docker-ucsb-5
• docker-ucsb-6
• docker-ucsb-7
• docker-dev-ucsb-1
• docker-dev-ucsb-2