k8s admin config context not authorized

[gothub]

Sending the following command to prod k8s produces an error:

$ kubectl get pods,services --all-namespaces -o wide
error: You must be logged in to the server (Unauthorized)

The admin context is currently being used:

$ kubectl config get-contexts
CURRENT   NAME                          CLUSTER      AUTHINFO           NAMESPACE
          gnis                          kubernetes   gnis               gnis
*         kubernetes-admin@kubernetes   kubernetes   kubernetes-admin

[gothub]

The k8s certificates appear to be up-to-date:

$ sudo kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W0223 09:28:54.897792    8033 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Aug 18, 2022 23:01 UTC   176d            ca                      no
apiserver                  Aug 18, 2022 22:59 UTC   176d            ca                      no
apiserver-etcd-client      Aug 18, 2022 22:59 UTC   176d            etcd-ca                 no
apiserver-kubelet-client   Aug 18, 2022 22:59 UTC   176d            ca                      no
controller-manager.conf    Aug 18, 2022 23:00 UTC   176d            ca                      no
etcd-healthcheck-client    Aug 18, 2022 22:58 UTC   176d            etcd-ca                 no
etcd-peer                  Aug 18, 2022 22:58 UTC   176d            etcd-ca                 no
etcd-server                Aug 18, 2022 22:58 UTC   176d            etcd-ca                 no
front-proxy-client         Aug 18, 2022 22:59 UTC   176d            front-proxy-ca          no
scheduler.conf             Aug 18, 2022 23:00 UTC   176d            ca                      no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Feb 15, 2030 21:01 UTC   7y              no
etcd-ca                 Feb 15, 2030 21:01 UTC   7y              no
front-proxy-ca          Feb 15, 2030 21:01 UTC   7y              no

[gothub]

The authorization problem was a result of /etc/kubernetes/admin.conf being updated. This file contains the client-certificate-data and the client-key-data values for the kubernetes-admin context, which is also in the ~/.kube/config file for admin users.

This file may have been updated when kubeadm was updated via linux system upgrades. Typically this package has a hold on it so that it is not upgraded during routine OS upgrades.

This was an opportune moment to upgrade k8s, so prod k8s has been upgraded to v1.23.4, which is the most recent.

Note that the new values for the admin kubectl config context will be updated in the NCEAS secure repo.

[amoeba]

Hey @gothub, did the config in the secure repo get updated yet?