search

stormshift / support

This repo should serve as a central source for reporting issues with stormshift
GNU General Public License v3.0
3 stars 0 forks source link

OCP4 not available #85

Closed Javatar81 closed 2 years ago

Javatar81 commented 2 years ago

API not available oc get nodes => no response Found the following in control-o with journalctl -b -f -u kubelet.service -u crio.service

Jun 02 08:18:28 control-0.ocp4.stormshift.coe.muc.redhat.com hyperkube[44497]: E0602 08:18:28.938422 44497 server.go:292] "Failed to run kubelet" err="failed to run Kubelet: failed to initialize client certificate manager: could not convert data from \"/var/lib/kubelet/pki/kubelet-client-current.pem\" into cert/key pair: tls: failed to find any PEM data in certificate input"

Javatar81 commented 2 years ago

Similar issue: https://github.com/kubernetes/kubernetes/issues/65991

Javatar81 commented 2 years ago

Moved mv /var/lib/kubelet/pki/kubelet-client-current.pem /var/lib/kubelet/pki/kubelet-client-current.pem_bak and restarted kubelet.

Javatar81 commented 2 years ago 
[root@ocp4bastion ~]# oc get nodes
NAME                                           STATUS     ROLES    AGE      VERSION
compute-0.ocp4.stormshift.coe.muc.redhat.com   Ready      worker   2y136d   v1.21.8+ee73ea2
compute-1.ocp4.stormshift.coe.muc.redhat.com   Ready      worker   2y136d   v1.21.8+ee73ea2
compute-2.ocp4.stormshift.coe.muc.redhat.com   Ready      worker   600d     v1.21.8+ee73ea2
control-0.ocp4.stormshift.coe.muc.redhat.com   NotReady   master   2y136d   v1.21.8+ee73ea2
control-1.ocp4.stormshift.coe.muc.redhat.com   Ready      master   2y136d   v1.21.8+ee73ea2
control-2.ocp4.stormshift.coe.muc.redhat.com   NotReady   master   2y136d   v1.21.8+ee73ea2
gpu.ocp4.stormshift.coe.muc.redhat.com         NotReady   worker   207d     v1.21.8+ee73ea2
Javatar81 commented 2 years ago

Recovered kubelet and approved csr

Javatar81 commented 2 years ago 
oc get nodes                                                                                                        ocp4bastion.stormshift.coe.muc.redhat.com: Thu Jun  2 10:50:05 2022

NAME                                           STATUS                     ROLES    AGE      VERSION
compute-0.ocp4.stormshift.coe.muc.redhat.com   Ready                      worker   2y136d   v1.21.8+ee73ea2
compute-1.ocp4.stormshift.coe.muc.redhat.com   Ready                      worker   2y136d   v1.21.8+ee73ea2
compute-2.ocp4.stormshift.coe.muc.redhat.com   Ready                      worker   600d     v1.21.8+ee73ea2
control-0.ocp4.stormshift.coe.muc.redhat.com   Ready                      master   2y136d   v1.21.8+ee73ea2
control-1.ocp4.stormshift.coe.muc.redhat.com   Ready                      master   2y136d   v1.21.8+ee73ea2
control-2.ocp4.stormshift.coe.muc.redhat.com   Ready,SchedulingDisabled   master   2y136d   v1.21.8+ee73ea2
gpu.ocp4.stormshift.coe.muc.redhat.com         NotReady                   worker   207d     v1.21.8+ee73ea2
Javatar81 commented 2 years ago

Control 0 and Control 2 were rebooted automatically. After reboot:

oc get nodes                                                                                                        ocp4bastion.stormshift.coe.muc.redhat.com: Thu Jun  2 10:52:33 2022

NAME                                           STATUS     ROLES    AGE      VERSION
compute-0.ocp4.stormshift.coe.muc.redhat.com   Ready      worker   2y136d   v1.21.8+ee73ea2
compute-1.ocp4.stormshift.coe.muc.redhat.com   Ready      worker   2y136d   v1.21.8+ee73ea2
compute-2.ocp4.stormshift.coe.muc.redhat.com   Ready      worker   600d     v1.21.8+ee73ea2
control-0.ocp4.stormshift.coe.muc.redhat.com   Ready      master   2y136d   v1.21.8+ee73ea2
control-1.ocp4.stormshift.coe.muc.redhat.com   Ready      master   2y136d   v1.21.8+ee73ea2
control-2.ocp4.stormshift.coe.muc.redhat.com   Ready      master   2y136d   v1.21.8+ee73ea2
gpu.ocp4.stormshift.coe.muc.redhat.com         NotReady   worker   207d     v1.21.8+ee73ea2