search

RedHat-EMEA-SSA-Team / hetzner-ocp4

Installing OCP 4 on single bare metal server.
Apache License 2.0
183 stars 114 forks source link

401 (Unauthorized) When trying to login to OpenShift #301

Closed kb-namatedev closed 10 months ago

kb-namatedev commented 10 months ago

When I try to connect to Openshift from the web console. It keeps on refreshing the page on the same error:

Could not get OpenAPI definitions a: Unauthorized POST https://console-openshift-console.apps.{domain}/api/kubernetes/apis/authorization.k8s.io/v1/selfsubjectaccessreviews 401 (Unauthorized) GET https://console-openshift-console.apps.digisntuat.namategroup.com/api/kubernetes/openapi/v2 401 (Unauthorized)

Even when I try using oc login from my local machine I get the following error: Login failed (401 Unauthorized) Verify you have provided correct credentials.

I have verified the credentials and they are correct.

Everything was working fine for about 1 year now until it stopped working.

rbo commented 10 months ago

I don't know, all VMs are running? Is disk space available? What If you try oc get nodes, oc get co with the provided kubeconfig ?

kb-namatedev commented 10 months ago

All VMs are running:

[root@CentOS-80-stream-amd64-base ~]# virsh list --all
 Id   Name                   State
--------------------------------------
 2    ocp4-master-0    running
 3    ocp4-master-1    running
 4    ocp4-master-2    running
 5    ocp4-compute-0   running
 6    ocp4-compute-1   running

I have 71G Disk space available.

oc get nodes, oc get co works with export KUBECONFIG=hetzner-ocp4/ocp4/auth/kubeconfig

[root@CentOS-80-stream-amd64-base ~]# oc get nodes
NAME        STATUS   ROLES    AGE    VERSION
compute-0   Ready    worker   328d   v1.23.5+3afdacb
compute-1   Ready    worker   328d   v1.23.5+3afdacb
master-0    Ready    master   328d   v1.23.5+3afdacb
master-1    Ready    master   328d   v1.23.5+3afdacb
master-2    Ready    master   328d   v1.23.5+3afdacb
[root@CentOS-80-stream-amd64-base ~]# oc get co
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.10.16   True        False         False      12h
baremetal                                  4.10.16   True        False         False      328d
cloud-controller-manager                   4.10.16   True        False         False      328d
cloud-credential                           4.10.16   True        False         False      328d
cluster-autoscaler                         4.10.16   True        False         False      328d
config-operator                            4.10.16   True        False         False      328d
console                                    4.10.16   True        False         False      12h
csi-snapshot-controller                    4.10.16   True        False         False      328d
dns                                        4.10.16   True        False         False      328d
etcd                                       4.10.16   True        False         False      328d
image-registry                             4.10.16   True        False         False      328d
ingress                                    4.10.16   True        False         False      328d
insights                                   4.10.16   True        False         False      328d
kube-apiserver                             4.10.16   True        True          True       328d    InstallerPodContainerWaitingDegraded: Pod "installer-86-master-2" on node "master-2" container "installer" is waiting since 2023-11-13 23:22:09 +0000 UTC because ContainerCreating...
kube-controller-manager                    4.10.16   True        False         False      328d
kube-scheduler                             4.10.16   True        False         False      328d
kube-storage-version-migrator              4.10.16   True        False         False      328d
machine-api                                4.10.16   True        False         False      328d
machine-approver                           4.10.16   True        False         False      328d
machine-config                             4.10.16   True        False         False      328d
marketplace                                4.10.16   True        False         False      328d
monitoring                                 4.10.16   False       True          True       42d     Rollout of the monitoring stack failed and is degraded. Please investigate the degraded status error.
network                                    4.10.16   True        False         False      328d
node-tuning                                4.10.16   True        False         False      328d
openshift-apiserver                        4.10.16   True        False         False      328d
openshift-controller-manager               4.10.16   False       True          False      41d     Available: no daemon pods available on any node.
openshift-samples                          4.10.16   True        False         False      328d
operator-lifecycle-manager                 4.10.16   True        False         False      328d
operator-lifecycle-manager-catalog         4.10.16   True        False         False      328d
operator-lifecycle-manager-packageserver   4.10.16   True        False         False      328d
service-ca                                 4.10.16   True        False         False      328d
storage
rbo commented 10 months ago

Mh looks not too bad. I don't know to be honest.

kb-namatedev commented 10 months ago

Do you think it might be related to this error on co kube-apiserver:

Message:               InstallerPodContainerWaitingDegraded: Pod "installer-86-master-2" on node "master-2" container "installer" is waiting since 2023-11-13 23:22:09 +0000 UTC because ContainerCreating
InstallerPodNetworkingDegraded: Pod "installer-86-master-2" on node "master-2" observed degraded networking: (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-86-master-2_openshift-kube-apiserver_26ab6ca4-eb80-4001-bd5e-4d12a453ae35_0(fd3a2ec1f495cc5c6bd6014223deed5ae5c7814e98d500e5e1b9841a8146b87e): error adding pod openshift-kube-apiserver_installer-86-master-2 to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [openshift-kube-apiserver/installer-86-master-2/26ab6ca4-eb80-4001-bd5e-4d12a453ae35:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[openshift-kube-apiserver/installer-86-master-2 fd3a2ec1f495cc5c6bd6014223deed5ae5c7814e98d500e5e1b9841a8146b87e] [openshift-kube-apiserver/installer-86-master-2 fd3a2ec1f495cc5c6bd6014223deed5ae5c7814e98d500e5e1b9841a8146b87e] failed to configure pod interface: timed out waiting for OVS port binding (ovn-installed) for 0a:58:0a:82:00:da [10.130.0.218/23]
InstallerPodNetworkingDegraded: '
    Reason:                InstallerPodContainerWaiting_ContainerCreating::InstallerPodNetworking_FailedCreatePodSandBox
    Status:                True
    Type:                  Degraded
    Last Transition Time:  2023-09-29T23:51:09Z
    Message:               NodeInstallerProgressing: 3 nodes are at revision 76; 0 nodes have achieved new revision 86
    Reason:                NodeInstaller
    Status:                True
    Type:                  Progressing
[root@CentOS-80-stream-amd64-base ~]# oc describe pod installer-86-master-2
Name:                 installer-86-master-2
Namespace:            openshift-kube-apiserver
Priority:             2000001000
Priority Class Name:  system-node-critical
Node:                 master-2/192.168.50.12
Start Time:           Tue, 14 Nov 2023 00:22:09 +0100
Labels:               app=installer
.....
Events:
  Type     Reason                  Age                    From     Message
  ----     ------                  ----                   ----     -------
  Warning  FailedCreatePodSandBox  5m7s (x1197 over 40h)  kubelet  (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-86-master-2_openshift-kube-apiserver_26ab6ca4-eb80-4001-bd5e-4d12a453ae35_0(1ebb2d89edbab36463bb7bc5cb644d8b919465067ae1cdbce3e0c67a9324cdc8): error adding pod openshift-kube-apiserver_installer-86-master-2 to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [openshift-kube-apiserver/installer-86-master-2/26ab6ca4-eb80-4001-bd5e-4d12a453ae35:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[openshift-kube-apiserver/installer-86-master-2 1ebb2d89edbab36463bb7bc5cb644d8b919465067ae1cdbce3e0c67a9324cdc8] [openshift-kube-apiserver/installer-86-master-2 1ebb2d89edbab36463bb7bc5cb644d8b919465067ae1cdbce3e0c67a9324cdc8] failed to configure pod interface: timed out waiting for OVS port binding (ovn-installed) for 0a:58:0a:82:00:da [10.130.0.218/23]
'
kb-namatedev commented 10 months ago

Solved by restarting the cluster.