search

crc-org / crc

CRC is a tool to help you run containers. It manages a local OpenShift 4.x cluster, Microshift or a Podman VM optimized for testing and development purposes
https://crc.dev
Apache License 2.0
1.25k stars 236 forks source link

[BUG] Openshift always in starting. Openshift console failing #4321

Closed joaofeteira closed 1 month ago

joaofeteira commented 1 month ago

General information

CRC version

CRC version: 2.40.0+a14925
OpenShift version: 4.16.4
MicroShift version: 4.16.4

CRC status

DEBU CRC version: 2.40.0+a14925
DEBU OpenShift version: 4.16.4
DEBU MicroShift version: 4.16.4
DEBU Running 'crc status'
CRC VM:          Running
OpenShift:       Starting (v4.15.0-0.okd-2024-02-23-163410)
RAM Usage:       7.083GB of 20.96GB
Disk Usage:      27.86GB of 47.71GB (Inside the CRC VM)
Cache Usage:     27.69GB
Cache Directory: /home/dsmlab/.crc/cache

CRC config

- consent-telemetry                     : no
- cpus                                  : 8
- disk-size                             : 45
- enable-cluster-monitoring             : true
- memory                                : 20480
- preset                                : okd

Host Operating System

NAME="Rocky Linux"
VERSION="8.9 (Green Obsidian)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.9"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Rocky Linux 8.9 (Green Obsidian)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:8:GA"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
SUPPORT_END="2029-05-31"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-8"
ROCKY_SUPPORT_PRODUCT_VERSION="8.9"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.9"

Steps to reproduce

  1. crc setup
  2. crc start
  3. Web Gui doesn't work and Openshift is always "Starting" in crc status

Expected

Openshift Web GUI working

Actual

Openshift console not working but cli commands seem ok

Noticed this on the openshift console pods:

 oc get pods -n openshift-console
NAME                         READY   STATUS             RESTARTS         AGE
console-899b78946-xrxvq      0/1     CrashLoopBackOff   16 (4m12s ago)   62m
console-b4955768b-v2zl5      0/1     CrashLoopBackOff   16 (119s ago)    60m
downloads-855978b7d5-j8ksf   1/1     Running            1 (63m ago)      167d

oc describe pods console-899b78946-xrxvq -n  openshift-console
Name:                 console-899b78946-xrxvq
Namespace:            openshift-console
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Service Account:      console
Node:                 crc/192.168.126.11
Start Time:           Mon, 19 Aug 2024 14:29:55 +0100
Labels:               app=console
                      component=ui
                      pod-template-hash=899b78946
Annotations:          console.openshift.io/console-config-version: 37387
                      console.openshift.io/image: quay.io/openshift/okd-content@sha256:4077bb57db7936261f0f689d9a31282e9b203cd207b7356dcfe7e03f772e56ef
                      console.openshift.io/infrastructure-config-version: 441
                      console.openshift.io/oauth-secret-version: 12457
                      console.openshift.io/oauth-serving-cert-config-version: 12453
                      console.openshift.io/proxy-config-version: 457
                      console.openshift.io/service-ca-config-version: 13031
                      console.openshift.io/trusted-ca-config-version: 13146
                      k8s.ovn.org/pod-networks:
                        {"default":{"ip_addresses":["10.217.0.54/23"],"mac_address":"0a:58:0a:d9:00:36","gateway_ips":["10.217.0.1"],"routes":[{"dest":"10.217.0.0...
                      k8s.v1.cni.cncf.io/network-status:
                        [{
                            "name": "ovn-kubernetes",
                            "interface": "eth0",
                            "ips": [
                                "10.217.0.54"
                            ],
                            "mac": "0a:58:0a:d9:00:36",
                            "default": true,
                            "dns": {}
                        }]
                      openshift.io/scc: restricted-v2
                      seccomp.security.alpha.kubernetes.io/pod: runtime/default
Status:               Running
SeccompProfile:       RuntimeDefault
IP:                   10.217.0.54
IPs:
  IP:           10.217.0.54
Controlled By:  ReplicaSet/console-899b78946
Containers:
  console:
    Container ID:  cri-o://59512d72fe62a468228e8f1d207c673a75f13873816d1f7c07a475bb059fe5f1
    Image:         quay.io/openshift/okd-content@sha256:4077bb57db7936261f0f689d9a31282e9b203cd207b7356dcfe7e03f772e56ef
    Image ID:      f465e1bf43b7dd3473709cdd0b72fc89ffccf4120771a49c25f185d59aed99b0
    Port:          8443/TCP
    Host Port:     0/TCP
    Command:
      /opt/bridge/bin/bridge
      --public-dir=/opt/bridge/static
      --config=/var/console-config/console-config.yaml
      --service-ca-file=/var/service-ca/service-ca.crt
      --v=2
    State:       Waiting
      Reason:    CrashLoopBackOff
    Last State:  Terminated
      Reason:    Error
      Message:   I0819 14:33:33.868597       1 main.go:210] The following console plugins are enabled:
I0819 14:33:33.868726       1 main.go:212]  - monitoring-plugin
W0819 14:33:33.868780       1 authoptions.go:99] Flag inactivity-timeout is set to less then 300 seconds and will be ignored!
F0819 14:33:38.900050       1 authoptions.go:286] Error initializing authenticator: failed to construct OAuth endpoint cache: failed to setup an async cache - caching func returned error: request to OAuth issuer endpoint https://oauth-openshift.apps-crc.testing/oauth/token failed: Head "https://oauth-openshift.apps-crc.testing": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

      Exit Code:    255
      Started:      Mon, 19 Aug 2024 15:33:33 +0100
      Finished:     Mon, 19 Aug 2024 15:33:38 +0100
    Ready:          False
    Restart Count:  17
    Requests:
      cpu:      10m
      memory:   100Mi
    Liveness:   http-get https://:8443/health delay=150s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get https://:8443/health delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:  console-899b78946-xrxvq (v1:metadata.name)
    Mounts:
      /etc/pki/ca-trust/extracted/pem from trusted-ca-bundle (ro)
      /var/console-config from console-config (ro)
      /var/oauth-config from console-oauth-config (ro)
      /var/oauth-serving-cert from oauth-serving-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-f8n9s (ro)
      /var/service-ca from service-ca (ro)
      /var/serving-cert from console-serving-cert (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  console-serving-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  console-serving-cert
    Optional:    false
  console-oauth-config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  console-oauth-config
    Optional:    false
  console-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      console-config
    Optional:  false
  service-ca:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      service-ca
    Optional:  false
  trusted-ca-bundle:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      trusted-ca-bundle
    Optional:  false
  oauth-serving-cert:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      oauth-serving-cert
    Optional:  false
  kube-api-access-f8n9s:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
    ConfigMapName:           openshift-service-ca.crt
    ConfigMapOptional:       <nil>
QoS Class:                   Burstable
Node-Selectors:              node-role.kubernetes.io/master=
Tolerations:                 node-role.kubernetes.io/master:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 120s
Events:
  Type     Reason          Age                From               Message
  ----     ------          ----               ----               -------
  Normal   Scheduled       64m                default-scheduler  Successfully assigned openshift-console/console-899b78946-xrxvq to crc
  Normal   AddedInterface  64m                multus             Add eth0 [10.217.0.54/23] from ovn-kubernetes
  Warning  ProbeError      64m (x5 over 64m)  kubelet            Readiness probe error: Get "https://10.217.0.54:8443/health": dial tcp 10.217.0.54:8443: connect: connection refused
body:
  Warning  Unhealthy  64m (x5 over 64m)     kubelet  Readiness probe failed: Get "https://10.217.0.54:8443/health": dial tcp 10.217.0.54:8443: connect: connection refused
  Normal   Started    63m (x3 over 64m)     kubelet  Started container console
  Normal   Pulled     63m (x4 over 64m)     kubelet  Container image "quay.io/openshift/okd-content@sha256:4077bb57db7936261f0f689d9a31282e9b203cd207b7356dcfe7e03f772e56ef" already present on machine
  Normal   Created    63m (x4 over 64m)     kubelet  Created container console
  Warning  BackOff    4m4s (x279 over 64m)  kubelet  Back-off restarting failed container console in pod console-899b78946-xrxvq_openshift-console(e87fecf9-2efb-4109-97c9-25a6de59700d)

Already tried to launch the VM from scratch multiple times with no success.

Logs

Before gather the logs try following if that fix your issue

$ crc delete -f
$ crc cleanup
$ crc setup
$ crc start --log-level debug

Please consider posting the output of crc start --log-level debug on http://gist.github.com/ and post the link in the issue.

Part 1 of crc start --log: https://gist.github.com/joaofeteira/e192795cd17ca57a14296783a8143c1f

Part 2 of crc start --log: https://gist.github.com/joaofeteira/e15b8d7562539ef1a244c64e0b42dd86

praveenkumar commented 1 month ago

This looks like nested setup, can you try to run it this directly on the host instead of this nested setup?

There is lot of issues due to nested virt setup https://github.com/crc-org/crc/labels/nested-virt and we don't have infra to test all the possible scenario.

joaofeteira commented 1 month ago

RIght now I can't because i need multiple of these setups hence the virtualization. I have all the vt-x etc flags enabled and it also worked in the past (sometimes I had to relaunch it but it worked). But thanks for the help nevertheless

praveenkumar commented 1 month ago

I can see you used OKD preset, can you also try with OCP because OKD one is a old version and new is not yet available upstream also.

joaofeteira commented 1 month ago

yes I used it because i wanted the OKD one. with OCP you need the Redhat key i think right?

praveenkumar commented 1 month ago

yes I used it because i wanted the OKD one. with OCP you need the Redhat key i think right?

You need pull secret which you can get it from https://console.redhat.com/openshift/create/local here.

joaofeteira commented 1 month ago

yes indeed but for automation it doesn't work ok. nvm I will try another setup. thanks for the help nevertheless