search

crc-org / crc

CRC is a tool to help you run containers. It manages a local OpenShift 4.x cluster, Microshift or a Podman VM optimized for testing and development purposes
https://crc.dev
Apache License 2.0
1.26k stars 241 forks source link

CRC 4.8.2 status showing Degraded at the startup #2628

Closed rupeshad81 closed 3 years ago

rupeshad81 commented 3 years ago

General information

CRC version

# Put `crc version` output here
[crc@eaas-crc ~]$ crc version
CodeReady Containers version: 1.30.1+376cc3c1
OpenShift version: 4.8.2 (embedded in executable)

## CRC status
```bash
# Put `crc status --log-level debug` output here
[crc@eaas-crc ~]$ crc status
CRC VM:          Running
OpenShift:       Degraded (v4.8.2)
Disk Usage:      15.19GB of 32.74GB (Inside the CRC VM)
Cache Usage:     17.27GB
Cache Directory: /home/crc/.crc/cache

## CRC config
```bash
# Put `crc config view` output here
[crc@eaas-crc ~]$ crc config view
- consent-telemetry                     : no
- http-proxy                            : http://10.232.233.70:8080
- https-proxy                           : http://10.232.233.70:8080
- no-proxy                              : 127.0.0.1,localhost,.corp.amdocs.com,.apps-crc.testing,.crc.testing

## Host Operating System
```bash
# Put the output of `cat /etc/os-release` in case of Linux
# put the output of `sw_vers` in case of Mac
# Put the output of `systeminfo` in case of Windows

For crc 4.8.2 , when started  getting in the "Degraded" status. why it showing this status, it should show  "Running" status. I had restarted many times but same status showing "Degraded".

### Steps to reproduce

  1.  CRC installed , after the crc status was showing the " Degraded"
  2. [crc@eaas-crc ~]$ crc start --cpus 12 --memory 40000
INFO Checking if running as non-root
INFO Checking if running inside WSL2
INFO Checking if crc-admin-helper executable is cached
INFO Checking for obsolete admin-helper executable
INFO Checking if running on a supported CPU architecture
INFO Checking minimum RAM requirements
INFO Checking if Virtualization is enabled
INFO Checking if KVM is enabled
INFO Checking if libvirt is installed
INFO Checking if user is part of libvirt group
INFO Checking if active user/process is currently part of the libvirt group
INFO Checking if libvirt daemon is running
INFO Checking if a supported libvirt version is installed
INFO Checking if crc-driver-libvirt is installed
INFO Checking if systemd-networkd is running
INFO Checking if NetworkManager is installed
INFO Checking if NetworkManager service is running
INFO Checking if /etc/NetworkManager/conf.d/crc-nm-dnsmasq.conf exists
INFO Checking if /etc/NetworkManager/dnsmasq.d/crc.conf exists
INFO Checking if libvirt 'crc' network is available
INFO Checking if libvirt 'crc' network is active
INFO Starting CodeReady Containers VM for OpenShift 4.8.2...
INFO CodeReady Containers instance is running with IP 192.168.130.11
INFO CodeReady Containers VM is running
INFO Check internal and public DNS query...
INFO Check DNS query from host...
INFO Adding proxy configuration to kubelet and crio service...
INFO Verifying validity of the kubelet certificates...
INFO Starting OpenShift kubelet service
INFO Waiting for kube-apiserver availability... [takes around 2min]
INFO Waiting for user's pull secret part of instance disk...
INFO Adding proxy configuration to the cluster...
INFO Starting OpenShift cluster... [waiting for the cluster to stabilize]
INFO Operator network is progressing
INFO Operator openshift-apiserver is not yet available
INFO Operator openshift-apiserver is not yet available
INFO Operator openshift-apiserver is not yet available
INFO Operator openshift-apiserver is not yet available
INFO Operator openshift-apiserver is degraded
INFO Operator openshift-apiserver is degraded
INFO Operator openshift-apiserver is degraded
INFO Operator openshift-apiserver is degraded
INFO Operator openshift-apiserver is degraded
INFO Operator openshift-apiserver is degraded
INFO Operator openshift-apiserver is degraded
INFO Operator openshift-apiserver is degraded
INFO Operator openshift-apiserver is degraded
INFO Operator openshift-apiserver is degraded
INFO Operator openshift-apiserver is degraded
INFO Operator openshift-apiserver is degraded
INFO Operator openshift-apiserver is degraded
INFO Operator openshift-apiserver is degraded
INFO Operator openshift-apiserver is degraded
ERRO Cluster is not ready: cluster operators are still not stable after 10m1.141047076s
INFO Waiting for the proxy configuration to be applied...
INFO Adding crc-admin and crc-developer contexts to kubeconfig...
ERRO Cannot update kubeconfig: Head "https://oauth-openshift.apps-crc.testing": dial tcp 192.168.130.11:443: connect: connection refused
Started the OpenShift cluster.

The server is accessible via web console at:
  https://console-openshift-console.apps-crc.testing

Log in as administrator:
  Username: kubeadmin
  Password: ZVQuq-2kAUQ-zqweE-ADAma

Log in as user:
  Username: developer
  Password: developer

Use the 'oc' command line interface:
  $ eval $(crc oc-env)
  $ oc login -u developer https://api.crc.testing:6443
  2. [crc@eaas-crc ~]$ crc status
CRC VM:          Running
OpenShift:       Degraded (v4.8.2)
Disk Usage:      15.19GB of 32.74GB (Inside the CRC VM)
Cache Usage:     17.27GB
Cache Directory: /home/crc/.crc/cache
  3. 
  4. 

### Expected
[crc@eaas-crc ~]$ crc status
CRC VM:          Running
OpenShift:       Running (v4.8.2)
Disk Usage:      15.19GB of 32.74GB (Inside the CRC VM)
Cache Usage:     17.27GB
Cache Directory: /home/crc/.crc/cache

### Actual

### Logs

Before gather the logs try following if that fix your issue
```bash
$ crc delete -f
$ crc cleanup
$ crc setup
$ crc start --log-level debug

Please consider posting the output of crc start --log-level debug on http://gist.github.com/ and post the link in the issue.

rupeshad81 commented 3 years ago

CRC_4_8_2_Installation_Steps_03Aug2021.txt CRC installation steps has been send, please review it.

praveenkumar commented 3 years ago

@rupeshad81 Can you please provide following info also?

OS: Linux / macOS / Windows
Hypervisor: KVM / Hyper-V / hyperkit
Running CRC on: Laptop / Baremetal-Server / VM
rupeshad81 commented 3 years ago

Hi,

It is running on 7.6 redhat Linux Eaas VM.

Sent from Nine

From: Praveen Kumar @.***> Sent: Wednesday, August 4, 2021 9:34 PM To: code-ready/crc Cc: Rupesh Admane; Mention Subject: Re: [code-ready/crc] CRC 4.8.2 status showing Degraded at the startup (#2628)

@rupeshad81https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Frupeshad81&data=04%7C01%7CRupesh.Admane%40amdocs.com%7C2e1c99effef14711294308d957617caf%7Cc8eca3ca127646d59d9da0f2a028920f%7C0%7C0%7C637636898479712639%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=8FtqFgLLWeLsrxcFEsQulWesP7Nn8L2qbX17gg1Cdgg%3D&reserved=0 Can you please provide following info also?

OS: Linux / macOS / Windows Hypervisor: KVM / Hyper-V / hyperkit Running CRC on: Laptop / Baremetal-Server / VM

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcode-ready%2Fcrc%2Fissues%2F2628%23issuecomment-892781365&data=04%7C01%7CRupesh.Admane%40amdocs.com%7C2e1c99effef14711294308d957617caf%7Cc8eca3ca127646d59d9da0f2a028920f%7C0%7C0%7C637636898479722607%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=fYlfCzJX1wcdKI7KmFjr7CfgVTj84EOwnYH4mqC8pi4%3D&reserved=0, or unsubscribehttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAJQYKCNEWB62TET5TWVF57TT3FQFTANCNFSM5BRFUL5A&data=04%7C01%7CRupesh.Admane%40amdocs.com%7C2e1c99effef14711294308d957617caf%7Cc8eca3ca127646d59d9da0f2a028920f%7C0%7C0%7C637636898479722607%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=aP4HEyMrKd4YMVz3O%2BcOdDxkD6UL0nQA22RcueR3Ih8%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7CRupesh.Admane%40amdocs.com%7C2e1c99effef14711294308d957617caf%7Cc8eca3ca127646d59d9da0f2a028920f%7C0%7C0%7C637636898479732564%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=AarF51W54LuKxiAUjA3jSZrwAsWuaQzBASPZqhc4uJ4%3D&reserved=0 or Androidhttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26utm_campaign%3Dnotification-email&data=04%7C01%7CRupesh.Admane%40amdocs.com%7C2e1c99effef14711294308d957617caf%7Cc8eca3ca127646d59d9da0f2a028920f%7C0%7C0%7C637636898479742510%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=XmGb89c%2F1lzqVl9I8YskfPPam%2BGu8Ff6gES9%2Fv4dZgk%3D&reserved=0. This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at https://www.amdocs.com/about/email-disclaimer https://www.amdocs.com/about/email-disclaimer

Amdocs Development Centre India Private Limited having CIN: U72200PN2004PTC0188320 converted into Amdocs Development Centre India LLP (A limited liability partner­ship with LLP Identification Number: AAI-6901 effective 28th Feb 2017)

cfergeau commented 3 years ago

7.6 is probably too old, would be better if you can upgrade to latest RHEL7 release first. And running crc in a VM requires nested virtualization, and is strongly discouraged as it has caused very unexpected/non-obvious failures in the past. Do you have a way of trying this on real hardware to see if this works better?

rupeshad81 commented 3 years ago

Hi, We had deployed CRC 4.7.11 till on the same RHEL version 7.6 without any issue. I don't think this , it can OS version issue.

rupeshad81 commented 3 years ago

Hi,

Please can anyone assist urgently , we need to release to customer.

cfergeau commented 3 years ago

crc 1.30.1 creates OpenShift 4.8.2 clusters. OpenShift 4.8 comes with significant changes to the kubernetes/OpenShift layer, but also down to the low-level CoreOS/ostree layer, which is now derived from RHEL 8.4. These major changes might be uncovering new bugs in this platform of yours which were not showing before. Hard to tell if the issues you are having are due to your platform or to crc without a lot more investigation :(

My understanding is that this RHEL 7.6 instance is running in a VM? Can you provide details regarding what the L0 hypervisor is and what OS and CPU this is running on?

rupeshad81 commented 3 years ago

Hi, Yes RHEL is 7.6

[root@eaas-crc ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.6 (Maipo)

what do you mean by L0 hypervisor ?

[root@eaas-crc ~]# free -g
              total        used        free      shared  buff/cache   available
Mem:             39          19           8           0          11          19
Swap:             0           0           0
[root@eaas-crc ~]# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                12
On-line CPU(s) list:   0-11
Thread(s) per core:    1
Core(s) per socket:    1
Socket(s):             12
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 63
Model name:            Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
Stepping:              2
CPU MHz:               2497.108
BogoMIPS:              4994.21
Virtualization:        VT-x
Hypervisor vendor:     VMware
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              30720K
NUMA node0 CPU(s):     0-11
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc eagerfpu pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm ssbd ibrs ibpb stibp tpr_shadow vnmi ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid xsaveopt arat spec_ctrl intel_stibp flush_l1d arch_capabilities
praveenkumar commented 3 years ago

what do you mean by L0 hypervisor ?

@rupeshad81 On which host this RHEL VM is running?

cfergeau commented 3 years ago

what do you mean by L0 hypervisor ?

I was asking about the hardware which is running your RHEL 7.6 virtual machine. Hypervisor vendor: VMware means it's VMware, most likely running using Xeon CPUs (Model name: Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz)

rupeshad81 commented 3 years ago

Hi,

So which I had provided information is ok ?

Or Do need any more information. Thanks

cfergeau commented 3 years ago

There has been multiple issues related to VMware nested virtualization in our bug tracker https://github.com/code-ready/crc/issues?q=label%3Anested-virt , some of them went away after upgrading to latest RHEL. I'm afraid we don't support VMware nested virtualization, nor RHEL 7.6. If this is reproducible on a baremetal RHEL 7.9 install, it should be easier to investigate.

gbraad: changed link from https://github.com/code-ready/crc/issues?q=is%3Aissue+vmware to the nested-virt label

rupeshad81 commented 3 years ago

Hi,

Please can you share any RH release notes for CRC 1.30.1 creates OpenShift 4.8.2 , then we can check for compatibility for OS version. If we have 7.8 RHEL , then it ok.?

Thanks

gbraad commented 3 years ago

We do not have the --log-level debug. Can you provide this?

rupeshad81 commented 3 years ago

CRC_4_8_2_Installation_Debug_05Aug2021.txt

Please check startup of crc in debug mode.

rupeshad81 commented 3 years ago

CRC_4_8_2_Installation_OS_8_4_Debug_05Aug2021.txt

Hi,

I had same issue on the OS version : 8.4 

[crc@eaas-crc ~]$ cat /etc/redhat-release
Red Hat Enterprise Linux release 8.4 (Ootpa)
[crc@eaas-crc ~]$
[crc@eaas-crc ~]$ lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              14
On-line CPU(s) list: 0-13
Thread(s) per core:  1
Core(s) per socket:  1
Socket(s):           14
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               63
Model name:          Intel(R) Xeon(R) Gold 5220R CPU @ 2.20GHz
Stepping:            0
CPU MHz:             2194.844
BogoMIPS:            4389.68
Virtualization:      VT-x
Hypervisor vendor:   VMware
Virtualization type: full
L1d cache:           32K
L1i cache:           32K
L2 cache:            1024K
L3 cache:            36608K
NUMA node0 CPU(s):   0-13
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc cpuid pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm cpuid_fault invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid xsaveopt arat md_clear flush_l1d arch_capabilities
[crc@eaas-crc ~]$  free -g
              total        used        free      shared  buff/cache   available
Mem:             39          15           0           0          23          23
Swap:             0           0           0

Thanks

gbraad commented 3 years ago

Hypervisor vendor: VMware

This is an unsupported setup. Too many issues that we can not explain. If you can try this out on baremetal or KVM+Libvirt as the hypervisor? As that is a setup we know works and has been tested.

rupeshad81 commented 3 years ago

As per the release notes, it should be compatible.

https://access.redhat.com/documentation/en-us/red_hat_codeready_containers/1.30/html-single/getting_started_guide/index#minimum-system-requirements_gsg

2.1.2.3. Linux

cfergeau commented 3 years ago

This is something we forgot to mention in our documentation, but there have been known issues on various nested virtualization setups (see the list of issues I linked to previously for vmware), and we don't have the resources to test/debug on nested virtualization.

rupeshad81 commented 3 years ago

Hi, Please can you setup the call for discussion.

Thanks

gbraad commented 3 years ago

This is something we forgot to mention in our documentation,

We have refrained from mentioning the nested virtualization in our documentation. The mention that this supports native hypervisors should indicate this clearly enough, this is however not resolved yet. Any nested setup or other solution might or might not work.

Please can you setup the call for discussion.

As mentioned; nested virtualization with VMware is untested; in short YMMV. Please contact GSS if you need further assistance.

rupeshad81 commented 3 years ago

Hi, Actually we had RHEL 8.4 and RHEL 7.8 on the both , getting the issue. We do not have baremetal or KVM+Libvirt as the hypervisor , as you are suggesting. Till now ,crc was working fine 4.7 but for 4.8 it had started giving the issue. So we need the exact resolution for it.

Thanks

gbraad commented 3 years ago

So this is an OpenShift issue. For now, continue using CRC 1.29 with OCP 4.7

Note: understand GSS already reached out. Closing this issue to prevent conflicting information.