[BUG]: Can't get 'crc start' to work on Fedora 37 workstation. Just spins at -> oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig

General information

OS: Linux / macOS / Windows
Hypervisor: KVM / Hyper-V / hyperkit / vfkit (The one that comes by default when you install crc from scratch <== ???
Did you run crc setup before starting it (Yes/No)? Yes
Running CRC on: Laptop / Baremetal-Server / VM LAPTOP

CRC version

CRC version: 2.11.0+823e40d
OpenShift version: 4.11.13
Podman version: 4.2.0

CRC status

DEBU CRC version: 2.11.0+823e40d                  
DEBU OpenShift version: 4.11.13                   
DEBU Podman version: 4.2.0                        
DEBU Running 'crc status'                         
CRC VM:          Running
OpenShift:       Unreachable (v4.11.13)
RAM Usage:       3.14GB of 9.399GB
Disk Usage:      11.16GB of 32.74GB (Inside the CRC VM)
Cache Usage:     15.99GB
Cache Directory: /home/reselbob/.crc/cache

CRC config

- consent-telemetry                     : yes

Host Operating System

NAME="Fedora Linux"
VERSION="37 (Workstation Edition)"
ID=fedora
VERSION_ID=37
VERSION_CODENAME=""
PLATFORM_ID="platform:f37"
PRETTY_NAME="Fedora Linux 37 (Workstation Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:37"
DEFAULT_HOSTNAME="fedora"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f37/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=37
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=37
VARIANT="Workstation Edition"
VARIANT_ID=workstation

Steps to reproduce

I did a fresh install of Fedora 37 that wiped the disk clean. There is nothing on this machine out of the ordinary.

I installed NetworkManager as the docs requires.

I tried to get it all up and running. No dice. I did some research on the bug and then did the below about 5 times.

crc delete -f
crc cleanup
crc setup
crc start --log-level debug

Expected

A running instance of crc with Developer and Admin access

Actual

Just spins here:

Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124```

Logs

$ crc setup
INFO Using bundle path /home/reselbob/.crc/cache/crc_libvirt_4.11.13_amd64.crcbundle 
INFO Checking if running as non-root              
INFO Checking if running inside WSL2              
INFO Checking if crc-admin-helper executable is cached 
INFO Checking for obsolete admin-helper executable 
INFO Checking if running on a supported CPU architecture 
INFO Checking minimum RAM requirements            
INFO Checking if crc executable symlink exists    
INFO Creating symlink for crc executable          
INFO Checking if Virtualization is enabled        
INFO Checking if KVM is enabled                   
INFO Checking if libvirt is installed             
INFO Checking if user is part of libvirt group    
INFO Checking if active user/process is currently part of the libvirt group 
INFO Checking if libvirt daemon is running        
INFO Checking if a supported libvirt version is installed 
INFO Checking if crc-driver-libvirt is installed  
INFO Checking crc daemon systemd service          
INFO Setting up crc daemon systemd service        
INFO Checking crc daemon systemd socket units     
INFO Setting up crc daemon systemd socket units   
INFO Checking if systemd-networkd is running      
INFO Checking if NetworkManager is installed      
INFO Checking if NetworkManager service is running 
INFO Checking if dnsmasq configurations file exist for NetworkManager 
INFO Checking if the systemd-resolved service is running 
INFO Checking if /etc/NetworkManager/dispatcher.d/99-crc.sh exists 
INFO Writing NetworkManager dispatcher file for crc 
INFO Using root access: Writing NetworkManager configuration to /etc/NetworkManager/dispatcher.d/99-crc.sh 
INFO Using root access: Changing permissions for /etc/NetworkManager/dispatcher.d/99-crc.sh to 755  
INFO Using root access: Executing systemctl daemon-reload command 
INFO Using root access: Executing systemctl reload NetworkManager 
INFO Checking if libvirt 'crc' network is available 
INFO Setting up libvirt 'crc' network             
INFO Checking if libvirt 'crc' network is active  
INFO Starting libvirt 'crc' network               
INFO Checking if CRC bundle is extracted in '$HOME/.crc' 
INFO Checking if /home/reselbob/.crc/cache/crc_libvirt_4.11.13_amd64.crcbundle exists 
Your system is correctly setup for using CRC. Use 'crc start' to start the instance

$ crc start --log-level debug
DEBU CRC version: 2.11.0+823e40d                  
DEBU OpenShift version: 4.11.13                   
DEBU Podman version: 4.2.0                        
DEBU Running 'crc start'                          
DEBU Total memory of system is 16628711424 bytes  
DEBU No new version available. The latest version is 2.11.0 
DEBU Checking file: /home/reselbob/.crc/machines/crc/.crc-exist 
DEBU Checking if systemd-resolved.service is running 
DEBU Running 'systemctl status systemd-resolved.service' 
DEBU systemd-resolved.service is already running  
INFO Checking if running as non-root              
INFO Checking if running inside WSL2              
INFO Checking if crc-admin-helper executable is cached 
DEBU Running '/home/reselbob/.crc/bin/crc-admin-helper-linux --version' 
DEBU Found crc-admin-helper-linux version 0.0.11  
DEBU crc-admin-helper executable already cached   
INFO Checking for obsolete admin-helper executable 
DEBU Checking if an older admin-helper executable is installed 
DEBU No older admin-helper executable found       
INFO Checking if running on a supported CPU architecture 
DEBU GOARCH is amd64 GOOS is linux                
INFO Checking minimum RAM requirements            
DEBU Total memory of system is 16628711424 bytes  
INFO Checking if crc executable symlink exists    
INFO Checking if Virtualization is enabled        
DEBU Checking if the vmx/svm flags are present in /proc/cpuinfo 
DEBU CPU virtualization flags are good            
INFO Checking if KVM is enabled                   
DEBU Checking if /dev/kvm exists                  
DEBU /dev/kvm was found                           
INFO Checking if libvirt is installed             
DEBU Checking if 'virsh' is available             
DEBU 'virsh' was found in /usr/bin/virsh          
DEBU Checking 'virsh capabilities' for libvirtd/qemu availability 
DEBU Running 'virsh --readonly --connect qemu:///system capabilities' 
DEBU Found x86_64 hypervisor with 'hvm' capabilities 
INFO Checking if user is part of libvirt group    
DEBU Checking if current user is part of the libvirt group 
DEBU Current user is already in the libvirt group 
INFO Checking if active user/process is currently part of the libvirt group 
INFO Checking if libvirt daemon is running        
DEBU Checking if libvirtd service is running      
DEBU Running 'systemctl status virtqemud.socket'  
DEBU virtqemud.socket is running                  
INFO Checking if a supported libvirt version is installed 
DEBU Checking if libvirt version is >=3.4.0       
DEBU Running 'virsh -v'                           
INFO Checking if crc-driver-libvirt is installed  
DEBU Checking if crc-driver-libvirt is installed  
DEBU Running '/home/reselbob/.crc/bin/crc-driver-libvirt version' 
DEBU Found crc-driver-libvirt version 0.13.5      
DEBU crc-driver-libvirt is already installed      
INFO Checking crc daemon systemd socket units     
DEBU Checking crc daemon systemd socket units     
DEBU Checking if crc-http.socket is running       
DEBU Running 'systemctl --user status crc-http.socket' 
DEBU crc-http.socket is listening                 
DEBU Checking if crc-http.socket has the expected content 
DEBU Checking if crc-vsock.socket is running      
DEBU Running 'systemctl --user status crc-vsock.socket' 
DEBU crc-vsock.socket is listening                
DEBU Checking if crc-vsock.socket has the expected content 
INFO Checking if systemd-networkd is running      
DEBU Checking if systemd-networkd.service is running 
DEBU Running 'systemctl status systemd-networkd.service' 
DEBU Command failed: exit status 3                
DEBU stdout: * systemd-networkd.service - Network Configuration
     Loaded: loaded (/usr/lib/systemd/system/systemd-networkd.service; disabled; preset: disabled)
     Active: inactive (dead)
TriggeredBy: * systemd-networkd.socket
       Docs: man:systemd-networkd.service(8) 
DEBU stderr:                                      
DEBU systemd-networkd.service is not running      
INFO Checking if NetworkManager is installed      
DEBU Checking if 'nmcli' is available             
DEBU 'nmcli' was found in /usr/bin/nmcli          
INFO Checking if NetworkManager service is running 
DEBU Checking if NetworkManager.service is running 
DEBU Running 'systemctl status NetworkManager.service' 
DEBU NetworkManager.service is already running    
INFO Checking if dnsmasq configurations file exist for NetworkManager 
INFO Checking if the systemd-resolved service is running 
DEBU Checking if systemd-resolved.service is running 
DEBU Running 'systemctl status systemd-resolved.service' 
DEBU systemd-resolved.service is already running  
INFO Checking if /etc/NetworkManager/dispatcher.d/99-crc.sh exists 
DEBU Checking NetworkManager dispatcher file for crc network 
DEBU Dispatcher file has the expected content     
INFO Checking if libvirt 'crc' network is available 
DEBU Checking if libvirt 'crc' network exists     
DEBU Running 'virsh --connect qemu:///system net-info crc' 
DEBU Checking if libvirt 'crc' definition is up to date 
DEBU Running 'virsh --connect qemu:///system net-dumpxml --inactive crc' 
DEBU libvirt 'crc' network has the expected value 
INFO Checking if libvirt 'crc' network is active  
DEBU Checking if libvirt 'crc' network is active  
DEBU Running 'virsh --connect qemu:///system net-info crc' 
DEBU libvirt 'crc' network is already active      
DEBU Checking file: /home/reselbob/.crc/machines/crc/.crc-exist 
INFO Loading bundle: crc_libvirt_4.11.13_amd64... 
DEBU Cannot load secret from configuration: empty path 
DEBU Cannot load secret from keyring: secret not found in keyring 
CRC requires a pull secret to download content from Red Hat.
You can copy it from the Pull Secret section of https://console.redhat.com/openshift/create/local.
? Please enter the pull secret *********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************

INFO Creating CRC VM for openshift 4.11.13...     
DEBU Found binary path at /home/reselbob/.crc/bin/crc-driver-libvirt 
DEBU Launching plugin server for driver libvirt   
DEBU Plugin server listening at address 127.0.0.1:35421 
DEBU () Calling .GetVersion                       
DEBU Using API Version 1                          
DEBU () Calling .SetConfigRaw                     
DEBU () Calling .GetMachineName                   
DEBU (crc) Calling .GetMachineName                
DEBU (crc) Calling .DriverName                    
DEBU Running pre-create checks...                 
DEBU (crc) Calling .PreCreateCheck                
DEBU (crc) DBG | time="2022-12-02T18:33:09-05:00" level=debug msg="About to check libvirt version" 
DEBU (crc) DBG | time="2022-12-02T18:33:09-05:00" level=debug msg="Validating network" 
DEBU (crc) DBG | time="2022-12-02T18:33:09-05:00" level=debug msg="Validating storage pool" 
DEBU (crc) DBG | time="2022-12-02T18:33:09-05:00" level=debug msg="Could not find storage pool 'crc', trying to create it" 
DEBU (crc) DBG | time="2022-12-02T18:33:09-05:00" level=debug msg="Creating storage pool" 
DEBU (crc) DBG | time="2022-12-02T18:33:09-05:00" level=info msg="Creating storage pool with XML <pool type=\"dir\">\n  <name>crc</name>\n  <target>\n    <path>/home/reselbob/.crc/machines/crc</path>\n  </target>\n</pool>" 
DEBU (crc) DBG | time="2022-12-02T18:33:09-05:00" level=debug msg="Activating pool 'crc'" 
DEBU (crc) Calling .GetConfigRaw                  
DEBU Creating machine...                          
DEBU (crc) Calling .Create                        
DEBU (crc) DBG | time="2022-12-02T18:33:09-05:00" level=debug msg="Preparing /home/reselbob/.crc/machines/crc/crc.qcow2 for machine use" 
DEBU (crc) DBG | time="2022-12-02T18:33:10-05:00" level=debug msg="image creation took 322.050758ms" 
DEBU (crc) DBG | time="2022-12-02T18:33:10-05:00" level=debug msg="Verifying executable bit set on /home/reselbob/.crc/machines/crc" 
DEBU (crc) DBG | time="2022-12-02T18:33:10-05:00" level=debug msg="Verifying executable bit set on /home/reselbob/.crc/machines" 
DEBU (crc) DBG | time="2022-12-02T18:33:10-05:00" level=debug msg="Verifying executable bit set on /home/reselbob/.crc" 
DEBU (crc) DBG | time="2022-12-02T18:33:10-05:00" level=debug msg="Verifying executable bit set on /home/reselbob" 
DEBU (crc) DBG | time="2022-12-02T18:33:10-05:00" level=debug msg="Verifying executable bit set on /home" 
DEBU (crc) DBG | time="2022-12-02T18:33:10-05:00" level=debug msg="Defining VM..." 
DEBU (crc) DBG | time="2022-12-02T18:33:10-05:00" level=debug msg="Found x86_64 hypervisor with 'hvm' capabilities" 
DEBU (crc) DBG | time="2022-12-02T18:33:10-05:00" level=debug msg="Found q35 machine type" 
DEBU (crc) DBG | time="2022-12-02T18:33:10-05:00" level=debug msg="Found x86_64 hypervisor with 'hvm' capabilities" 
DEBU (crc) DBG | time="2022-12-02T18:33:10-05:00" level=debug msg="Found q35 machine type" 
DEBU (crc) DBG | time="2022-12-02T18:33:10-05:00" level=debug msg="disk image capacity is already 33285996544 bytes" 
INFO Generating new SSH key pair...               
INFO Generating new password for the kubeadmin user 
DEBU Created /home/reselbob/.crc/machines/crc/.crc-exist 
DEBU Machine successfully created                 
DEBU Making call to close driver server           
DEBU (crc) Calling .Close                         
DEBU (crc) DBG | time="2022-12-02T18:33:10-05:00" level=debug msg="Closing plugin on server side" 
DEBU Successfully made call to close driver server 
DEBU Making call to close connection to plugin binary 
DEBU Checking file: /home/reselbob/.crc/machines/crc/.crc-exist 
DEBU Found binary path at /home/reselbob/.crc/bin/crc-driver-libvirt 
DEBU Launching plugin server for driver libvirt   
DEBU Plugin server listening at address 127.0.0.1:44535 
DEBU () Calling .GetVersion                       
DEBU Using API Version 1                          
DEBU () Calling .SetConfigRaw                     
DEBU () Calling .GetMachineName                   
DEBU (crc) Calling .GetBundleName                 
DEBU (crc) Calling .GetState                      
DEBU (crc) DBG | time="2022-12-02T18:33:10-05:00" level=debug msg="Getting current state..." 
DEBU (crc) DBG | time="2022-12-02T18:33:10-05:00" level=debug msg="Fetching VM..." 
INFO Starting CRC VM for openshift 4.11.13...     
DEBU Updating CRC VM configuration                
DEBU (crc) Calling .GetConfigRaw                  
DEBU (crc) Calling .Start                         
DEBU (crc) DBG | time="2022-12-02T18:33:10-05:00" level=debug msg="Starting VM crc" 
DEBU (crc) DBG | time="2022-12-02T18:33:10-05:00" level=debug msg="Validating network" 
DEBU (crc) DBG | time="2022-12-02T18:33:10-05:00" level=debug msg="Validating storage pool" 
DEBU (crc) DBG | time="2022-12-02T18:33:16-05:00" level=debug msg="GetIP called for crc" 
DEBU (crc) DBG | time="2022-12-02T18:33:16-05:00" level=debug msg="Getting current state..." 
DEBU (crc) DBG | time="2022-12-02T18:33:16-05:00" level=debug msg="Waiting for machine to come up 0/60" 
DEBU (crc) DBG | time="2022-12-02T18:33:19-05:00" level=debug msg="GetIP called for crc" 
DEBU (crc) DBG | time="2022-12-02T18:33:19-05:00" level=debug msg="Getting current state..." 
DEBU (crc) DBG | time="2022-12-02T18:33:19-05:00" level=debug msg="Waiting for machine to come up 1/60" 
DEBU (crc) DBG | time="2022-12-02T18:33:22-05:00" level=debug msg="GetIP called for crc" 
DEBU (crc) DBG | time="2022-12-02T18:33:22-05:00" level=debug msg="Getting current state..." 
DEBU (crc) DBG | time="2022-12-02T18:33:22-05:00" level=debug msg="Waiting for machine to come up 2/60" 
DEBU (crc) DBG | time="2022-12-02T18:33:25-05:00" level=debug msg="GetIP called for crc" 
DEBU (crc) DBG | time="2022-12-02T18:33:25-05:00" level=debug msg="Getting current state..." 
DEBU (crc) DBG | time="2022-12-02T18:33:25-05:00" level=debug msg="Waiting for machine to come up 3/60" 
DEBU (crc) DBG | time="2022-12-02T18:33:28-05:00" level=debug msg="GetIP called for crc" 
DEBU (crc) DBG | time="2022-12-02T18:33:28-05:00" level=debug msg="Getting current state..." 
DEBU (crc) DBG | time="2022-12-02T18:33:28-05:00" level=debug msg="Waiting for machine to come up 4/60" 
DEBU (crc) DBG | time="2022-12-02T18:33:31-05:00" level=debug msg="GetIP called for crc" 
DEBU (crc) DBG | time="2022-12-02T18:33:31-05:00" level=debug msg="Getting current state..." 
DEBU (crc) DBG | time="2022-12-02T18:33:31-05:00" level=debug msg="Waiting for machine to come up 5/60" 
DEBU (crc) DBG | time="2022-12-02T18:33:34-05:00" level=debug msg="GetIP called for crc" 
DEBU (crc) DBG | time="2022-12-02T18:33:34-05:00" level=debug msg="Getting current state..." 
DEBU (crc) DBG | time="2022-12-02T18:33:34-05:00" level=debug msg="Waiting for machine to come up 6/60" 
DEBU (crc) DBG | time="2022-12-02T18:33:37-05:00" level=debug msg="GetIP called for crc" 
DEBU (crc) DBG | time="2022-12-02T18:33:37-05:00" level=debug msg="Getting current state..." 
DEBU (crc) DBG | time="2022-12-02T18:33:37-05:00" level=debug msg="Waiting for machine to come up 7/60" 
DEBU (crc) DBG | time="2022-12-02T18:33:40-05:00" level=debug msg="GetIP called for crc" 
DEBU (crc) DBG | time="2022-12-02T18:33:40-05:00" level=debug msg="Getting current state..." 
DEBU (crc) DBG | time="2022-12-02T18:33:40-05:00" level=debug msg="Waiting for machine to come up 8/60" 
DEBU (crc) DBG | time="2022-12-02T18:33:43-05:00" level=debug msg="GetIP called for crc" 
DEBU (crc) DBG | time="2022-12-02T18:33:43-05:00" level=debug msg="Getting current state..." 
DEBU (crc) DBG | time="2022-12-02T18:33:43-05:00" level=debug msg="Waiting for machine to come up 9/60" 
DEBU (crc) DBG | time="2022-12-02T18:33:46-05:00" level=debug msg="GetIP called for crc" 
DEBU (crc) DBG | time="2022-12-02T18:33:46-05:00" level=debug msg="Getting current state..." 
DEBU (crc) DBG | time="2022-12-02T18:33:46-05:00" level=debug msg="Waiting for machine to come up 10/60" 
DEBU (crc) DBG | time="2022-12-02T18:33:49-05:00" level=debug msg="GetIP called for crc" 
DEBU (crc) DBG | time="2022-12-02T18:33:49-05:00" level=debug msg="Getting current state..." 
DEBU (crc) DBG | time="2022-12-02T18:33:49-05:00" level=debug msg="Waiting for machine to come up 11/60" 
DEBU (crc) DBG | time="2022-12-02T18:33:52-05:00" level=debug msg="GetIP called for crc" 
DEBU (crc) DBG | time="2022-12-02T18:33:52-05:00" level=debug msg="Getting current state..." 
DEBU (crc) DBG | time="2022-12-02T18:33:52-05:00" level=debug msg="Waiting for machine to come up 12/60" 
DEBU (crc) DBG | time="2022-12-02T18:33:55-05:00" level=debug msg="GetIP called for crc" 
DEBU (crc) DBG | time="2022-12-02T18:33:55-05:00" level=debug msg="Getting current state..." 
DEBU (crc) DBG | time="2022-12-02T18:33:55-05:00" level=debug msg="Waiting for machine to come up 13/60" 
DEBU (crc) DBG | time="2022-12-02T18:33:58-05:00" level=debug msg="GetIP called for crc" 
DEBU (crc) DBG | time="2022-12-02T18:33:58-05:00" level=debug msg="Getting current state..." 
DEBU (crc) DBG | time="2022-12-02T18:33:58-05:00" level=debug msg="Waiting for machine to come up 14/60" 
DEBU (crc) DBG | time="2022-12-02T18:34:01-05:00" level=debug msg="GetIP called for crc" 
DEBU (crc) DBG | time="2022-12-02T18:34:01-05:00" level=debug msg="Getting current state..." 
DEBU (crc) DBG | time="2022-12-02T18:34:01-05:00" level=debug msg="Waiting for machine to come up 15/60" 
DEBU (crc) DBG | time="2022-12-02T18:34:04-05:00" level=debug msg="GetIP called for crc" 
DEBU (crc) DBG | time="2022-12-02T18:34:04-05:00" level=debug msg="Getting current state..." 
DEBU (crc) DBG | time="2022-12-02T18:34:04-05:00" level=debug msg="Waiting for machine to come up 16/60" 
DEBU (crc) DBG | time="2022-12-02T18:34:07-05:00" level=debug msg="GetIP called for crc" 
DEBU (crc) DBG | time="2022-12-02T18:34:07-05:00" level=debug msg="Getting current state..." 
DEBU (crc) DBG | time="2022-12-02T18:34:07-05:00" level=debug msg="Waiting for machine to come up 17/60" 
DEBU (crc) DBG | time="2022-12-02T18:34:10-05:00" level=debug msg="GetIP called for crc" 
DEBU (crc) DBG | time="2022-12-02T18:34:10-05:00" level=debug msg="Getting current state..." 
DEBU (crc) DBG | time="2022-12-02T18:34:10-05:00" level=debug msg="Waiting for machine to come up 18/60" 
DEBU (crc) DBG | time="2022-12-02T18:34:13-05:00" level=debug msg="GetIP called for crc" 
DEBU (crc) DBG | time="2022-12-02T18:34:13-05:00" level=debug msg="Getting current state..." 
DEBU (crc) DBG | time="2022-12-02T18:34:13-05:00" level=debug msg="Waiting for machine to come up 19/60" 
DEBU (crc) DBG | time="2022-12-02T18:34:16-05:00" level=debug msg="GetIP called for crc" 
DEBU (crc) DBG | time="2022-12-02T18:34:16-05:00" level=debug msg="Getting current state..." 
DEBU (crc) DBG | time="2022-12-02T18:34:16-05:00" level=debug msg="Waiting for machine to come up 20/60" 
DEBU (crc) DBG | time="2022-12-02T18:34:19-05:00" level=debug msg="GetIP called for crc" 
DEBU (crc) DBG | time="2022-12-02T18:34:19-05:00" level=debug msg="Getting current state..." 
DEBU (crc) DBG | time="2022-12-02T18:34:19-05:00" level=debug msg="IP address: 192.168.130.11" 
DEBU (crc) DBG | time="2022-12-02T18:34:19-05:00" level=info msg="Found IP for machine: 192.168.130.11" 
DEBU (crc) Calling .GetConfigRaw                  
DEBU Waiting for machine to be running, this may take a few minutes... 
DEBU retry loop: attempt 0                        
DEBU (crc) Calling .GetState                      
DEBU (crc) DBG | time="2022-12-02T18:34:19-05:00" level=debug msg="Getting current state..." 
DEBU Machine is up and running!                   
DEBU (crc) Calling .GetState                      
DEBU (crc) DBG | time="2022-12-02T18:34:19-05:00" level=debug msg="Getting current state..." 
DEBU (crc) Calling .GetIP                         
DEBU (crc) DBG | time="2022-12-02T18:34:19-05:00" level=debug msg="GetIP called for crc" 
DEBU (crc) DBG | time="2022-12-02T18:34:19-05:00" level=debug msg="Getting current state..." 
DEBU (crc) DBG | time="2022-12-02T18:34:19-05:00" level=debug msg="IP address: 192.168.130.11" 
INFO CRC instance is running with IP 192.168.130.11 
DEBU (crc) Calling .GetIP                         
DEBU (crc) DBG | time="2022-12-02T18:34:19-05:00" level=debug msg="GetIP called for crc" 
DEBU (crc) DBG | time="2022-12-02T18:34:19-05:00" level=debug msg="Getting current state..." 
DEBU (crc) DBG | time="2022-12-02T18:34:19-05:00" level=debug msg="IP address: 192.168.130.11" 
DEBU Waiting until ssh is available               
DEBU retry loop: attempt 0                        
DEBU Running SSH command: exit 0                  
DEBU Using ssh private keys: [/home/reselbob/.crc/machines/crc/id_ecdsa /home/reselbob/.crc/cache/crc_libvirt_4.11.13_amd64/id_ecdsa_crc] 
DEBU SSH command results: err: <nil>, output:     
INFO CRC VM is running                            
DEBU Running SSH command: cat /home/core/.ssh/authorized_keys 
DEBU SSH command results: err: <nil>, output: ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHJGDAsMl6wzqmyBzeQUMIIkWaMmoEKh5nYb5BpBc8cT6EODXQpALSAin0BaHYGb83cjHwKKZ6X1d+SWTRW2n7IBQA3W3ph1yNaFUX+bBKmO0zi8mAk1O6Y3kyI8G5lFQ+1m+3nE2xBqiK/FJJuCHTou94Wb7bTfZ/GIJjTEXxmzq9GJw== core 
INFO Updating authorized keys...                  
DEBU Creating /home/core/.ssh/authorized_keys with permissions 0644 in the CRC VM 
DEBU Running SSH command: <hidden>                
DEBU SSH command succeeded                        
DEBU Running SSH command: rm /home/core/.ssh/authorized_keys.d/ignition 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU Running SSH command: realpath /dev/disk/by-label/root 
DEBU SSH command results: err: <nil>, output: /dev/vda4 
DEBU Using root access: Growing /dev/vda4 partition 
DEBU Running SSH command: sudo /usr/bin/growpart /dev/vda 4 
DEBU SSH command results: err: Process exited with status 1, output: NOCHANGE: partition 4 is size 63961055. it cannot be grown 
DEBU No free space after /dev/vda4, nothing to do 
DEBU Configuring shared directories               
DEBU (crc) Calling .GetSharedDirs                 
DEBU (crc) DBG | time="2022-12-02T18:34:22-05:00" level=debug msg="Found x86_64 hypervisor with 'hvm' capabilities" 
DEBU (crc) DBG | time="2022-12-02T18:34:22-05:00" level=debug msg="Found q35 machine type" 
INFO Configuring shared directories               
DEBU Using root access: Creating /home/reselbob   
DEBU Running SSH command: sudo mkdir -p /home/reselbob 
DEBU SSH command results: err: <nil>, output:     
DEBU Mounting tag dir0 at /home/reselbob          
DEBU Using root access: Mounting /home/reselbob   
DEBU Running SSH command: sudo mount -o context="system_u:object_r:container_file_t:s0" -t virtiofs dir0 /home/reselbob 
DEBU SSH command results: err: <nil>, output:     
DEBU Using root access: make root Podman socket accessible 
DEBU Running SSH command: sudo chmod 777 /run/podman/ /run/podman/podman.sock 
DEBU SSH command results: err: <nil>, output:     
DEBU Creating /var/srv/dnsmasq.conf with permissions 0644 in the CRC VM 
DEBU Running SSH command: <hidden>                
DEBU SSH command succeeded                        
DEBU Running SSH command: systemctl status crc-dnsmasq.service 
DEBU SSH command results: err: Process exited with status 3, output: ● crc-dnsmasq.service - Podman container-3fe9951884b44ede6c74587cce731028d884ebd0567d8b3e8b2f6a6249e92352.service
   Loaded: loaded (/etc/systemd/system/crc-dnsmasq.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
     Docs: man:podman-generate-systemd(1) 
DEBU Using root access: Executing systemctl enable crc-dnsmasq.service 
DEBU Running SSH command: sudo systemctl enable crc-dnsmasq.service 
DEBU SSH command results: err: <nil>, output:     
DEBU Using root access: Executing systemctl daemon-reload command 
DEBU Running SSH command: sudo systemctl daemon-reload 
DEBU SSH command results: err: <nil>, output:     
DEBU Using root access: Executing systemctl start crc-dnsmasq.service 
DEBU Running SSH command: sudo systemctl start crc-dnsmasq.service 
DEBU SSH command results: err: <nil>, output:     
DEBU Running '/home/reselbob/.crc/bin/crc-admin-helper-linux rm api.crc.testing oauth-openshift.apps-crc.testing console-openshift-console.apps-crc.testing downloads-openshift-console.apps-crc.testing canary-openshift-ingress-canary.apps-crc.testing default-route-openshift-image-registry.apps-crc.testing' 
DEBU Running '/home/reselbob/.crc/bin/crc-admin-helper-linux add 192.168.130.11 api.crc.testing oauth-openshift.apps-crc.testing console-openshift-console.apps-crc.testing downloads-openshift-console.apps-crc.testing canary-openshift-ingress-canary.apps-crc.testing default-route-openshift-image-registry.apps-crc.testing' 
DEBU Running SSH command: cat /etc/resolv.conf    
DEBU SSH command results: err: <nil>, output: # Generated by NetworkManager
nameserver 192.168.130.1 
DEBU Creating /etc/resolv.conf with permissions 0644 in the CRC VM 
DEBU Running SSH command: <hidden>                
DEBU SSH command succeeded                        
DEBU retry loop: attempt 0                        
DEBU Running SSH command: host -R 3 foo.apps-crc.testing 
DEBU SSH command results: err: <nil>, output: foo.apps-crc.testing has address 192.168.130.11 
INFO Check internal and public DNS query...       
DEBU Running SSH command: curl --head quay.io     
DEBU SSH command results: err: <nil>, output: HTTP/1.1 301 Moved Permanently
Server: awselb/2.0
Date: Fri, 02 Dec 2022 23:34:31 GMT
Content-Type: text/html
Content-Length: 134
Connection: keep-alive
Location: https://quay.io:443/

INFO Check DNS query from host...                 
DEBU api.crc.testing resolved to [192.168.130.11] 
DEBU foo.apps-crc.testing resolved to [192.168.130.11] 
INFO Verifying validity of the kubelet certificates... 
DEBU Running SSH command: date --date="$(sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -enddate | cut -d= -f 2)" --iso-8601=seconds 
DEBU SSH command results: err: <nil>, output: 2023-11-21T08:28:47+00:00 
DEBU Running SSH command: date --date="$(sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-server-current.pem -noout -enddate | cut -d= -f 2)" --iso-8601=seconds 
DEBU SSH command results: err: <nil>, output: 2023-11-21T08:32:30+00:00 
DEBU Running SSH command: date --date="$(sudo openssl x509 -in /etc/kubernetes/static-pod-resources/kube-apiserver-certs/configmaps/aggregator-client-ca/ca-bundle.crt -noout -enddate | cut -d= -f 2)" --iso-8601=seconds 
DEBU SSH command results: err: <nil>, output: 2023-11-21T08:32:55+00:00 
INFO Starting kubelet service                     
DEBU Using root access: Executing systemctl daemon-reload command 
DEBU Running SSH command: sudo systemctl daemon-reload 
DEBU SSH command results: err: <nil>, output:     
DEBU Using root access: Executing systemctl start kubelet 
DEBU Running SSH command: sudo systemctl start kubelet 
DEBU SSH command results: err: <nil>, output:     
INFO Waiting for kube-apiserver availability... [takes around 2min] 
DEBU retry loop: attempt 0                        
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 1                        
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 2                        
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 3                        
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 4                        
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 5                        
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 6                        
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 7                        
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 8                        
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 9                        
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 10                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 11                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 12                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 13                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 14                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 15                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 16                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 17                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 18                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 19                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 20                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 21                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 22                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 23                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 24                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 25                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 26                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 27                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 28                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 29                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 30                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 31                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 32                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 33                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 34                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 35                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 36                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 37                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 38                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 39                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 40                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 41                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 42                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 43                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 44                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 45                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 46                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 47                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 48                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 49                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 50                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 51                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 52                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 53                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 54                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 55                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 56                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 57                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 58                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU RetryAfter timeout after 59 tries            
DEBU Making call to close driver server           
DEBU (crc) Calling .Close                         
DEBU Successfully made call to close driver server 
DEBU Making call to close connection to plugin binary 
Error waiting for apiserver: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124

Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 (x26)
Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 (x32)

Then , I did the below a few times

Before gather the logs try following if that fix your issue--

$ crc delete -f
$ crc cleanup
$ crc setup
$ crc start --log-level debug

I hope you can help me. It's taken a day and a half of my life trying to get this to work. Help.... please....

@reselbob Process exited with status 124 is mostly because of timeout, what kind of hardware is this? Is it NUC because we got similar issue on NUC side https://github.com/crc-org/crc/issues/3294

Can you try ssh'ing into the VM as explained on https://github.com/crc-org/crc/wiki/Debugging-guide#entering-the-vm ? I want to make sure it's up and running. Is the system badly loaded (IO, CPU, ...) when the start fails?

Step 1: As recommended in the post above @cfergeau, I am doing the following:

$ crc ip
192.168.130.11

I can access the VM using the ssh config file.

Step 2: I run this command within the crc VM to demonstrate that I can reach the Internet, which I can do successfully:

curl https://quay.io

Step 3: Then I execute this command...

[core@crc-nv4w7-master-0 ~]$ KUBECONFIG=/opt/kubeconfig kubectl get co

... and get this result:

NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.11.13   True        False         False      3m48s   
config-operator                            4.11.13   True        False         False      15d     
console                                    4.11.13   True        False         False      13h     
dns                                        4.11.13   True        False         False      13h     
etcd                                       4.11.13   True        False         False      15d     
image-registry                             4.11.13   True        False         False      14h     
ingress                                    4.11.13   True        False         False      15d     
kube-apiserver                             4.11.13   True        False         False      15d     
kube-controller-manager                    4.11.13   True        False         False      15d     
kube-scheduler                             4.11.13   True        False         False      15d     
machine-api                                4.11.13   True        False         False      15d     
machine-approver                           4.11.13   True        False         False      15d     
machine-config                             4.11.13   True        False         False      15d     
marketplace                                4.11.13   True        False         False      15d     
network                                    4.11.13   True        False         False      15d     
node-tuning                                4.11.13   True        False         False      14h     
openshift-apiserver                        4.11.13   True        False         False      14h     
openshift-controller-manager               4.11.13   True        False         False      14h     
openshift-samples                          4.11.13   True        False         False      15d     
operator-lifecycle-manager                 4.11.13   True        False         False      15d     
operator-lifecycle-manager-catalog         4.11.13   True        False         False      15d     
operator-lifecycle-manager-packageserver   4.11.13   True        False         False      13h     
service-ca                                 4.11.13   True        False         False      15d

Step 4:Then I run this command ...

KUBECONFIG=/opt/kubeconfig kubectl get nodes

... and get this result

crc-nv4w7-master-0   Ready    master,worker   15d   v1.24.6+5157800

Step 5: Next I run this command ...

KUBECONFIG=/opt/kubeconfig kubectl describe nodes

... and get this very long output

Name:               crc-nv4w7-master-0
Roles:              master,worker
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=crc-nv4w7-master-0
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/master=
                    node-role.kubernetes.io/worker=
                    node.openshift.io/os_id=rhcos
                    topology.hostpath.csi/node=crc-nv4w7-master-0
Annotations:        csi.volume.kubernetes.io/nodeid: {"kubevirt.io.hostpath-provisioner":"crc-nv4w7-master-0"}
                    machine.openshift.io/machine: openshift-machine-api/crc-nv4w7-master-0
                    machineconfiguration.openshift.io/controlPlaneTopology: SingleReplica
                    machineconfiguration.openshift.io/currentConfig: rendered-master-7209aeeac8163ce4759eba596d501fa8
                    machineconfiguration.openshift.io/desiredConfig: rendered-master-7209aeeac8163ce4759eba596d501fa8
                    machineconfiguration.openshift.io/desiredDrain: uncordon-rendered-master-7209aeeac8163ce4759eba596d501fa8
                    machineconfiguration.openshift.io/lastAppliedDrain: uncordon-rendered-master-7209aeeac8163ce4759eba596d501fa8
                    machineconfiguration.openshift.io/reason: 
                    machineconfiguration.openshift.io/ssh: accessed
                    machineconfiguration.openshift.io/state: Done
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Sun, 20 Nov 2022 08:04:14 +0000
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  crc-nv4w7-master-0
  AcquireTime:     <unset>
  RenewTime:       Mon, 05 Dec 2022 16:31:38 +0000
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Mon, 05 Dec 2022 16:30:21 +0000   Mon, 05 Dec 2022 01:57:34 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Mon, 05 Dec 2022 16:30:21 +0000   Mon, 05 Dec 2022 01:57:34 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Mon, 05 Dec 2022 16:30:21 +0000   Mon, 05 Dec 2022 01:57:34 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Mon, 05 Dec 2022 16:30:21 +0000   Mon, 05 Dec 2022 01:57:34 +0000   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  192.168.126.11
  Hostname:    crc-nv4w7-master-0
Capacity:
  cpu:                4
  ephemeral-storage:  31970284Ki
  hugepages-2Mi:      0
  memory:             13999404Ki
  pods:               250
Allocatable:
  cpu:                3800m
  ephemeral-storage:  29463813686
  hugepages-2Mi:      0
  memory:             13538604Ki
  pods:               250
System Info:
  Machine ID:                                 32aa4b9405a7402dbd4edcfe82cc635a
  System UUID:                                c6a23913-a58e-4416-8e49-bd1db405ef21
  Boot ID:                                    64df86c9-6bb0-47ce-a769-acfd6f5a7156
  Kernel Version:                             4.18.0-372.32.1.el8_6.x86_64
  OS Image:                                   Red Hat Enterprise Linux CoreOS 411.86.202210311708-0 (Ootpa)
  Operating System:                           linux
  Architecture:                               amd64
  Container Runtime Version:                  cri-o://1.24.3-5.rhaos4.11.gitc4567c0.el8
  Kubelet Version:                            v1.24.6+5157800
  Kube-Proxy Version:                         v1.24.6+5157800
Non-terminated Pods:                          (64 in total)
  Namespace                                   Name                                                      CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                                   ----                                                      ------------  ----------  ---------------  -------------  ---
  hostpath-provisioner                        csi-hostpathplugin-l6f4f                                  0 (0%)        0 (0%)      0 (0%)           0 (0%)         14d
  openshift-apiserver-operator                openshift-apiserver-operator-6c9b9cb4bd-cd9g2             10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         15d
  openshift-apiserver                         apiserver-78f94fdb47-265dd                                110m (2%)     0 (0%)      250Mi (1%)       0 (0%)         14d
  openshift-authentication-operator           authentication-operator-dbd5bd768-hsrbv                   20m (0%)      0 (0%)      200Mi (1%)       0 (0%)         15d
  openshift-authentication                    oauth-openshift-76f87b4579-t54rz                          10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         14d
  openshift-cluster-machine-approver          machine-approver-7b5bd86fbb-zqtlm                         20m (0%)      0 (0%)      70Mi (0%)        0 (0%)         15d
  openshift-cluster-node-tuning-operator      cluster-node-tuning-operator-659594b5cf-b24mh             10m (0%)      0 (0%)      20Mi (0%)        0 (0%)         15d
  openshift-cluster-node-tuning-operator      tuned-fq4dk                                               10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         15d
  openshift-cluster-samples-operator          cluster-samples-operator-b588d95bd-xg8w5                  20m (0%)      0 (0%)      100Mi (0%)       0 (0%)         15d
  openshift-cluster-version                   cluster-version-operator-8475fc4c6-sfkj6                  20m (0%)      0 (0%)      50Mi (0%)        0 (0%)         15d
  openshift-config-operator                   openshift-config-operator-674f98f947-9c2r8                10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         15d
  openshift-console-operator                  console-operator-7b68655f7c-vjjj5                         10m (0%)      0 (0%)      100Mi (0%)       0 (0%)         15d
  openshift-console                           console-64bf795968-42dxz                                  10m (0%)      0 (0%)      100Mi (0%)       0 (0%)         15d
  openshift-console                           downloads-5587558f5f-kg5bs                                10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         15d
  openshift-controller-manager-operator       openshift-controller-manager-operator-7c4dc65c97-qljfm    10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         15d
  openshift-controller-manager                controller-manager-wbxpj                                  100m (2%)     0 (0%)      100Mi (0%)       0 (0%)         14d
  openshift-dns-operator                      dns-operator-6998f45999-lrqsb                             20m (0%)      0 (0%)      69Mi (0%)        0 (0%)         15d
  openshift-dns                               dns-default-dzrll                                         60m (1%)      0 (0%)      110Mi (0%)       0 (0%)         15d
  openshift-dns                               node-resolver-kvr46                                       5m (0%)       0 (0%)      21Mi (0%)        0 (0%)         15d
  openshift-etcd-operator                     etcd-operator-78974f57b9-znq2m                            10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         15d
  openshift-etcd                              etcd-crc-nv4w7-master-0                                   410m (10%)    0 (0%)      980Mi (7%)       0 (0%)         15d
  openshift-image-registry                    cluster-image-registry-operator-759dbc577c-28qjt          10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         15d
  openshift-image-registry                    image-registry-5d4bf896f4-459rx                           100m (2%)     0 (0%)      256Mi (1%)       0 (0%)         14d
  openshift-image-registry                    node-ca-q4f98                                             10m (0%)      0 (0%)      10Mi (0%)        0 (0%)         15d
  openshift-ingress-canary                    ingress-canary-dtwfl                                      10m (0%)      0 (0%)      20Mi (0%)        0 (0%)         15d
  openshift-ingress-operator                  ingress-operator-7f998586cd-6wxwb                         20m (0%)      0 (0%)      96Mi (0%)        0 (0%)         15d
  openshift-ingress                           router-default-68fdfcd4bd-w876v                           100m (2%)     0 (0%)      256Mi (1%)       0 (0%)         15d
  openshift-kube-apiserver-operator           kube-apiserver-operator-7cd64755f8-wdqdb                  10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         15d
  openshift-kube-apiserver                    kube-apiserver-crc-nv4w7-master-0                         290m (7%)     0 (0%)      1224Mi (9%)      0 (0%)         14d
  openshift-kube-controller-manager-operator  kube-controller-manager-operator-5fc769ddf6-kfdts         10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         15d
  openshift-kube-controller-manager           kube-controller-manager-crc-nv4w7-master-0                80m (2%)      0 (0%)      500Mi (3%)       0 (0%)         15d
  openshift-kube-scheduler-operator           openshift-kube-scheduler-operator-7c65bc5d9b-cxgvf        10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         15d
  openshift-kube-scheduler                    openshift-kube-scheduler-crc-nv4w7-master-0               25m (0%)      0 (0%)      150Mi (1%)       0 (0%)         15d
  openshift-machine-api                       machine-api-controllers-f4655d89f-xddjn                   70m (1%)      0 (0%)      140Mi (1%)       0 (0%)         15d
  openshift-machine-api                       machine-api-operator-5df8cd445c-vd65g                     20m (0%)      0 (0%)      70Mi (0%)        0 (0%)         15d
  openshift-machine-config-operator           machine-config-controller-57887dcc89-dldj2                40m (1%)      0 (0%)      100Mi (0%)       0 (0%)         15d
  openshift-machine-config-operator           machine-config-daemon-nr56c                               40m (1%)      0 (0%)      100Mi (0%)       0 (0%)         15d
  openshift-machine-config-operator           machine-config-operator-54ccccf58d-kjq6c                  20m (0%)      0 (0%)      50Mi (0%)        0 (0%)         15d
  openshift-machine-config-operator           machine-config-server-6kwq4                               20m (0%)      0 (0%)      50Mi (0%)        0 (0%)         15d
  openshift-marketplace                       certified-operators-p5gqr                                 10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         14h
  openshift-marketplace                       certified-operators-qd5c7                                 10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         15d
  openshift-marketplace                       community-operators-s6qx4                                 10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         14h
  openshift-marketplace                       community-operators-sbz8c                                 10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         15d
  openshift-marketplace                       marketplace-operator-857855d858-7dgs7                     10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         15d
  openshift-marketplace                       redhat-marketplace-6ngq5                                  10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         14h
  openshift-marketplace                       redhat-marketplace-mfs6j                                  10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         15d
  openshift-marketplace                       redhat-operators-b68f6                                    10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         14h
  openshift-marketplace                       redhat-operators-pqgd7                                    10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         15d
  openshift-multus                            multus-additional-cni-plugins-tzbx8                       10m (0%)      0 (0%)      10Mi (0%)        0 (0%)         15d
  openshift-multus                            multus-admission-controller-g8h6h                         20m (0%)      0 (0%)      70Mi (0%)        0 (0%)         15d
  openshift-multus                            multus-z76dw                                              10m (0%)      0 (0%)      65Mi (0%)        0 (0%)         15d
  openshift-multus                            network-metrics-daemon-lz5tb                              20m (0%)      0 (0%)      120Mi (0%)       0 (0%)         15d
  openshift-network-diagnostics               network-check-source-b5988947-sstsh                       10m (0%)      0 (0%)      40Mi (0%)        0 (0%)         15d
  openshift-network-diagnostics               network-check-target-swnrs                                10m (0%)      0 (0%)      15Mi (0%)        0 (0%)         15d
  openshift-network-operator                  network-operator-5499bf8b85-85t62                         10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         15d
  openshift-oauth-apiserver                   apiserver-5cb6ddc4c5-znqh2                                150m (3%)     0 (0%)      200Mi (1%)       0 (0%)         15d
  openshift-operator-lifecycle-manager        catalog-operator-5655866444-r44kz                         10m (0%)      0 (0%)      80Mi (0%)        0 (0%)         15d
  openshift-operator-lifecycle-manager        olm-operator-86cd44d7b4-s2vhx                             10m (0%)      0 (0%)      160Mi (1%)       0 (0%)         15d
  openshift-operator-lifecycle-manager        package-server-manager-644454f859-zr8c8                   10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         15d
  openshift-operator-lifecycle-manager        packageserver-77b84f7f67-bgqnr                            10m (0%)      0 (0%)      50Mi (0%)        0 (0%)         15d
  openshift-sdn                               sdn-bbzns                                                 110m (2%)     0 (0%)      220Mi (1%)       0 (0%)         15d
  openshift-sdn                               sdn-controller-fhfck                                      20m (0%)      0 (0%)      70Mi (0%)        0 (0%)         15d
  openshift-service-ca-operator               service-ca-operator-6bf9d9448c-hvntv                      10m (0%)      0 (0%)      80Mi (0%)        0 (0%)         15d
  openshift-service-ca                        service-ca-66cfb699d9-nrzwg                               10m (0%)      0 (0%)      120Mi (0%)       0 (0%)         15d
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests      Limits
  --------           --------      ------
  cpu                2290m (60%)   0 (0%)
  memory             7592Mi (57%)  0 (0%)
  ephemeral-storage  0 (0%)        0 (0%)
  hugepages-2Mi      0 (0%)        0 (0%)
Events:              <none>

Step 6: I run this command ...

KUBECONFIG=/opt/kubeconfig kubectl get pods -A

... and get this output:

NAMESPACE                                    NAME                                                     READY   STATUS             RESTARTS       AGE
hostpath-provisioner                         csi-hostpathplugin-l6f4f                                 4/4     Running            8              14d
openshift-apiserver-operator                 openshift-apiserver-operator-6c9b9cb4bd-cd9g2            1/1     Running            1              15d
openshift-apiserver                          apiserver-78f94fdb47-265dd                               2/2     Running            8              14d
openshift-authentication-operator            authentication-operator-dbd5bd768-hsrbv                  1/1     Running            1              15d
openshift-authentication                     oauth-openshift-76f87b4579-t54rz                         1/1     Running            1              14d
openshift-cluster-machine-approver           machine-approver-7b5bd86fbb-zqtlm                        2/2     Running            6              15d
openshift-cluster-node-tuning-operator       cluster-node-tuning-operator-659594b5cf-b24mh            1/1     Running            4              15d
openshift-cluster-node-tuning-operator       tuned-fq4dk                                              1/1     Running            1              15d
openshift-cluster-samples-operator           cluster-samples-operator-b588d95bd-xg8w5                 2/2     Running            2              15d
openshift-cluster-version                    cluster-version-operator-8475fc4c6-sfkj6                 1/1     Running            1              15d
openshift-config-operator                    openshift-config-operator-674f98f947-9c2r8               1/1     Running            1              15d
openshift-console-operator                   console-operator-7b68655f7c-vjjj5                        1/1     Running            1              15d
openshift-console                            console-64bf795968-42dxz                                 1/1     Running            3              15d
openshift-console                            downloads-5587558f5f-kg5bs                               1/1     Running            4 (13h ago)    15d
openshift-controller-manager-operator        openshift-controller-manager-operator-7c4dc65c97-qljfm   1/1     Running            1              15d
openshift-controller-manager                 controller-manager-wbxpj                                 1/1     Running            1              14d
openshift-dns-operator                       dns-operator-6998f45999-lrqsb                            2/2     Running            2              15d
openshift-dns                                dns-default-dzrll                                        2/2     Running            2              15d
openshift-dns                                node-resolver-kvr46                                      1/1     Running            1              15d
openshift-etcd-operator                      etcd-operator-78974f57b9-znq2m                           1/1     Running            1              15d
openshift-etcd                               etcd-crc-nv4w7-master-0                                  5/5     Running            7              15d
openshift-image-registry                     cluster-image-registry-operator-759dbc577c-28qjt         1/1     Running            1              15d
openshift-image-registry                     image-registry-5d4bf896f4-459rx                          1/1     Running            1              14d
openshift-image-registry                     node-ca-q4f98                                            1/1     Running            1              15d
openshift-ingress-canary                     ingress-canary-dtwfl                                     1/1     Running            1              15d
openshift-ingress-operator                   ingress-operator-7f998586cd-6wxwb                        2/2     Running            6 (13h ago)    15d
openshift-ingress                            router-default-68fdfcd4bd-w876v                          1/1     Running            5 (13h ago)    15d
openshift-kube-apiserver-operator            kube-apiserver-operator-7cd64755f8-wdqdb                 1/1     Running            1              15d
openshift-kube-apiserver                     kube-apiserver-crc-nv4w7-master-0                        5/5     Running            11 (14h ago)   14d
openshift-kube-apiserver                     revision-pruner-9-crc-nv4w7-master-0                     0/1     Completed          0              14d
openshift-kube-controller-manager-operator   kube-controller-manager-operator-5fc769ddf6-kfdts        1/1     Running            1              15d
openshift-kube-controller-manager            kube-controller-manager-crc-nv4w7-master-0               4/4     Running            17 (10h ago)   15d
openshift-kube-controller-manager            revision-pruner-6-crc-nv4w7-master-0                     0/1     Completed          0              14d
openshift-kube-scheduler-operator            openshift-kube-scheduler-operator-7c65bc5d9b-cxgvf       1/1     Running            1              15d
openshift-kube-scheduler                     openshift-kube-scheduler-crc-nv4w7-master-0              3/3     Running            5              15d
openshift-kube-scheduler                     revision-pruner-8-crc-nv4w7-master-0                     0/1     Completed          0              14d
openshift-machine-api                        machine-api-controllers-f4655d89f-xddjn                  7/7     Running            11             15d
openshift-machine-api                        machine-api-operator-5df8cd445c-vd65g                    2/2     Running            2              15d
openshift-machine-config-operator            machine-config-controller-57887dcc89-dldj2               2/2     Running            2              15d
openshift-machine-config-operator            machine-config-daemon-nr56c                              2/2     Running            2              15d
openshift-machine-config-operator            machine-config-operator-54ccccf58d-kjq6c                 1/1     Running            2              15d
openshift-machine-config-operator            machine-config-server-6kwq4                              1/1     Running            1              15d
openshift-marketplace                        certified-operators-p5gqr                                0/1     ImagePullBackOff   0              14h
openshift-marketplace                        certified-operators-qd5c7                                0/1     ImagePullBackOff   3              15d
openshift-marketplace                        community-operators-s6qx4                                0/1     ImagePullBackOff   0              14h
openshift-marketplace                        community-operators-sbz8c                                0/1     ImagePullBackOff   3              15d
openshift-marketplace                        marketplace-operator-857855d858-7dgs7                    1/1     Running            4              15d
openshift-marketplace                        redhat-marketplace-6ngq5                                 0/1     ImagePullBackOff   0              14h
openshift-marketplace                        redhat-marketplace-mfs6j                                 0/1     ImagePullBackOff   3              15d
openshift-marketplace                        redhat-operators-b68f6                                   0/1     ImagePullBackOff   0              14h
openshift-marketplace                        redhat-operators-pqgd7                                   0/1     ImagePullBackOff   3              15d
openshift-multus                             multus-additional-cni-plugins-tzbx8                      1/1     Running            1              15d
openshift-multus                             multus-admission-controller-g8h6h                        2/2     Running            2              15d
openshift-multus                             multus-z76dw                                             1/1     Running            1              15d
openshift-multus                             network-metrics-daemon-lz5tb                             2/2     Running            2              15d
openshift-network-diagnostics                network-check-source-b5988947-sstsh                      1/1     Running            3              15d
openshift-network-diagnostics                network-check-target-swnrs                               1/1     Running            1              15d
openshift-network-operator                   network-operator-5499bf8b85-85t62                        1/1     Running            1              15d
openshift-oauth-apiserver                    apiserver-5cb6ddc4c5-znqh2                               1/1     Running            7 (13h ago)    15d
openshift-operator-lifecycle-manager         catalog-operator-5655866444-r44kz                        1/1     Running            1              15d
openshift-operator-lifecycle-manager         collect-profiles-27837600-n8wbc                          0/1     Completed          0              33m
openshift-operator-lifecycle-manager         collect-profiles-27837615-tglqz                          0/1     Completed          0              18m
openshift-operator-lifecycle-manager         collect-profiles-27837630-4z6q5                          0/1     Completed          0              3m53s
openshift-operator-lifecycle-manager         olm-operator-86cd44d7b4-s2vhx                            1/1     Running            1              15d
openshift-operator-lifecycle-manager         package-server-manager-644454f859-zr8c8                  1/1     Running            1              15d
openshift-operator-lifecycle-manager         packageserver-77b84f7f67-bgqnr                           1/1     Running            1              15d
openshift-sdn                                sdn-bbzns                                                2/2     Running            2              15d
openshift-sdn                                sdn-controller-fhfck                                     2/2     Running            4              15d
openshift-service-ca-operator                service-ca-operator-6bf9d9448c-hvntv                     1/1     Running            1              15d
openshift-service-ca                         service-ca-66cfb699d9-nrzwg                              1/1     Running            1

NOTE: It seems that the following pods are reporting an ImagePullBackOff error:

openshift-marketplace                        certified-operators-p5gqr                                0/1     ImagePullBackOff   0              14h
openshift-marketplace                        certified-operators-qd5c7                                0/1     ImagePullBackOff   3              15d
openshift-marketplace                        community-operators-s6qx4                                0/1     ImagePullBackOff   0              14h
openshift-marketplace                        community-operators-sbz8c                                0/1     ImagePullBackOff   3              15d
openshift-marketplace                        redhat-marketplace-6ngq5                                 0/1     ImagePullBackOff   0              14h
openshift-marketplace                        redhat-marketplace-mfs6j                                 0/1     ImagePullBackOff   3              15d
openshift-marketplace                        redhat-operators-b68f6                                   0/1     ImagePullBackOff   0              14h
openshift-marketplace                        redhat-operators-pqgd7                                   0/1     ImagePullBackOff   3              15d

Step 7: Finally, I run this...

sudo crictl ps | head

... and get this output

CONTAINER           IMAGE                                                              CREATED             STATE               NAME                                          ATTEMPT             POD ID              POD
2c63fb32de216       39efb65aeb548bd8da8ceeee6fa494bdd5062bf3195057b5acfe1ff39fc34771   11 hours ago        Running             kube-controller-manager                       8                   6a219830d6054       kube-controller-manager-crc-nv4w7-master-0
796e6cf7210db       d0babe3730cb842a849d99ee3dcc6fe7827c62c29e07141380b78863c2e03b3f   14 hours ago        Running             ingress-operator                              5                   1b904d7ed8aee       ingress-operator-7f998586cd-6wxwb
add2d8815c89e       6c5702fcd61a863d8339c1224c884e0234bf77eba3cbd23d6ec8f78e7b12ae5a   14 hours ago        Running             registry                                      1                   c61be54a946fd       image-registry-5d4bf896f4-459rx
a56f30cb92bf1       b483ddcc2699b9c9e2ec77c66bfc407aafff0cb826195193dd61d7a59025d940   14 hours ago        Running             kube-rbac-proxy-mhc-mtrc                      1                   630db5d379f58       machine-api-controllers-f4655d89f-xddjn
9206f83da1426       a00c072e7d0abe733a2a30a9792cf619d1b918e57cc5d036ccd2f6d4f47f892c   14 hours ago        Running             oauth-apiserver                               7                   49b26ce4da9aa       apiserver-5cb6ddc4c5-znqh2
2a6fcbdd64efd       b483ddcc2699b9c9e2ec77c66bfc407aafff0cb826195193dd61d7a59025d940   14 hours ago        Running             kube-rbac-proxy-machine-mtrc                  1                   630db5d379f58       machine-api-controllers-f4655d89f-xddjn
4a0e4cedce987       87f86ced126bbd73bd81a2a9b97ec797e021c71ac91eea96b45bb22216f54163   14 hours ago        Running             router                                        5                   3167a16e71b10       router-default-68fdfcd4bd-w876v
ff44536f97727       e9ff29cded183e62a7ad10cdfb41f8e714732de4c493d1369fac1d2d688adb41   14 hours ago        Running             kube-multus-additional-cni-plugins            1                   c70ef39d4af6f       multus-additional-cni-plugins-tzbx8
069e6d7e1348b       b483ddcc2699b9c9e2ec77c66bfc407aafff0cb826195193dd61d7a59025d940   14 hours ago        Running             kube-rbac-proxy-machineset-mtrc               1                   630db5d379f58       machine-api-controllers-f4655d89f-xddjn

I hope this information is useful. Tell me please, does it help identify what the problem is?

Also, @cfergeau I ran crc start again and then tried to access the web page https://console-openshift-console.apps-crc.testing.

The web page displayed this error:

error | "server_error"
-- | --
error_description | "The authorization server encountered an unexpected condition that prevented it from fulfilling the request."
state | "8c500e8d"

I hope this information is useful. Tell me please, does it help identify what the problem is?

I'm afraid this helps identify where the problem is not :-/ Basically, oc get nodes output is fine, oc get co output is also fine. Would be worth testing these same steps right after a crc start failure, but this looks like what Praveen mentioned in https://github.com/crc-org/crc/issues/3442#issuecomment-1336744217 , the cluster being slower than expected to start, and crc erroring out because of this.

Is there any way to address the problem, maybe keep the crc start active for a longer period before it gives up?

we are https://github.com/crc-org/crc/issues/3443 as of now which is under discussion as of now. The best bet is you can wait more and after crc start fails then try crc stop => crc start again which might put addition time for operators and work since the memory in your system is only 16GB so it is not possible to provide more memory to crc VM.

@reselbob can you try https://github.com/praveenkumar/crc/releases/download/1.21.0/crc binary which created using https://github.com/crc-org/crc/pull/3445, with following options?

$ CRC_OC_TIMEOUT=40s ./crc start --log-level debug

I'll give the new binary a try @praveenkumar.

@praveenkumar Sad to say that running crc start with the new version is not working for me. Here is the output along with the verification of the groups the user reselbob belongs to. Also, I ran crc setup before running crc start.

$ CRC_OC_TIMEOUT=120s ./crc start --log-level debug
DEBU CRC version: 2.11.0+5699383a                 
DEBU OpenShift version: 4.11.13                   
DEBU Podman version: 4.2.0                        
DEBU Running 'crc start'                          
DEBU Total memory of system is 16628711424 bytes  
DEBU No new version available. The latest version is 2.11.0 
DEBU Checking file: /home/reselbob/.crc/machines/crc/.crc-exist 
DEBU Checking if systemd-resolved.service is running 
DEBU Running 'systemctl status systemd-resolved.service' 
DEBU systemd-resolved.service is already running  
INFO Checking if running as non-root              
INFO Checking if running inside WSL2              
INFO Checking if crc-admin-helper executable is cached 
DEBU Running '/home/reselbob/.crc/bin/crc-admin-helper-linux --version' 
DEBU Found crc-admin-helper-linux version 0.0.11  
DEBU crc-admin-helper executable already cached   
INFO Checking for obsolete admin-helper executable 
DEBU Checking if an older admin-helper executable is installed 
DEBU No older admin-helper executable found       
INFO Checking if running on a supported CPU architecture 
DEBU GOARCH is amd64 GOOS is linux                
INFO Checking minimum RAM requirements            
DEBU Total memory of system is 16628711424 bytes  
INFO Checking if crc executable symlink exists    
INFO Checking if Virtualization is enabled        
DEBU Checking if the vmx/svm flags are present in /proc/cpuinfo 
DEBU CPU virtualization flags are good            
INFO Checking if KVM is enabled                   
DEBU Checking if /dev/kvm exists                  
DEBU /dev/kvm was found                           
INFO Checking if libvirt is installed             
DEBU Checking if 'virsh' is available             
DEBU 'virsh' was found in /usr/bin/virsh          
DEBU Checking 'virsh capabilities' for libvirtd/qemu availability 
DEBU Running 'virsh --readonly --connect qemu:///system capabilities' 
DEBU Found x86_64 hypervisor with 'hvm' capabilities 
INFO Checking if user is part of libvirt group    
DEBU Checking if current user is part of the libvirt group 
DEBU currentUser.GroupIds() failed: user: GroupIds requires cgo 
DEBU Failed to get the groups user 'reselbob' belongs to 
WARN Preflight checks failed during `crc start`, please try to run `crc setup` first in case you haven't done so yet 
Failed to get the groups user 'reselbob' belongs to

[reselbob@reselbobpc-lan bin]$ groups reselbob
reselbob : reselbob wheel libvirt

That's a known issue with cross-built linux binaries, we'll need to provide a new one

Is there anything I can do in the meantime @cfergeau ? Also, please allow me to offer a sincere thanks to you and to @praveenkumar for all the help you are giving me on this matter. Getting crc up and running on this Fedora machine is critical path for work I am doing.

Not fully sure it's going to be accessible, but you can try https://drive.google.com/file/d/1xnZ7u_V7D5aloUcpPBaE5OtoFAJUl95F/view?usp=sharing instead of Praveen's binary. Its sha256sum is f46e07474dcd271f142be722009f9606ab2ac94a3ea41ca0f16c93d8215948b7

The version posted above moves the through thecrc start process, but it still can't execute get nodes after 227 tries.

Here is the start command:

CRC_OC_TIMEOUT=120s ./crc start --log-level debug

Here is the last set of screen dumps before it gives up @cfergeau

DEBU RetryAfter timeout after 227 tries           
DEBU Making call to close driver server           
DEBU (crc) Calling .Close                         
DEBU Successfully made call to close driver server 
DEBU Making call to close connection to plugin binary 
Error waiting for apiserver: Temporary error: ssh command error:
command : timeout 2m0s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 125

Should I increase the value of CRC_OC_TIMEOUT?

@reselbob can you again https://github.com/crc-org/crc/wiki/Debugging-guide#entering-the-vm use this and check the journal logs of kubelet service ?

Wait, that's not the complete dump.

upload it to gist and also sudo crictl ps -a

As requested on crictl

sudo crictl ps -a
CONTAINER           IMAGE                                                              CREATED             STATE               NAME                                          ATTEMPT             POD ID              POD
7319054cb75da       b41c2d079cb83be67ccb2eb97486a0e25aef2782ba9e0735b491e130cb94c157   2 minutes ago       Exited              collect-profiles                              0                   3e9fade443ac3       collect-profiles-27841935-9pv65
97a4889006e19       a00c072e7d0abe733a2a30a9792cf619d1b918e57cc5d036ccd2f6d4f47f892c   10 minutes ago      Running             oauth-apiserver                               2                   6bbfbf83c3cc0       apiserver-5cb6ddc4c5-znqh2
09db13f3e2dca       39efb65aeb548bd8da8ceeee6fa494bdd5062bf3195057b5acfe1ff39fc34771   10 minutes ago      Running             kube-controller-manager                       10                  0d8e24163f835       kube-controller-manager-crc-nv4w7-master-0
9bd68c3725c73       39efb65aeb548bd8da8ceeee6fa494bdd5062bf3195057b5acfe1ff39fc34771   11 minutes ago      Exited              kube-controller-manager                       9                   0d8e24163f835       kube-controller-manager-crc-nv4w7-master-0
df9f0805f849b       ce3f348ab8b2d141dd0a318c18ae30bf74248bea0d4f94f936570f67ea5b5580   11 minutes ago      Running             machine-controller                            4                   836d2a296a79e       machine-api-controllers-f4655d89f-xddjn
f0ec0cf4e95f6       b41c2d079cb83be67ccb2eb97486a0e25aef2782ba9e0735b491e130cb94c157   17 minutes ago      Exited              collect-profiles                              0                   7c93b6329fc69       collect-profiles-27841920-j6cvm
622d78c009a6f       ce3f348ab8b2d141dd0a318c18ae30bf74248bea0d4f94f936570f67ea5b5580   25 minutes ago      Exited              machine-controller                            3                   836d2a296a79e       machine-api-controllers-f4655d89f-xddjn
13a8a9715160a       b41c2d079cb83be67ccb2eb97486a0e25aef2782ba9e0735b491e130cb94c157   32 minutes ago      Exited              collect-profiles                              0                   70e1bb4f1a7fb       collect-profiles-27841905-s8rq7
14875da91755e       a00c072e7d0abe733a2a30a9792cf619d1b918e57cc5d036ccd2f6d4f47f892c   16 hours ago        Exited              oauth-apiserver                               1                   6bbfbf83c3cc0       apiserver-5cb6ddc4c5-znqh2
a01c43adf71c6       7db2d8f1d3c512e8ee18ee4f83ba649eb86d044fdac55be1a994bb8ea0cf9f53   16 hours ago        Exited              image-pruner                                  0                   bde4aef032187       image-pruner-27840960-8dww4
447b6ccda8d5c       f6b063c7c208cee2a8db639ca9a5dacdd22b8af8b13e369007e9b339398066c6   19 hours ago        Running             console                                       1                   b57efad29a054       console-64bf795968-42dxz
f246fbcc23f96       87f86ced126bbd73bd81a2a9b97ec797e021c71ac91eea96b45bb22216f54163   19 hours ago        Running             router                                        2                   b7ce68ec02c25       router-default-68fdfcd4bd-w876v
b421fac66a9cb       b483ddcc2699b9c9e2ec77c66bfc407aafff0cb826195193dd61d7a59025d940   19 hours ago        Running             kube-rbac-proxy-mhc-mtrc                      0                   836d2a296a79e       machine-api-controllers-f4655d89f-xddjn
8cc5ebd222191       b483ddcc2699b9c9e2ec77c66bfc407aafff0cb826195193dd61d7a59025d940   19 hours ago        Running             kube-rbac-proxy-machine-mtrc                  0                   836d2a296a79e       machine-api-controllers-f4655d89f-xddjn
9aa95c057ea97       6c5702fcd61a863d8339c1224c884e0234bf77eba3cbd23d6ec8f78e7b12ae5a   19 hours ago        Running             registry                                      0                   41ba613615744       image-registry-5d4bf896f4-459rx
7d35113b00e0b       b483ddcc2699b9c9e2ec77c66bfc407aafff0cb826195193dd61d7a59025d940   19 hours ago        Running             kube-rbac-proxy-machineset-mtrc               0                   836d2a296a79e       machine-api-controllers-f4655d89f-xddjn
03251372a8d2e       d0babe3730cb842a849d99ee3dcc6fe7827c62c29e07141380b78863c2e03b3f   19 hours ago        Running             ingress-operator                              1                   306688f131835       ingress-operator-7f998586cd-6wxwb
0781dcc7e9f23       1acadf8a9029f3e3ce8cc86e79dc2df36a5316c3ea90c80f94ffd08c713a3eb4   19 hours ago        Running             machine-api-operator                          0                   b26eec107e38b       machine-api-operator-5df8cd445c-vd65g
0a948d9ac9edf       1acadf8a9029f3e3ce8cc86e79dc2df36a5316c3ea90c80f94ffd08c713a3eb4   19 hours ago        Running             machine-healthcheck-controller                0                   836d2a296a79e       machine-api-controllers-f4655d89f-xddjn
89298b943e5a3       7fae034a29b58fa94e1b2d00bc2eb292d742f06c274fa45620887c83619cab90   19 hours ago        Running             console-operator                              0                   31fc556dc4ef8       console-operator-7b68655f7c-vjjj5
d908a1aaec9f2       b483ddcc2699b9c9e2ec77c66bfc407aafff0cb826195193dd61d7a59025d940   19 hours ago        Running             kube-rbac-proxy                               0                   92bca682e421e       dns-default-dzrll
87992289a9040       b483ddcc2699b9c9e2ec77c66bfc407aafff0cb826195193dd61d7a59025d940   19 hours ago        Running             kube-rbac-proxy                               0                   b26eec107e38b       machine-api-operator-5df8cd445c-vd65g
a1faa3cffdc5e       a00c072e7d0abe733a2a30a9792cf619d1b918e57cc5d036ccd2f6d4f47f892c   19 hours ago        Exited              fix-audit-permissions                         0                   6bbfbf83c3cc0       apiserver-5cb6ddc4c5-znqh2
c7f73117be5f2       52b78e09560b98fe4a8b3e53226190fb5d449c79c9fd2c3087c5270b83d1380a   19 hours ago        Running             download-server                               1                   0e5bec55220f6       downloads-5587558f5f-kg5bs
db45a6d534a22       e9ff29cded183e62a7ad10cdfb41f8e714732de4c493d1369fac1d2d688adb41   19 hours ago        Running             kube-multus-additional-cni-plugins            0                   fb4ca1c12ed0d       multus-additional-cni-plugins-tzbx8
83bb8f4303117       1acadf8a9029f3e3ce8cc86e79dc2df36a5316c3ea90c80f94ffd08c713a3eb4   19 hours ago        Running             nodelink-controller                           0                   836d2a296a79e       machine-api-controllers-f4655d89f-xddjn
49d15e9e9d66b       e20a6f493d3dd0601b147f1c0f127a69e99dd74f7652ac5356417675647457dd   19 hours ago        Running             csi-provisioner                               0                   1305d8b71ee5d       csi-hostpathplugin-l6f4f
0a165d8ce8b96       24d8a08997d22f349a0d600c119b8d85709c1fc6d6e999212af2832c1245a58f   19 hours ago        Running             cluster-node-tuning-operator                  0                   1ef3868b66735       cluster-node-tuning-operator-659594b5cf-b24mh
e9005fe82cd82       170dd9027c930cf47bf71702a12a970d06fb10e79c5cf0a704d404084dc58720   19 hours ago        Running             openshift-apiserver-check-endpoints           0                   00b001d6bf2c7       apiserver-78f94fdb47-265dd
5107203d0a2d6       170dd9027c930cf47bf71702a12a970d06fb10e79c5cf0a704d404084dc58720   19 hours ago        Running             kube-apiserver-operator                       0                   276a0bf0f65a6       kube-apiserver-operator-7cd64755f8-wdqdb
d7d324108112f       81c92ad2da4a74d478008dcc8fd37c5d0f8b0dbf7684b10a13b08094d024bcc8   19 hours ago        Running             cluster-image-registry-operator               0                   3c6be98a589b0       cluster-image-registry-operator-759dbc577c-28qjt
90debf864d4ba       a70bc8c83a0b3324262cac886237118b934e15136cf971fc07f59aaffa74ca88   19 hours ago        Running             dns                                           0                   92bca682e421e       dns-default-dzrll
aec866d586af3       77dfd61254f676b3a63f15d6151ff2c9102c88870f3cadcda12c5351dd0c4877   19 hours ago        Running             cluster-samples-operator-watch                0                   dbb93391be6aa       cluster-samples-operator-b588d95bd-xg8w5
de4d003aa8bfb       87f86ced126bbd73bd81a2a9b97ec797e021c71ac91eea96b45bb22216f54163   19 hours ago        Exited              router                                        1                   b7ce68ec02c25       router-default-68fdfcd4bd-w876v
2c1380b34e529       1e09cc7caf35ab1edfb128ec71fecd50f597f37f961f954663f49e7489d8dda6   19 hours ago        Exited              whereabouts-cni                               0                   fb4ca1c12ed0d       multus-additional-cni-plugins-tzbx8
2994e55711865       04677434703e9b819188ef31c83fd501b54a85c43d26ee9bc4602f13c1d12ae4   19 hours ago        Running             openshift-apiserver                           0                   00b001d6bf2c7       apiserver-78f94fdb47-265dd
54f1e044ee30b       26443969e8864d385273cf5363f115ff4edf75c615f82e9d6de449bf133888dc   19 hours ago        Running             liveness-probe                                0                   1305d8b71ee5d       csi-hostpathplugin-l6f4f
5d9a182a47e25       e42db308831e05b45d81c6fe5391c1ee0e8a2acabf04d71e44edcd46569b1144   19 hours ago        Running             network-check-target-container                0                   38555cb935092       network-check-target-swnrs
09b3d4a705407       b76a80a63519564785469746b4e5d86521abf7fe2b11da9f7ed00918403ef014   19 hours ago        Running             authentication-operator                       0                   b55587344c1eb       authentication-operator-dbd5bd768-hsrbv
22afad7de7a08       b41c2d079cb83be67ccb2eb97486a0e25aef2782ba9e0735b491e130cb94c157   19 hours ago        Running             catalog-operator                              0                   2cf5763a91477       catalog-operator-5655866444-r44kz
241e95025d7f0       8d6d50f1e0f5a8efbc4f5404880092808e0753cc51559e849ac29189cabcb040   19 hours ago        Running             oauth-proxy                                   0                   bf40f118840b1       machine-config-controller-57887dcc89-dldj2
5a0a909d44223       b483ddcc2699b9c9e2ec77c66bfc407aafff0cb826195193dd61d7a59025d940   19 hours ago        Running             kube-rbac-proxy                               0                   982f983304f57       multus-admission-controller-g8h6h
51303b477d1d2       52b78e09560b98fe4a8b3e53226190fb5d449c79c9fd2c3087c5270b83d1380a   19 hours ago        Exited              download-server                               0                   0e5bec55220f6       downloads-5587558f5f-kg5bs
c27cdea9bca8d       b41c2d079cb83be67ccb2eb97486a0e25aef2782ba9e0735b491e130cb94c157   19 hours ago        Running             packageserver                                 0                   7a06ee9b705db       packageserver-77b84f7f67-bgqnr
d57ade18d826b       dffa5b0c651bebf5ad2149ece2ee89b22498403d9c44f24d583490209d4fe686   19 hours ago        Running             openshift-controller-manager-operator         0                   0b88b6cc05ecd       openshift-controller-manager-operator-7c4dc65c97-qljfm
0ecdc1ec4874b       77dfd61254f676b3a63f15d6151ff2c9102c88870f3cadcda12c5351dd0c4877   19 hours ago        Running             cluster-samples-operator                      0                   dbb93391be6aa       cluster-samples-operator-b588d95bd-xg8w5
a24990ba9ff02       b41c2d079cb83be67ccb2eb97486a0e25aef2782ba9e0735b491e130cb94c157   19 hours ago        Running             package-server-manager                        0                   77ddeb803e9dc       package-server-manager-644454f859-zr8c8
62fba7c9cbb1c       1e09cc7caf35ab1edfb128ec71fecd50f597f37f961f954663f49e7489d8dda6   19 hours ago        Exited              whereabouts-cni-bincopy                       0                   fb4ca1c12ed0d       multus-additional-cni-plugins-tzbx8
f1512a706ef59       26b3b6bd7150ed41f02b6c2ad9289238e911b079caa413a3198b53b56abd47b2   19 hours ago        Running             service-ca-operator                           0                   98ce33b7696da       service-ca-operator-6bf9d9448c-hvntv
64b44619b0110       535d1d71e2aca0b8d9848a960b92cc65030fb01d603cf6e38da8a42378cb00ee   19 hours ago        Running             openshift-config-operator                     0                   cbaae7ecf7cd8       openshift-config-operator-674f98f947-9c2r8
51e51d4b64ef4       f030954c81158addfb8189e0e5f0c6482532a8206a72de8a24e18c96d7777d5b   19 hours ago        Running             openshift-apiserver-operator                  0                   85e8c9119f861       openshift-apiserver-operator-6c9b9cb4bd-cd9g2
60430484e614e       f6b063c7c208cee2a8db639ca9a5dacdd22b8af8b13e369007e9b339398066c6   19 hours ago        Exited              console                                       0                   b57efad29a054       console-64bf795968-42dxz
697bcfbcf75df       b483ddcc2699b9c9e2ec77c66bfc407aafff0cb826195193dd61d7a59025d940   19 hours ago        Running             kube-rbac-proxy                               0                   306688f131835       ingress-operator-7f998586cd-6wxwb
738a2aa7bfbea       b483ddcc2699b9c9e2ec77c66bfc407aafff0cb826195193dd61d7a59025d940   19 hours ago        Running             kube-rbac-proxy                               0                   152783a8bcdbb       dns-operator-6998f45999-lrqsb
af819892c760a       1acadf8a9029f3e3ce8cc86e79dc2df36a5316c3ea90c80f94ffd08c713a3eb4   19 hours ago        Running             machineset-controller                         0                   836d2a296a79e       machine-api-controllers-f4655d89f-xddjn
da6e09179b365       9fd1583d37cffe26f37645a51be318875000afe3cb50b248f292cfb3b5f95a45   19 hours ago        Exited              pruner                                        0                   a7d1a5a92b06a       revision-pruner-6-crc-nv4w7-master-0
e01a78a6b070a       aa6e024ad7e1f07f813d365cf21a0bc9a52b0e2146027a7c638ab0b2a9685f8a   19 hours ago        Running             controller-manager                            0                   c11fb5787eaf3       controller-manager-wbxpj
7ef490d3b1b10       e37110150618c14f85c69547efc0ebb0acb953076cd836d8175dca5575996a75   19 hours ago        Running             kube-scheduler-operator-container             0                   b5090700f1648       openshift-kube-scheduler-operator-7c65bc5d9b-cxgvf
301fa36b8ad4d       04677434703e9b819188ef31c83fd501b54a85c43d26ee9bc4602f13c1d12ae4   19 hours ago        Exited              fix-audit-permissions                         0                   00b001d6bf2c7       apiserver-78f94fdb47-265dd
1e3854e98ca39       d9aa3b26a4d15fe8f2618f6cb2df6ea46cc919dff2431c395c96f0f5fa8440cf   19 hours ago        Running             machine-config-operator                       0                   2e44e2654333e       machine-config-operator-54ccccf58d-kjq6c
2e4bd46f92664       0d6229ff85fd2f078030499bd312d2c1ed1b62d202abe16ff5d1d690cfff6eda   19 hours ago        Running             node-driver-registrar                         0                   1305d8b71ee5d       csi-hostpathplugin-l6f4f
0f675c98926f8       8251abf898bf29f1624b322df60ae88ae4166c2a83e2591977fafaf04667bb65   19 hours ago        Running             etcd-operator                                 0                   69ca763f66a1e       etcd-operator-78974f57b9-znq2m
45e3b1e2b9cc6       e42db308831e05b45d81c6fe5391c1ee0e8a2acabf04d71e44edcd46569b1144   19 hours ago        Running             check-endpoints                               0                   4892937de5bb6       network-check-source-b5988947-sstsh
78672a05ed4fa       9fd1583d37cffe26f37645a51be318875000afe3cb50b248f292cfb3b5f95a45   19 hours ago        Running             kube-controller-manager-operator              0                   2a2edf7b71225       kube-controller-manager-operator-5fc769ddf6-kfdts
115038798bfe0       5c40f5c3dcf2dc989b8c115a9a8191dc85db54f84ef4aae1bf4ed0cf13f14d90   19 hours ago        Running             oauth-openshift                               0                   71162b34bad14       oauth-openshift-76f87b4579-t54rz
4297270928bfc       d9aa3b26a4d15fe8f2618f6cb2df6ea46cc919dff2431c395c96f0f5fa8440cf   19 hours ago        Running             machine-config-controller                     0                   bf40f118840b1       machine-config-controller-57887dcc89-dldj2
2affee758c97a       170dd9027c930cf47bf71702a12a970d06fb10e79c5cf0a704d404084dc58720   19 hours ago        Exited              pruner                                        0                   a7fcfd968ec5a       revision-pruner-9-crc-nv4w7-master-0
13e481cc9f9ff       a868b761b5861a75805ca5a9f093287b1cd2aa7df7e82155c5835fe522b1f5fc   19 hours ago        Running             multus-admission-controller                   0                   982f983304f57       multus-admission-controller-g8h6h
c63006360e884       26b3b6bd7150ed41f02b6c2ad9289238e911b079caa413a3198b53b56abd47b2   19 hours ago        Running             service-ca-controller                         0                   8830cdb52a523       service-ca-66cfb699d9-nrzwg
da3568d8a8621       b483ddcc2699b9c9e2ec77c66bfc407aafff0cb826195193dd61d7a59025d940   19 hours ago        Running             kube-rbac-proxy                               0                   7cc580a2b33ba       network-metrics-daemon-lz5tb
849c056938022       1bf520aa85b3c121c2ec696b4e45c64464eb5652ca5850476051e492dd526231   19 hours ago        Running             hostpath-provisioner                          0                   1305d8b71ee5d       csi-hostpathplugin-l6f4f
ede84100882c0       d0babe3730cb842a849d99ee3dcc6fe7827c62c29e07141380b78863c2e03b3f   19 hours ago        Exited              ingress-operator                              0                   306688f131835       ingress-operator-7f998586cd-6wxwb
fa850ca1f2d83       dcecd3235ba7c422b13c6080eccf586f5189a15d9193df3aadf6eb9bc5a6caf1   19 hours ago        Exited              routeoverride-cni                             0                   fb4ca1c12ed0d       multus-additional-cni-plugins-tzbx8
23f6c0b242c5d       b41c2d079cb83be67ccb2eb97486a0e25aef2782ba9e0735b491e130cb94c157   19 hours ago        Running             olm-operator                                  0                   63bd73c805323       olm-operator-86cd44d7b4-s2vhx
08d5b3fda778e       640f4476f4091a54de93f039aad33e58ba4f634b88eb6e96c1c067aae7ddac84   19 hours ago        Running             dns-operator                                  0                   152783a8bcdbb       dns-operator-6998f45999-lrqsb
ecd71c7288a5c       cf5834044832bbc57e833912e851631e617ac6c7a1d70a618a5b8c939f4c9eab   19 hours ago        Running             network-metrics-daemon                        0                   7cc580a2b33ba       network-metrics-daemon-lz5tb
787bc1e0fb276       cac8c4fc96acc8a9f0d441d4b328d286583ced8bb35bfe6272fc412aa027a8bd   19 hours ago        Exited              bond-cni-plugin                               0                   fb4ca1c12ed0d       multus-additional-cni-plugins-tzbx8
6c28b36cc16cf       209bb515373ffc9ec86408623f524abb0335219d6d51502c43eda4aa45e24a05   19 hours ago        Running             marketplace-operator                          0                   95a310bd6bec9       marketplace-operator-857855d858-7dgs7
dfe76443840c0       d0babe3730cb842a849d99ee3dcc6fe7827c62c29e07141380b78863c2e03b3f   19 hours ago        Running             serve-healthcheck-canary                      0                   53b294bc9d231       ingress-canary-dtwfl
9968e2646a60d       e37110150618c14f85c69547efc0ebb0acb953076cd836d8175dca5575996a75   19 hours ago        Exited              pruner                                        0                   a0669222b51e0       revision-pruner-8-crc-nv4w7-master-0
7b4622e45eddf       9e8879a67f4f4e96c2e1991dc3286dc7bb1ea72da1e3a6ff514e565db8a2de8a   19 hours ago        Exited              cni-plugins                                   0                   fb4ca1c12ed0d       multus-additional-cni-plugins-tzbx8
bc4d0ea161937       b483ddcc2699b9c9e2ec77c66bfc407aafff0cb826195193dd61d7a59025d940   19 hours ago        Running             kube-rbac-proxy                               0                   9ee7926ed5bab       sdn-bbzns
f79cd1df1fc4e       8d6d50f1e0f5a8efbc4f5404880092808e0753cc51559e849ac29189cabcb040   19 hours ago        Running             oauth-proxy                                   0                   9c8e3ca392e2f       machine-config-daemon-nr56c
ebbe3c9dd842b       2d432ee038fbf7642eaeb6ea2a205c1acc2b669cb66974c38809f9e4b12316d2   19 hours ago        Running             machine-approver-controller                   0                   473fead9bc2cb       machine-approver-7b5bd86fbb-zqtlm
96066c56a8201       b483ddcc2699b9c9e2ec77c66bfc407aafff0cb826195193dd61d7a59025d940   19 hours ago        Running             kube-rbac-proxy                               0                   5e71783897e4d       sdn-controller-fhfck
8331b716bc279       7db2d8f1d3c512e8ee18ee4f83ba649eb86d044fdac55be1a994bb8ea0cf9f53   19 hours ago        Running             dns-node-resolver                             0                   1fd324da13107       node-resolver-kvr46
b36b7e23a7875       1a257fbd4a61b9ea101e6a1e5ba88a5485f3d45ac771d49c0856c98605094b9a   19 hours ago        Exited              egress-router-binary-copy                     0                   fb4ca1c12ed0d       multus-additional-cni-plugins-tzbx8
98af05850cd79       24d8a08997d22f349a0d600c119b8d85709c1fc6d6e999212af2832c1245a58f   19 hours ago        Running             tuned                                         0                   8b7c9b343921a       tuned-fq4dk
3bb9830c9cd6a       d9aa3b26a4d15fe8f2618f6cb2df6ea46cc919dff2431c395c96f0f5fa8440cf   19 hours ago        Running             machine-config-server                         0                   2282aff3d45ec       machine-config-server-6kwq4
0c368b0581f7f       d9aa3b26a4d15fe8f2618f6cb2df6ea46cc919dff2431c395c96f0f5fa8440cf   19 hours ago        Running             machine-config-daemon                         0                   9c8e3ca392e2f       machine-config-daemon-nr56c
2315ba22f5f18       e42db308831e05b45d81c6fe5391c1ee0e8a2acabf04d71e44edcd46569b1144   19 hours ago        Running             network-operator                              0                   aad08c9ea47eb       network-operator-5499bf8b85-85t62
af96f12498a73       d453a8a4edfb3a00a01efe4956d3d9d098d5665feb5290d9e62e73750fe0b83c   19 hours ago        Running             sdn                                           0                   9ee7926ed5bab       sdn-bbzns
88518d82ed216       e9ff29cded183e62a7ad10cdfb41f8e714732de4c493d1369fac1d2d688adb41   19 hours ago        Running             kube-multus                                   0                   7bbc375f1799f       multus-z76dw
d593cbe271183       b483ddcc2699b9c9e2ec77c66bfc407aafff0cb826195193dd61d7a59025d940   19 hours ago        Running             kube-rbac-proxy                               0                   473fead9bc2cb       machine-approver-7b5bd86fbb-zqtlm
7c185514d966a       d453a8a4edfb3a00a01efe4956d3d9d098d5665feb5290d9e62e73750fe0b83c   19 hours ago        Running             sdn-controller                                0                   5e71783897e4d       sdn-controller-fhfck
3b667b8d29941       abdf6f2afb6bb0d92789d95a8a6ed51f9ccac6631a512ff695910aea4c98e6cb   19 hours ago        Running             cluster-version-operator                      0                   f08740370c8c0       cluster-version-operator-8475fc4c6-sfkj6
b49b1611312fd       6c5702fcd61a863d8339c1224c884e0234bf77eba3cbd23d6ec8f78e7b12ae5a   19 hours ago        Running             node-ca                                       0                   ab1ee0653a93d       node-ca-q4f98
619d2b57cc918       8251abf898bf29f1624b322df60ae88ae4166c2a83e2591977fafaf04667bb65   19 hours ago        Running             etcd-readyz                                   0                   944c2d44afcb6       etcd-crc-nv4w7-master-0
6e63c2ec2e96a       8251abf898bf29f1624b322df60ae88ae4166c2a83e2591977fafaf04667bb65   19 hours ago        Running             etcd-health-monitor                           0                   944c2d44afcb6       etcd-crc-nv4w7-master-0
052d749adb025       c223f5d86120bc6e4059b4e58dd54e3508b1ed821058063ae74be795e32ad075   19 hours ago        Running             etcd-metrics                                  0                   944c2d44afcb6       etcd-crc-nv4w7-master-0
efbda5470b0ce       e37110150618c14f85c69547efc0ebb0acb953076cd836d8175dca5575996a75   19 hours ago        Running             kube-scheduler-recovery-controller            0                   e48397f59cdfe       openshift-kube-scheduler-crc-nv4w7-master-0
476fdc747ab8a       c223f5d86120bc6e4059b4e58dd54e3508b1ed821058063ae74be795e32ad075   19 hours ago        Running             etcd                                          0                   944c2d44afcb6       etcd-crc-nv4w7-master-0
4a77b16caaea3       170dd9027c930cf47bf71702a12a970d06fb10e79c5cf0a704d404084dc58720   19 hours ago        Running             kube-apiserver-check-endpoints                0                   132b9f83f7a8b       kube-apiserver-crc-nv4w7-master-0
087cc49dd465e       170dd9027c930cf47bf71702a12a970d06fb10e79c5cf0a704d404084dc58720   19 hours ago        Running             kube-apiserver-insecure-readyz                0                   132b9f83f7a8b       kube-apiserver-crc-nv4w7-master-0
9562a2fdacdb9       e37110150618c14f85c69547efc0ebb0acb953076cd836d8175dca5575996a75   19 hours ago        Running             kube-scheduler-cert-syncer                    0                   e48397f59cdfe       openshift-kube-scheduler-crc-nv4w7-master-0
0b70ba2a9bf79       c223f5d86120bc6e4059b4e58dd54e3508b1ed821058063ae74be795e32ad075   19 hours ago        Running             etcdctl                                       0                   944c2d44afcb6       etcd-crc-nv4w7-master-0
bf3992c9b2c33       170dd9027c930cf47bf71702a12a970d06fb10e79c5cf0a704d404084dc58720   19 hours ago        Running             kube-apiserver-cert-regeneration-controller   0                   132b9f83f7a8b       kube-apiserver-crc-nv4w7-master-0
b9a0ed2d82592       c223f5d86120bc6e4059b4e58dd54e3508b1ed821058063ae74be795e32ad075   19 hours ago        Exited              etcd-resources-copy                           0                   944c2d44afcb6       etcd-crc-nv4w7-master-0
cd8c1be4914c9       39efb65aeb548bd8da8ceeee6fa494bdd5062bf3195057b5acfe1ff39fc34771   19 hours ago        Running             kube-scheduler                                0                   e48397f59cdfe       openshift-kube-scheduler-crc-nv4w7-master-0
36fdf2410ade9       170dd9027c930cf47bf71702a12a970d06fb10e79c5cf0a704d404084dc58720   19 hours ago        Running             kube-apiserver-cert-syncer                    0                   132b9f83f7a8b       kube-apiserver-crc-nv4w7-master-0
81dadd8bf95c8       9fd1583d37cffe26f37645a51be318875000afe3cb50b248f292cfb3b5f95a45   19 hours ago        Running             kube-controller-manager-recovery-controller   0                   0d8e24163f835       kube-controller-manager-crc-nv4w7-master-0
9b2816edd3f1b       39efb65aeb548bd8da8ceeee6fa494bdd5062bf3195057b5acfe1ff39fc34771   19 hours ago        Running             kube-apiserver                                0                   132b9f83f7a8b       kube-apiserver-crc-nv4w7-master-0
b966ed8fd9865       9fd1583d37cffe26f37645a51be318875000afe3cb50b248f292cfb3b5f95a45   19 hours ago        Running             kube-controller-manager-cert-syncer           0                   0d8e24163f835       kube-controller-manager-crc-nv4w7-master-0
882e8635c5ccc       c223f5d86120bc6e4059b4e58dd54e3508b1ed821058063ae74be795e32ad075   19 hours ago        Exited              etcd-ensure-env-vars                          0                   944c2d44afcb6       etcd-crc-nv4w7-master-0
6830857826e7b       7105c5bef6da1558999b383de2a503e2e11e2cd9d7d3ccf1c2a0284f38d37276   19 hours ago        Running             cluster-policy-controller                     0                   0d8e24163f835       kube-controller-manager-crc-nv4w7-master-0
5cc8eaf59217e       39efb65aeb548bd8da8ceeee6fa494bdd5062bf3195057b5acfe1ff39fc34771   19 hours ago        Exited              wait-for-host-port                            0                   e48397f59cdfe       openshift-kube-scheduler-crc-nv4w7-master-0
ec20fa78e7aeb       39efb65aeb548bd8da8ceeee6fa494bdd5062bf3195057b5acfe1ff39fc34771   19 hours ago        Exited              setup                                         0                   132b9f83f7a8b       kube-apiserver-crc-nv4w7-master-0
919d2b7349296       c223f5d86120bc6e4059b4e58dd54e3508b1ed821058063ae74be795e32ad075   19 hours ago        Exited              setup

Here is a link to the journalctl logs @praveenkumar. It's filtered according to kubelet.service :

https://drive.google.com/file/d/10ezF4fmtOjiEJuEHDfXW3uHKOG5TDlFA/view?usp=sharing

@reselbob I went through the kubelet logs and didn't find anything suspicious, one of the logs https://github.com/crc-org/crc/issues/3442#issuecomment-1337701878 you mentioned that from inside the VM everything works as expected so now how come from ssh command when we want to run same command it is not provide us same output :(

$ oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig

Also https://github.com/crc-org/crc/issues/3442#issuecomment-1341599469 Process exited with status 125 which means somehow timeout command itself failing, not sure why :(

@praveenkumar I am a bit confused; not sure what you want me to do; do you want me to run the oc get nodes command listed above?

@reselbob I want to you to perform following after the failure

SSH to the vm and try to execute oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig if that works.
Then you need to execute following and see if have same issue

ssh -i ~/.crc/machines/crc/id_ecdsa -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null  core@192.168.130.11 -- timeout 30s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig

and also check above command without any timeout

ssh -i ~/.crc/machines/crc/id_ecdsa -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null  core@192.168.130.11 -- oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig

Here is the result of the first command you asked me to execute @praveenkumar :

[core@crc-nv4w7-master-0 ~]$ oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
NAME                 STATUS   ROLES           AGE   VERSION
crc-nv4w7-master-0   Ready    master,worker   23d   v1.24.6+5157800

Here is the second command @praveenkumar :

[reselbob@reselbobpc-lan ~]$ ssh -i ~/.crc/machines/crc/id_ecdsa -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null  core@192.168.130.11 -- timeout 30s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
Warning: Permanently added '192.168.130.11' (ED25519) to the list of known hosts.
NAME                 STATUS   ROLES           AGE   VERSION
crc-nv4w7-master-0   Ready    master,worker   23d   v1.24.6+5157800

Here is the third @praveenkumar:

[reselbob@reselbobpc-lan ~]$ ssh -i ~/.crc/machines/crc/id_ecdsa -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null  core@192.168.130.11 -- oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
Warning: Permanently added '192.168.130.11' (ED25519) to the list of known hosts.
NAME                 STATUS   ROLES           AGE   VERSION
crc-nv4w7-master-0   Ready    master,worker   23d   v1.24.6+5157800

I can login to the crc instance using oc like so @praveenkumar

  $ oc login -u developer https://api.crc.testing:6443
[reselbob@reselbobpc-lan ~]$ eval $(crc oc-env)
[reselbob@reselbobpc-lan ~]$ oc login -u developer https://api.crc.testing:6443
The server uses a certificate signed by an unknown authority.
You can bypass the certificate check, but any data you send to the server could be intercepted by others.
Use insecure connections? (y/n): y

Authentication required for https://api.crc.testing:6443 (openshift)
Console URL: https://api.crc.testing:6443/console
Username: developer
Password: 
Login failed (401 Unauthorized)
Verify you have provided correct credentials.
[reselbob@reselbobpc-lan ~]$ oc login -u developer https://api.crc.testing:6443
The server uses a certificate signed by an unknown authority.
You can bypass the certificate check, but any data you send to the server could be intercepted by others.
Use insecure connections? (y/n): y

Authentication required for https://api.crc.testing:6443 (openshift)
Console URL: https://api.crc.testing:6443/console
Username: developer
Password: 
Login successful.

You don't have any projects. You can try to create a new project, by running

    oc new-project <projectname>

Welcome! See 'oc help' to get started.

Did you try these commands immediatly after crc start failed? Or did crc start succeed this time?

And yet, trying to login as kubeadmin fails:

[reselbob@reselbobpc-lan ~]$ oc login -u kubeadmin https://api.crc.testing:6443
Authentication required for https://api.crc.testing:6443 (openshift)
Console URL: https://api.crc.testing:6443/console
Username: kubeadmin
Password: 
Login failed (401 Unauthorized)
Verify you have provided correct credentials.

Here is the output of crc start:

[reselbob@reselbobpc-lan ~]$ crc start
INFO Loading bundle: crc_libvirt_4.11.13_amd64... 
INFO A CRC VM for OpenShift 4.11.13 is already running 
Started the OpenShift cluster.

The server is accessible via web console at:
  https://console-openshift-console.apps-crc.testing

Log in as administrator:
  Username: kubeadmin
  Password: t9oLq-sUUeM-iC5sS-wNdN9

Log in as user:
  Username: developer
  Password: developer

Use the 'oc' command line interface:
  $ eval $(crc oc-env)
  $ oc login -u developer https://api.crc.testing:6443

Still can't access the site via the browser:

Screen Shot 2022-12-13 at 8 22 15 AM

Please don't hesitate to let me know any additional tasks I can perform to help get to the bottom of things. I really appreciate all the attention you are giving me.

Can you try executing following commands and let us know the results? I am hoping you can still login to developer user but not to kubeadmin one.

$ oc --kubeconfig=$HOME/.crc/machines/crc/kubeconfig get co

Here you go, logged in as developer.

$ oc --kubeconfig=$HOME/.crc/machines/crc/kubeconfig get co 
error: stat /home/reselbob/.crc/machines/crc/kubeconfig: no such file or directory

@praveenkumar I did run this command:

sudo find /  kubeconfig

And got this result:

find: ‘kubeconfig’: No such file or directory

@reselbob which means the kubeconfig is not even updated back to host, can you ssh to the VM and get copy the kubeconfig ( /opt/kubeconfig ) to host and then try it?

Did copied kubeconfig as you asked. Here is the result:

oc --kubeconfig=$HOME/.crc/machines/crc/kubeconfig get co 
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.11.13   True        False         False      56s     
config-operator                            4.11.13   True        False         False      24d     
console                                    4.11.13   True        False         False      7d8h    
dns                                        4.11.13   True        False         False      7d8h    
etcd                                       4.11.13   True        False         False      24d     
image-registry                             4.11.13   True        False         False      7d8h    
ingress                                    4.11.13   True        False         False      24d     
kube-apiserver                             4.11.13   True        False         False      24d     
kube-controller-manager                    4.11.13   True        False         False      24d     
kube-scheduler                             4.11.13   True        False         False      24d     
machine-api                                4.11.13   True        False         False      24d     
machine-approver                           4.11.13   True        False         False      24d     
machine-config                             4.11.13   True        False         False      24d     
marketplace                                4.11.13   True        False         False      24d     
network                                    4.11.13   True        False         False      24d     
node-tuning                                4.11.13   True        False         False      7d8h    
openshift-apiserver                        4.11.13   True        False         False      5d4h    
openshift-controller-manager               4.11.13   True        False         False      7d8h    
openshift-samples                          4.11.13   True        False         False      24d     
operator-lifecycle-manager                 4.11.13   True        False         False      24d     
operator-lifecycle-manager-catalog         4.11.13   True        False         False      24d     
operator-lifecycle-manager-packageserver   4.11.13   True        False         False      7d8h

Still can't get into the web site.

oc login works a developer

$ oc login -u developer https://api.crc.testing:6443
Authentication required for https://api.crc.testing:6443 (openshift)
Console URL: https://api.crc.testing:6443/console
Username: developer
Password: 
Login successful.

You don't have any projects. You can try to create a new project, by running

    oc new-project <projectname>

oc login -u kubeadmin https://api.crc.testing:6443
Authentication required for https://api.crc.testing:6443 (openshift)
Console URL: https://api.crc.testing:6443/console
Username: kubeadmin
Password: 
Login failed (401 Unauthorized)
Verify you have provided correct credentials.

Is there something more I can do? Should I run crc stop and start it again?

@reselbob I was intersted to console/auth and both are up I am not sure why you are not able to start console on the browse (login with kubeadmin error is understandable because it was slow process on your system so that is not even updated ) because console is up and running. you might need to check the logs from console pod.

authentication                             4.11.13   True        False         False      56s     
config-operator                            4.11.13   True        False         False      24d     
console                                    4.11.13   True        False         False      7d8h

The web console issue is my mistake. I was confused and tried to login from a browser on the same remote machine I was using to SSH into the machine with the issue. Please forgive my dumbness.

When I go to the machine on which crc is running I can login to the Web UI as developer. I cannot login with when using the kubeadmin username and the password provided by crc start.

Hi @praveenkumar and @cfergeau:

Tell me please, do you have any suggestion about how I can go about making it so I can log in to crc as kubadmin?

@reselbob developer login works because developer password didn't change during the crc start but kubeadmin password does change and in your case it break before even that happen which break the auth for this user.

If you want to use kubeadmin user for the cluster then follow https://github.com/crc-org/crc/wiki/Add-another-user-to-cluster here instead of adding another user you just need to set kubeadmin user password and wait till auth operator is available. Use the kubeconfig file which you copied from the VM https://github.com/crc-org/crc/issues/3442#issuecomment-1352579106 to access the cluster.

Thank you @praveenkumar I am in!

Tell please, will there be a new version of crc available that makes all of this easier?

Again, I really appreciate all the help you've given me.

@reselbob Problem which you had initially because of slow system or high load IO operations which lead failure during crc start and since it is failed to start all other steps which it do as part of code also didn't happen like kubeadmin password change. This is not the crc issue but mostly the hardware limitation. So I am not sure if some changes we need to do as part of crc.

closing it because it is the hardware specific. please reopen if you think otherwise.

crc-org / crc