search

crc-org / crc

CRC is a tool to help you run containers. It manages a local OpenShift 4.x cluster, Microshift or a Podman VM optimized for testing and development purposes
https://crc.dev
Apache License 2.0
1.25k stars 240 forks source link

[BUG] Unable to start CRC: Machine does not exist #3548

Closed bmaupin closed 1 year ago

bmaupin commented 1 year ago

General information

CRC version

CRC version: 2.15.0+cc05160
OpenShift version: 4.12.5
Podman version: 4.3.1

CRC status

DEBU CRC version: 2.15.0+cc05160                  
DEBU OpenShift version: 4.12.5                    
DEBU Podman version: 4.3.1                        
DEBU Running 'crc status'                         
Machine does not exist. Use 'crc start' to create it

CRC config

- consent-telemetry                     : yes

Host Operating System

PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Steps to reproduce

  1. Run crc setup
  2. Run crc start

Expected

I expected crc start to start CRC

Actual

$ crc start
Machine does not exist. Use 'crc start' to create it
$ crc delete -f
Machine does not exist. Use 'crc start' to create it
$ crc cleanup
...
$ crc setup
...
$ crc start --log-level debug
Machine does not exist. Use 'crc start' to create it

Logs

Before gather the logs try following if that fix your issue

$ crc delete -f
$ crc cleanup
$ crc setup
$ crc start --log-level debug

Please consider posting the output of crc start --log-level debug on http://gist.github.com/ and post the link in the issue.

$ crc setup --log-level debug
DEBU CRC version: 2.15.0+cc05160                  
DEBU OpenShift version: 4.12.5                    
DEBU Podman version: 4.3.1                        
DEBU Running 'crc setup'                          
INFO Using bundle path /home/user/.crc/cache/crc_libvirt_4.12.5_amd64.crcbundle 
DEBU Checking if systemd-resolved.service is running 
DEBU Running 'systemctl status systemd-resolved.service' 
DEBU systemd-resolved.service is already running  
INFO Checking if running as non-root              
INFO Checking if running inside WSL2              
INFO Checking if crc-admin-helper executable is cached 
DEBU Running '/home/user/.crc/bin/crc-admin-helper-linux --version' 
DEBU Found crc-admin-helper-linux version 0.0.12  
DEBU crc-admin-helper executable already cached   
INFO Checking if running on a supported CPU architecture 
DEBU GOARCH is amd64 GOOS is linux                
INFO Checking minimum RAM requirements            
DEBU Total memory of system is 33021034496 bytes  
INFO Checking if crc executable symlink exists    
INFO Checking if Virtualization is enabled        
DEBU Checking if the vmx/svm flags are present in /proc/cpuinfo 
DEBU CPU virtualization flags are good            
INFO Checking if KVM is enabled                   
DEBU Checking if /dev/kvm exists                  
DEBU /dev/kvm was found                           
INFO Checking if libvirt is installed             
DEBU Checking if 'virsh' is available             
DEBU 'virsh' was found in /usr/bin/virsh          
DEBU Checking 'virsh capabilities' for libvirtd/qemu availability 
DEBU Running 'virsh --readonly --connect qemu:///system capabilities' 
DEBU Found x86_64 hypervisor with 'hvm' capabilities 
INFO Checking if user is part of libvirt group    
DEBU Checking if current user is part of the libvirt group 
DEBU Current user is already in the libvirt group 
INFO Checking if active user/process is currently part of the libvirt group 
DEBU libvirt group is active for the current user/process 
INFO Checking if libvirt daemon is running        
DEBU Checking if libvirtd service is running      
DEBU Running 'systemctl status virtqemud.socket'  
DEBU Command failed: exit status 4                
DEBU stdout:                                      
DEBU stderr: Unit virtqemud.socket could not be found. 
DEBU virtqemud.socket is neither running nor listening 
DEBU Running 'systemctl status libvirtd.socket'   
DEBU libvirtd.socket is running                   
INFO Checking if a supported libvirt version is installed 
DEBU Checking if libvirt version is >=3.4.0       
DEBU Running 'virsh -v'                           
INFO Checking if crc-driver-libvirt is installed  
DEBU Checking if crc-driver-libvirt is installed  
DEBU Running '/home/user/.crc/bin/crc-driver-libvirt version' 
DEBU Found crc-driver-libvirt version 0.13.5      
DEBU crc-driver-libvirt is already installed      
INFO Checking crc daemon systemd service          
DEBU Checking crc daemon systemd service          
DEBU Checking if crc-daemon.service is running    
DEBU Running 'systemctl --user status crc-daemon.service' 
DEBU Command failed: exit status 3                
DEBU stdout: * crc-daemon.service - CRC daemon
     Loaded: loaded (/home/user/.config/systemd/user/crc-daemon.service; static)
     Active: inactive (dead)
TriggeredBy: * crc-vsock.socket
             * crc-http.socket

Mar 16 11:12:32 hostname systemd[2623]: Starting CRC daemon...
Mar 16 11:12:32 hostname crc[47345]: level=info msg="using socket provided by crc-vsock.socket"
Mar 16 11:12:32 hostname crc[47345]: level=info msg="using socket provided by crc-http.socket"
Mar 16 11:12:32 hostname systemd[2623]: Started CRC daemon.
Mar 16 11:12:32 hostname crc[47345]: @ - - [16/Mar/2023:11:12:32 -0400] "GET /api/status HTTP/1.1" 500 52
Mar 16 11:14:10 hostname systemd[2623]: Stopping CRC daemon...
Mar 16 11:14:11 hostname systemd[2623]: Stopped CRC daemon. 
DEBU stderr:                                      
DEBU crc-daemon.service is neither running nor listening 
DEBU Checking if crc-daemon.service has the expected content 
INFO Checking crc daemon systemd socket units     
DEBU Checking crc daemon systemd socket units     
DEBU Checking if crc-http.socket is running       
DEBU Running 'systemctl --user status crc-http.socket' 
DEBU crc-http.socket is listening                 
DEBU Checking if crc-http.socket has the expected content 
DEBU Checking if crc-vsock.socket is running      
DEBU Running 'systemctl --user status crc-vsock.socket' 
DEBU crc-vsock.socket is listening                
DEBU Checking if crc-vsock.socket has the expected content 
INFO Checking if AppArmor is configured           
INFO Checking if systemd-networkd is running      
DEBU Checking if systemd-networkd.service is running 
DEBU Running 'systemctl status systemd-networkd.service' 
DEBU Command failed: exit status 3                
DEBU stdout: * systemd-networkd.service - Network Configuration
     Loaded: loaded (/lib/systemd/system/systemd-networkd.service; disabled; vendor preset: enabled)
     Active: inactive (dead)
TriggeredBy: * systemd-networkd.socket
       Docs: man:systemd-networkd.service(8) 
DEBU stderr:                                      
DEBU systemd-networkd.service is not running      
INFO Checking if NetworkManager is installed      
DEBU Checking if 'nmcli' is available             
DEBU 'nmcli' was found in /usr/bin/nmcli          
INFO Checking if NetworkManager service is running 
DEBU Checking if NetworkManager.service is running 
DEBU Running 'systemctl status NetworkManager.service' 
DEBU NetworkManager.service is already running    
INFO Checking if dnsmasq configurations file exist for NetworkManager 
INFO Checking if the systemd-resolved service is running 
DEBU Checking if systemd-resolved.service is running 
DEBU Running 'systemctl status systemd-resolved.service' 
DEBU systemd-resolved.service is already running  
INFO Checking if /etc/NetworkManager/dispatcher.d/99-crc.sh exists 
DEBU Checking NetworkManager dispatcher file for crc network 
DEBU Dispatcher file has the expected content     
INFO Checking if libvirt 'crc' network is available 
DEBU Checking if libvirt 'crc' network exists     
DEBU Running 'virsh --connect qemu:///system net-info crc' 
DEBU Checking if libvirt 'crc' definition is up to date 
DEBU Running 'virsh --connect qemu:///system net-dumpxml --inactive crc' 
DEBU libvirt 'crc' network has the expected value 
INFO Checking if libvirt 'crc' network is active  
DEBU Checking if libvirt 'crc' network is active  
DEBU Running 'virsh --connect qemu:///system net-info crc' 
DEBU libvirt 'crc' network is already active      
INFO Checking if CRC bundle is extracted in '$HOME/.crc' 
INFO Checking if /home/user/.crc/cache/crc_libvirt_4.12.5_amd64.crcbundle exists 
DEBU /home/user/.crc/cache/crc_libvirt_4.12.5_amd64.crcbundle exists 
Your system is correctly setup for using CRC. Use 'crc start' to start the instance
bmaupin commented 1 year ago

Things I've tried:

cfergeau commented 1 year ago

This is very odd, I've just tried latest crc on an ubuntu 20.04, and I could not reproduce.

$ crc start --log-level debug
Machine does not exist. Use 'crc start' to create it

does not print any additional logs? Right after enabling its logging system, crc prints its version, it's surprising it errors out before this.

bmaupin commented 1 year ago

@cfergeau No, it doesn't print any additional logs at all. I found it very weird too.

I also forgot to mention, earlier this morning crc setup was failing because I was connected to a VPN (via Cisco AnyConnect):

INFO Starting libvirt 'crc' network               
Failed to start libvirt 'crc' network 
 exit status 1: error: Failed to start network crc
error: internal error: Network is already in use by interface cscotun0

I was also using CRC 2.10 so I upgraded to the latest version.

I'll try rebooting in case maybe the Cisco VPN client has my network in a weird state. But I find it very odd that I'm not getting any other logs

bmaupin commented 1 year ago

I rebooted; still stuck :confused:

Is there any other cleanup I'm missing? I've completely removed ~/.crc and reinstalled and it's still failing.

$ crc setup
INFO Using bundle path /home/user/.crc/cache/crc_libvirt_4.12.5_amd64.crcbundle 
INFO Checking if running as non-root              
INFO Checking if running inside WSL2              
INFO Checking if crc-admin-helper executable is cached 
INFO Checking if running on a supported CPU architecture 
INFO Checking minimum RAM requirements            
INFO Checking if crc executable symlink exists    
INFO Checking if Virtualization is enabled        
INFO Checking if KVM is enabled                   
INFO Checking if libvirt is installed             
INFO Checking if user is part of libvirt group    
INFO Checking if active user/process is currently part of the libvirt group 
INFO Checking if libvirt daemon is running        
INFO Checking if a supported libvirt version is installed 
INFO Checking if crc-driver-libvirt is installed  
INFO Checking crc daemon systemd service          
INFO Checking crc daemon systemd socket units     
INFO Checking if AppArmor is configured           
INFO Checking if vsock is correctly configured    
INFO Checking if CRC bundle is extracted in '$HOME/.crc' 
INFO Checking if /home/user/.crc/cache/crc_libvirt_4.12.5_amd64.crcbundle exists 
Your system is correctly setup for using CRC. Use 'crc start' to start the instance

$ crc start
Machine does not exist. Use 'crc start' to create it

$ crc delete -f
Machine does not exist. Use 'crc start' to create it

$ crc cleanup
INFO Removing vsock configuration                 
INFO Using root access: Removing udev rule in /etc/udev/rules.d/99-crc-vsock.rules 
INFO Using root access: Removing vsock module autoload file /etc/modules-load.d/vhost_vsock.conf 
INFO Removing 'crc' network from libvirt          
INFO Removing /etc/NetworkManager/dispatcher.d/99-crc.sh file 
INFO Cleaning up AppArmor configuration           
INFO Using root access: Updating AppArmor configuration 
INFO Using root access: Changing permissions for /etc/apparmor.d/libvirt/TEMPLATE.qemu to 644  
INFO Removing crc daemon systemd socket units     
INFO Removing crc daemon systemd service          
INFO Removing crc's virtual machine               
INFO Removing crc libvirt storage pool            
INFO Removing hosts file records added by CRC     
INFO Removing pull secret from the keyring        
INFO Removing older logs                          
INFO Removing CRC Machine Instance directory      
INFO Removing crc executable symlink              
Cleanup finished

$ crc setup
INFO Using bundle path /home/user/.crc/cache/crc_libvirt_4.12.5_amd64.crcbundle 
INFO Checking if running as non-root              
INFO Checking if running inside WSL2              
INFO Checking if crc-admin-helper executable is cached 
INFO Checking if running on a supported CPU architecture 
INFO Checking minimum RAM requirements            
INFO Checking if crc executable symlink exists    
INFO Creating symlink for crc executable          
INFO Checking if Virtualization is enabled        
INFO Checking if KVM is enabled                   
INFO Checking if libvirt is installed             
INFO Checking if user is part of libvirt group    
INFO Checking if active user/process is currently part of the libvirt group 
INFO Checking if libvirt daemon is running        
INFO Checking if a supported libvirt version is installed 
INFO Checking if crc-driver-libvirt is installed  
INFO Checking crc daemon systemd service          
INFO Setting up crc daemon systemd service        
INFO Checking crc daemon systemd socket units     
INFO Setting up crc daemon systemd socket units   
INFO Checking if AppArmor is configured           
INFO Updating AppArmor configuration              
INFO Using root access: Updating AppArmor configuration 
INFO Using root access: Changing permissions for /etc/apparmor.d/libvirt/TEMPLATE.qemu to 644  
INFO Checking if vsock is correctly configured    
INFO Setting up vsock support                     
INFO Using root access: Setting CAP_NET_BIND_SERVICE capability for /home/user/bin/crc executable 
INFO Using root access: Creating udev rule for /dev/vsock 
INFO Using root access: Changing permissions for /etc/udev/rules.d/99-crc-vsock.rules to 644  
INFO Using root access: Reloading udev rules database 
INFO Using root access: Loading vhost_vsock kernel module 
INFO Using root access: Creating file /etc/modules-load.d/vhost_vsock.conf 
INFO Using root access: Changing permissions for /etc/modules-load.d/vhost_vsock.conf to 644  
INFO Checking if CRC bundle is extracted in '$HOME/.crc' 
INFO Checking if /home/user/.crc/cache/crc_libvirt_4.12.5_amd64.crcbundle exists 
Your system is correctly setup for using CRC. Use 'crc start' to start the instance

$ crc start
Machine does not exist. Use 'crc start' to create it
praveenkumar commented 1 year ago

I was also using CRC 2.10 so I upgraded to the latest version.

@bmaupin did that 2.10 version work as expected on same host? can you check with sudo virsh list --all ?

bmaupin commented 1 year ago

did that 2.10 version work as expected on same host?

@praveenkumar It worked a while back but when I tried it again last week it didn't work, which is why I upgraded to 2.14.

can you check with sudo virsh list --all ?

$ sudo virsh list --all
 Id   Name        State
----------------------------
 -    minishift   shut off
cfergeau commented 1 year ago

Is this after crc delete? Or right after a failed crc start?

bmaupin commented 1 year ago

@cfergeau I'm sorry, I don't understand the quesiton. Does this help?

$ sudo virsh list --all
 Id   Name        State
----------------------------
 -    minishift   shut off

$ crc delete
Machine does not exist. Use 'crc start' to create it

$ sudo virsh list --all
 Id   Name        State
----------------------------
 -    minishift   shut off

$ crc setup
INFO Using bundle path /home/user/.crc/cache/crc_libvirt_4.12.5_amd64.crcbundle 
INFO Checking if running as non-root              
INFO Checking if running inside WSL2              
INFO Checking if crc-admin-helper executable is cached 
INFO Checking if running on a supported CPU architecture 
INFO Checking minimum RAM requirements            
INFO Checking if crc executable symlink exists    
INFO Checking if Virtualization is enabled        
INFO Checking if KVM is enabled                   
INFO Checking if libvirt is installed             
INFO Checking if user is part of libvirt group    
INFO Checking if active user/process is currently part of the libvirt group 
INFO Checking if libvirt daemon is running        
INFO Checking if a supported libvirt version is installed 
INFO Checking if crc-driver-libvirt is installed  
INFO Checking crc daemon systemd service          
INFO Checking crc daemon systemd socket units     
INFO Checking if AppArmor is configured           
INFO Checking if vsock is correctly configured    
INFO Checking if CRC bundle is extracted in '$HOME/.crc' 
INFO Checking if /home/user/.crc/cache/crc_libvirt_4.12.5_amd64.crcbundle exists 
Your system is correctly setup for using CRC. Use 'crc start' to start the instance

$ crc start
Machine does not exist. Use 'crc start' to create it

$ sudo virsh list --all
 Id   Name        State
----------------------------
 -    minishift   shut off
cfergeau commented 1 year ago

It does not help solve your issue, but makes it clear when you ran virsh, thanks!

bmaupin commented 1 year ago

This got me thinking that maybe minishift was conflicting with CRC, so I ran minishift delete to clean it up:

$ minishift status
Minishift:  Stopped
Profile:    minishift
OpenShift:  Stopped
DiskUsage:  Unknown
CacheUsage: 1.711 GB (used by oc binary, ISO or cached images)

$ ls ~/.minishift/machines/minishift
boot2docker.iso  config.json  id_rsa  id_rsa.pub  minishift.img

$ minishift delete
You are deleting the Minishift VM: 'minishift'. Do you want to continue [y/N]?: y
Removing entries from kubeconfig for cluster: :8443
Deleting the Minishift VM...
Minishift VM deleted.

$ sudo virsh list --all
 Id   Name   State
--------------------

Unfortunately CRC still won't start:

$ crc start
Machine does not exist. Use 'crc start' to create it

$ rm -rf ~/.minishift/

$ crc setup
INFO Using bundle path /home/user/.crc/cache/crc_libvirt_4.12.5_amd64.crcbundle 
INFO Checking if running as non-root              
INFO Checking if running inside WSL2              
INFO Checking if crc-admin-helper executable is cached 
INFO Checking if running on a supported CPU architecture 
INFO Checking minimum RAM requirements            
INFO Checking if crc executable symlink exists    
INFO Checking if Virtualization is enabled        
INFO Checking if KVM is enabled                   
INFO Checking if libvirt is installed             
INFO Checking if user is part of libvirt group    
INFO Checking if active user/process is currently part of the libvirt group 
INFO Checking if libvirt daemon is running        
INFO Checking if a supported libvirt version is installed 
INFO Checking if crc-driver-libvirt is installed  
INFO Checking crc daemon systemd service          
INFO Checking crc daemon systemd socket units     
INFO Checking if AppArmor is configured           
INFO Checking if vsock is correctly configured    
INFO Checking if CRC bundle is extracted in '$HOME/.crc' 
INFO Checking if /home/user/.crc/cache/crc_libvirt_4.12.5_amd64.crcbundle exists 
Your system is correctly setup for using CRC. Use 'crc start' to start the instance

$ crc start
Machine does not exist. Use 'crc start' to create it

(I also ran crc delete -f and crc cleanup again just in case)

bmaupin commented 1 year ago

I'm an idiot :facepalm:

I ran crc start while tailing ~/.crc/crc.log and I noticed this:

time="2023-03-21T14:21:31-04:00" level=debug msg="Running 'crc delete'"
time="2023-03-21T14:21:31-04:00" level=debug msg="Checking file: /home/user/.crc/machines/crc/.crc-exist"

I thought it was weird that it said Running 'crc delete' until I remembered that I'd added this to my ~/.bashrc to work around this other issue: https://github.com/crc-org/crc/issues/3414

crc() {
    if [ "$1" == "start" ]; then
        crc delete -f && crc cleanup && crc setup && command crc "$@" --log-level debug
    else
        command crc "$@"
    fi
}
cfergeau commented 1 year ago

Eh :( Great that you finally figured it out! (and I'm even more glad it's not an obscure crc bug ;)