search

crc-org / crc

CRC is a tool to help you run containers. It manages a local OpenShift 4.x cluster, Microshift or a Podman VM optimized for testing and development purposes
https://crc.dev
Apache License 2.0
1.25k stars 240 forks source link

[BUG] [Fedora-38] `crc setup` freezes at "Checking if libvirt 'crc' network is available" #3719

Closed datho7561 closed 1 year ago

datho7561 commented 1 year ago

General information

CRC version

CRC version: 2.20.0+f3a947
OpenShift version: 4.13.0
Podman version: 4.4.4

CRC status

DEBU CRC version: 2.20.0+f3a947
DEBU OpenShift version: 4.13.0
DEBU Podman version: 4.4.4
DEBU Running 'crc status'

CRC config

- consent-telemetry                     : yes

Host Operating System

NAME="Fedora Linux"
VERSION="38 (Workstation Edition)"
ID=fedora
VERSION_ID=38
VERSION_CODENAME=""
PLATFORM_ID="platform:f38"
PRETTY_NAME="Fedora Linux 38 (Workstation Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:38"
DEFAULT_HOSTNAME="fedora"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f38/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=38
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=38
SUPPORT_END=2024-05-14
VARIANT="Workstation Edition"
VARIANT_ID=workstation

Steps to reproduce

  1. Download CRC binary and put it on PATH
  2. Switch to MicroShift preset
  3. Set RAM to 8000MB (not sure if that's relevant)
  4. crc setup
  5. crc start
  6. I can't access the k8s API so I run crc stop
  7. I run crc delete in preparation to switch to regular OpenShift
  8. It freezes so I kill the process with ^C
  9. Switch to default OpenShift profile
  10. Run crc setup
  11. It freezes at "Checking if libvirt 'crc' network is available"

Expected

crc setup exits within a few seconds

Actual

crc setup freezes.

Logs

I can't run crc, and trying to run crc delete -f also freezes.

jwendell commented 1 year ago

crc setup worked fine for me but crc start hangs in the same stage. F38 as well.

jwendell commented 1 year ago

It looks like this is the issue: https://github.com/systemd/systemd/issues/27953

Workaround: systemctl restart libvirtd.service

datho7561 commented 1 year ago

Seems like libvirtd was disabled for me. After enabling it, crc seems to be working

gbraad commented 1 year ago

A restart might not work, like a fresh install. In that case libvirtd might not have been marked as enable and won't (re)start; Job for libvirtd.service canceled..

Better to do the following in that case:

$ sudo systemctl enable --now libvirtd.service
hippyod commented 1 year ago

I've been fighting this for the past few hours when I tried updating crc to the latest release. I completely uninstalled libvert daemon and qemu-kvm and reinstalled, but that fixed nothing. Finally got the it working after starting and enabling virtnetworkd, although I can't quite say for sure this is what fixed it. I am also pretty sure that libvirtd doesn't actually have anything to do with running the VM, because I can clearly see it isn't running from the Cockpit console?

The following was in the libvirtd logs:

Failed to acquire pid file '/run/libvirt/network/driver.pid': Resource temporarily unavailable

I have also seen similar messages referring to librvirtd and qemu*.pid.

I was able to get libvirtd started after explicitly shutting it down and starting it again with systemctl, but a couple minutes later it shut itself done even as the crc vm kept running just fine. I am pretty sure that libvirtd doesn't actually have anything to do with running the VM, because I can clearly see it isn't running from the Cockpit console.

I wish I understood these subsystems better to be more helpful, but even small data points like this might be helpful.

gbraad commented 1 year ago

Recently they split libvirtd into separate services. This is why the main service might be OK in your case, but the network management service was not running (unavailable). This is specific to Fedora/systemd, and I understand there is an issue and a fix.

We might have to check what we can do to better assist in this case, as we are able to check the state of the services... but not all distros work this way.

cfergeau commented 1 year ago 
$ sudo systemctl enable --now libvirtd.service

I would not recommend this unless you are absolutely sure your setup is not using modular libvirt. If your system is configured for modular libvirt, this could be doing more harm than good.

cfergeau commented 1 year ago

Fwiw, after installing the libvirt package on a freshly installed fedora system, all the services below are enabled and running, and virsh -c qemu:///system ... was functional. libvirtd.service or libvirtd.socket are not part of that set.

$ systemctl list-unit-files --all|grep virt |grep enabled
virtlxcd.service                           enabled         enabled
virtqemud.service                          enabled         enabled
virtxend.service                           enabled         enabled
virtinterfaced.socket                      enabled         enabled
virtlockd.socket                           enabled         disabled
virtlogd.socket                            enabled         disabled
virtlxcd-admin.socket                      enabled         disabled
virtlxcd-ro.socket                         enabled         disabled
virtlxcd.socket                            enabled         disabled
virtnetworkd.socket                        enabled         enabled
virtnodedevd.socket                        enabled         enabled
virtnwfilterd.socket                       enabled         enabled
virtproxyd.socket                          enabled         enabled
virtqemud-admin.socket                     enabled         disabled
virtqemud-ro.socket                        enabled         disabled
virtqemud.socket                           enabled         disabled
virtsecretd.socket                         enabled         enabled
virtstoraged.socket                        enabled         enabled
virtxend-admin.socket                      enabled         disabled
virtxend-ro.socket                         enabled         disabled
virtxend.socket                            enabled         disabled
hippyod commented 1 year ago

Bit of a follow up. So after a few weeks of updating Fedora 38, I'm not having problems starting crc anymore (at least over a few system reboots for testing purposes), but the VM is completely shut down on every reboot now rather than just being put to sleep. I went into Cockpit and saw it wasn't configured for an automatic start and see if that works, but it's a bit out of scope for this issue. I am still on the penultimate release of crc (2.22.1+e8068b4) that I was on when I originally posted.

Unless more knowledgeable people than me (i.e. almost everyone) can figure out whether anymore time needs to spent on this, seems to me this one doesn't need to be an issue anymore?

gbraad commented 1 year ago

autostart will not work, as when the VM is running, we considered CRC started... but in that case the openshift hasnt started as that is something we do as part of the crc start flow

From: hippyod @.> Sent: Thursday, July 20, 2023 3:31:42 AM To: crc-org/crc @.> Cc: Gerard Braad @.>; Comment @.> Subject: Re: [crc-org/crc] [BUG] [Fedora-38] crc setup freezes at "Checking if libvirt 'crc' network is available" (Issue #3719)

Bit of a follow up. So after a few weeks of updating Fedora 38, I'm not having problems starting crc anymore (at least over a few system reboots for testing purposes), but the VM is completely shut down on every reboot now rather than just being put to sleep. I went into Cockpit and saw it wasn't configured for an automatic start and see if that works, but it's a bit out of scope for this issue. I am still on the penultimate release of crc (2.22.1+e8068b4) that I was on when I originally posted.

Unless more knowledgeable people than me (i.e. almost everyone) can figure out whether anymore time needs to spent on this, seems to me this one doesn't need to be an issue anymore?

— Reply to this email directly, view it on GitHubhttps://github.com/crc-org/crc/issues/3719#issuecomment-1642644441, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAAAOZQMUYUXUD3KHB7XEOTXRAYZ5ANCNFSM6AAAAAAZH62RAU. You are receiving this because you commented.Message ID: @.***>

praveenkumar commented 1 year ago

Updated version of systemd now available in f38 which does resolve this issue. I updated my f38 system today and it fixes it. I am closing the issue now.

danpawlik commented 1 year ago

Just FYI, now the CentOS 9 Stream got same issue. Hope they will resolve that soon.