crc-org / crc

CRC is a tool to help you run containers. It manages a local OpenShift 4.x cluster, Microshift or a Podman VM optimized for testing and development purposes
https://crc.dev
Apache License 2.0
1.25k stars 240 forks source link

[BUG] Failed to connect socket to /var/run/libvirt/virtnetworkd-sock Libvirt network crc not found #2839

Closed wugutech closed 2 years ago

wugutech commented 2 years ago

General information

CRC version

# Put `crc version` output here
CodeReady Containers version: 1.35.0+751824a9
OpenShift version: 4.9.5 (embedded in executable)

CRC setup

# Put `crc setup --log-level debug` output here
DEBU CodeReady Containers version: 1.35.0+751824a9
DEBU OpenShift version: 4.9.5 (embedded in executable)
DEBU Running 'crc setup'
DEBU Checking if systemd-resolved.service is running
DEBU Running 'systemctl status systemd-resolved.service'
DEBU systemd-resolved.service is already running
INFO Checking if running as non-root
INFO Checking if running inside WSL2
INFO Checking if crc-admin-helper executable is cached
DEBU Running '/home/crc/.crc/bin/crc-admin-helper-linux --version'
DEBU Found crc-admin-helper-linux version v0.0.7
DEBU crc-admin-helper executable already cached
INFO Checking for obsolete admin-helper executable
DEBU Checking if an older admin-helper executable is installed
DEBU No older admin-helper executable found
INFO Checking if running on a supported CPU architecture
INFO Checking minimum RAM requirements
DEBU Total memory of system is 25038995456 bytes
INFO Checking if crc executable symlink exists
INFO Checking if Virtualization is enabled
DEBU Checking if the vmx/svm flags are present in /proc/cpuinfo
DEBU CPU virtualization flags are good
INFO Checking if KVM is enabled
DEBU Checking if /dev/kvm exists
DEBU /dev/kvm was found
INFO Checking if libvirt is installed
DEBU Checking if 'virsh' is available
DEBU 'virsh' was found in /usr/bin/virsh
DEBU Checking 'virsh capabilities' for libvirtd/qemu availability
DEBU Running 'virsh --readonly --connect qemu:///system capabilities'
DEBU Found x86_64 hypervisor with 'hvm' capabilities
INFO Checking if user is part of libvirt group
DEBU Checking if current user is part of the libvirt group
DEBU Current user is already in the libvirt group
INFO Checking if active user/process is currently part of the libvirt group
INFO Checking if libvirt daemon is running
DEBU Checking if libvirtd service is running
DEBU Running 'systemctl status virtqemud.socket'
DEBU virtqemud.socket is running
INFO Checking if a supported libvirt version is installed
DEBU Checking if libvirt version is >=3.4.0
DEBU Running 'virsh -v'
INFO Checking if crc-driver-libvirt is installed
DEBU Checking if crc-driver-libvirt is installed
DEBU Running '/home/crc/.crc/bin/crc-driver-libvirt version'
DEBU Found crc-driver-libvirt version 0.13.1
DEBU crc-driver-libvirt is already installed
INFO Checking crc daemon systemd service
DEBU Checking crc daemon systemd service
DEBU Checking if crc-daemon.service is running
DEBU Running 'systemctl --user status crc-daemon.service'
DEBU Command failed: exit status 3
DEBU stdout: * crc-daemon.service - CodeReady Containers daemon
   Loaded: loaded (/home/crc/.config/systemd/user/crc-daemon.service; static; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2021-11-16 16:04:45 WIB; 12min ago
  Process: 2031 ExecStart=/home/crc/.crc/bin/crc daemon (code=exited, status=2)
 Main PID: 2031 (code=exited, status=2)

Nov 16 16:04:45 openshift.crc.testing crc[2031]: level=info msg="Checking if NetworkManager is installed"
Nov 16 16:04:45 openshift.crc.testing crc[2031]: level=info msg="Checking if NetworkManager service is running"
Nov 16 16:04:45 openshift.crc.testing crc[2031]: level=info msg="Checking if dnsmasq configurations file exist for NetworkManager"
Nov 16 16:04:45 openshift.crc.testing crc[2031]: level=info msg="Checking if the systemd-resolved service is running"
Nov 16 16:04:45 openshift.crc.testing crc[2031]: level=info msg="Checking if /etc/NetworkManager/dispatcher.d/99-crc.sh exists"
Nov 16 16:04:45 openshift.crc.testing crc[2031]: level=info msg="Checking if libvirt 'crc' network is available"
Nov 16 16:04:45 openshift.crc.testing crc[2031]: Libvirt network crc not found
Nov 16 16:04:45 openshift.crc.testing systemd[1806]: crc-daemon.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Nov 16 16:04:45 openshift.crc.testing systemd[1806]: crc-daemon.service: Failed with result 'exit-code'.
Nov 16 16:04:45 openshift.crc.testing systemd[1806]: Failed to start CodeReady Containers daemon.
DEBU stderr:
DEBU Could not get crc-daemon.service  status: Executing systemctl action failed: * crc-daemon.service - CodeReady Containers daemon
   Loaded: loaded (/home/crc/.config/systemd/user/crc-daemon.service; static; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2021-11-16 16:04:45 WIB; 12min ago
  Process: 2031 ExecStart=/home/crc/.crc/bin/crc daemon (code=exited, status=2)
 Main PID: 2031 (code=exited, status=2)

Nov 16 16:04:45 openshift.crc.testing crc[2031]: level=info msg="Checking if NetworkManager is installed"
Nov 16 16:04:45 openshift.crc.testing crc[2031]: level=info msg="Checking if NetworkManager service is running"
Nov 16 16:04:45 openshift.crc.testing crc[2031]: level=info msg="Checking if dnsmasq configurations file exist for NetworkManager"
Nov 16 16:04:45 openshift.crc.testing crc[2031]: level=info msg="Checking if the systemd-resolved service is running"
Nov 16 16:04:45 openshift.crc.testing crc[2031]: level=info msg="Checking if /etc/NetworkManager/dispatcher.d/99-crc.sh exists"
Nov 16 16:04:45 openshift.crc.testing crc[2031]: level=info msg="Checking if libvirt 'crc' network is available"
Nov 16 16:04:45 openshift.crc.testing crc[2031]: Libvirt network crc not found
Nov 16 16:04:45 openshift.crc.testing systemd[1806]: crc-daemon.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Nov 16 16:04:45 openshift.crc.testing systemd[1806]: crc-daemon.service: Failed with result 'exit-code'.
Nov 16 16:04:45 openshift.crc.testing systemd[1806]: Failed to start CodeReady Containers daemon.
 exit status 3:
DEBU Checking if crc-daemon.service has the expected content
INFO Checking crc daemon systemd socket units
DEBU Checking crc daemon systemd socket units
DEBU Checking if crc-http.socket is running
DEBU Running 'systemctl --user status crc-http.socket'
DEBU crc-http.socket is listening
DEBU Checking if crc-http.socket has the expected content
DEBU Checking if crc-vsock.socket is running
DEBU Running 'systemctl --user status crc-vsock.socket'
DEBU crc-vsock.socket is listening
DEBU Checking if crc-vsock.socket has the expected content
INFO Checking if systemd-networkd is running
DEBU Checking if systemd-networkd.service is running
DEBU Running 'systemctl status systemd-networkd.service'
DEBU Command failed: exit status 4
DEBU stdout:
DEBU stderr: Unit systemd-networkd.service could not be found.
DEBU systemd-networkd.service is not running
INFO Checking if NetworkManager is installed
DEBU Checking if 'nmcli' is available
DEBU 'nmcli' was found in /usr/bin/nmcli
INFO Checking if NetworkManager service is running
DEBU Checking if NetworkManager.service is running
DEBU Running 'systemctl status NetworkManager.service'
DEBU NetworkManager.service is already running
INFO Checking if dnsmasq configurations file exist for NetworkManager
INFO Checking if the systemd-resolved service is running
DEBU Checking if systemd-resolved.service is running
DEBU Running 'systemctl status systemd-resolved.service'
DEBU systemd-resolved.service is already running
INFO Checking if /etc/NetworkManager/dispatcher.d/99-crc.sh exists
DEBU Checking NetworkManager dispatcher file for crc network
DEBU Dispatcher file has the expected content
INFO Checking if libvirt 'crc' network is available
DEBU Checking if libvirt 'crc' network exists
DEBU Running 'virsh --connect qemu:///system net-info crc'
DEBU Command failed: exit status 1
DEBU stdout:
DEBU stderr: error: failed to get network 'crc'
error: Failed to connect socket to '/var/run/libvirt/virtnetworkd-sock': No such file or directory
DEBU Libvirt network crc not found
INFO Setting up libvirt 'crc' network
DEBU Creating libvirt 'crc' network
DEBU Running 'virsh --connect qemu:///system net-destroy crc'
DEBU Command failed: exit status 1
DEBU stdout:
DEBU stderr: error: failed to get network 'crc'
error: Failed to connect socket to '/var/run/libvirt/virtnetworkd-sock': No such file or directory
DEBU Running 'virsh --connect qemu:///system net-undefine crc'
DEBU Command failed: exit status 1
DEBU stdout:
DEBU stderr: error: failed to get network 'crc'
error: Failed to connect socket to '/var/run/libvirt/virtnetworkd-sock': No such file or directory
DEBU exit status 1 : error: Failed to define network from /dev/stdin
error: Failed to connect socket to '/var/run/libvirt/virtnetworkd-sock': No such file or directory
Failed to create libvirt 'crc' network: exit status 1 - error: Failed to define network from /dev/stdin
error: Failed to connect socket to '/var/run/libvirt/virtnetworkd-sock': No such file or directory

CRC config

# Put `crc config view` output here
- consent-telemetry                     : no

Host Operating System

# Put the output of `cat /etc/os-release` in case of Linux
CentOS Linux release 8.4.2105

Steps to reproduce

Expected

Actual

Logs

Before gather the logs try following if that fix your issue

$ crc delete -f
$ crc cleanup
$ crc setup
$ crc start --log-level debug
DEBU CodeReady Containers version: 1.35.0+751824a9
DEBU OpenShift version: 4.9.5 (embedded in executable)
DEBU Running 'crc start'
DEBU Total memory of system is 25038995456 bytes
DEBU No new version available. The latest version is 1.35.0
DEBU Checking if systemd-resolved.service is running
DEBU Running 'systemctl status systemd-resolved.service'
DEBU systemd-resolved.service is already running
INFO Checking if running as non-root
INFO Checking if running inside WSL2
INFO Checking if crc-admin-helper executable is cached
DEBU Running '/home/crc/.crc/bin/crc-admin-helper-linux --version'
DEBU Found crc-admin-helper-linux version v0.0.7
DEBU crc-admin-helper executable already cached
INFO Checking for obsolete admin-helper executable
DEBU Checking if an older admin-helper executable is installed
DEBU No older admin-helper executable found
INFO Checking if running on a supported CPU architecture
INFO Checking minimum RAM requirements
DEBU Total memory of system is 25038995456 bytes
INFO Checking if crc executable symlink exists
INFO Checking if Virtualization is enabled
DEBU Checking if the vmx/svm flags are present in /proc/cpuinfo
DEBU CPU virtualization flags are good
INFO Checking if KVM is enabled
DEBU Checking if /dev/kvm exists
DEBU /dev/kvm was found
INFO Checking if libvirt is installed
DEBU Checking if 'virsh' is available
DEBU 'virsh' was found in /usr/bin/virsh
DEBU Checking 'virsh capabilities' for libvirtd/qemu availability
DEBU Running 'virsh --readonly --connect qemu:///system capabilities'
DEBU Found x86_64 hypervisor with 'hvm' capabilities
INFO Checking if user is part of libvirt group
DEBU Checking if current user is part of the libvirt group
DEBU Current user is already in the libvirt group
INFO Checking if active user/process is currently part of the libvirt group
INFO Checking if libvirt daemon is running
DEBU Checking if libvirtd service is running
DEBU Running 'systemctl status virtqemud.socket'
DEBU virtqemud.socket is running
INFO Checking if a supported libvirt version is installed
DEBU Checking if libvirt version is >=3.4.0
DEBU Running 'virsh -v'
INFO Checking if crc-driver-libvirt is installed
DEBU Checking if crc-driver-libvirt is installed
DEBU Running '/home/crc/.crc/bin/crc-driver-libvirt version'
DEBU Found crc-driver-libvirt version 0.13.1
DEBU crc-driver-libvirt is already installed
INFO Checking crc daemon systemd socket units
DEBU Checking crc daemon systemd socket units
DEBU Checking if crc-http.socket is running
DEBU Running 'systemctl --user status crc-http.socket'
DEBU crc-http.socket is listening
DEBU Checking if crc-http.socket has the expected content
DEBU Checking if crc-vsock.socket is running
DEBU Running 'systemctl --user status crc-vsock.socket'
DEBU crc-vsock.socket is listening
DEBU Checking if crc-vsock.socket has the expected content
INFO Checking if systemd-networkd is running
DEBU Checking if systemd-networkd.service is running
DEBU Running 'systemctl status systemd-networkd.service'
DEBU Command failed: exit status 4
DEBU stdout:
DEBU stderr: Unit systemd-networkd.service could not be found.
DEBU systemd-networkd.service is not running
INFO Checking if NetworkManager is installed
DEBU Checking if 'nmcli' is available
DEBU 'nmcli' was found in /usr/bin/nmcli
INFO Checking if NetworkManager service is running
DEBU Checking if NetworkManager.service is running
DEBU Running 'systemctl status NetworkManager.service'
DEBU NetworkManager.service is already running
INFO Checking if dnsmasq configurations file exist for NetworkManager
INFO Checking if the systemd-resolved service is running
DEBU Checking if systemd-resolved.service is running
DEBU Running 'systemctl status systemd-resolved.service'
DEBU systemd-resolved.service is already running
INFO Checking if /etc/NetworkManager/dispatcher.d/99-crc.sh exists
DEBU Checking NetworkManager dispatcher file for crc network
DEBU Dispatcher file has the expected content
INFO Checking if libvirt 'crc' network is available
DEBU Checking if libvirt 'crc' network exists
DEBU Running 'virsh --connect qemu:///system net-info crc'
DEBU Command failed: exit status 1
DEBU stdout:
DEBU stderr: error: failed to get network 'crc'
error: Failed to connect socket to '/var/run/libvirt/virtnetworkd-sock': No such file or directory
DEBU Libvirt network crc not found
WARN Preflight checks failed during `crc start`, please try to run `crc setup` first in case you haven't done so yet
Libvirt network crc not found

Please consider posting the output of crc start --log-level debug on http://gist.github.com/ and post the link in the issue.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

fOO223Fr commented 2 years ago

Hi there, I faced this issue recently and after some reading below is my understanding and the solution to the problem.

My Understanding:

libvirtd became modular from being monolithic.

each driver in libvirt (both hypervisor drivers & secondary drivers) will be serviced by its own dedicated daemon [1]

which makes virtnetworkd the driver modular too.

Solution:

I switched to modular daemons but following steps 1, 2, 3, 4 from libvirt.org [2]

Summary of commands:

$ systemctl stop libvirtd.service
$ systemctl stop libvirtd{,-ro,-admin,-tcp,-tls}.socket
$ systemctl disable libvirtd.service
$ systemctl disable libvirtd{,-ro,-admin,-tcp,-tls}.socket
$ for drv in qemu interface network nodedev nwfilter secret storage
  do
    systemctl unmask virt${drv}d.service
    systemctl unmask virt${drv}d{,-ro,-admin}.socket
    systemctl enable virt${drv}d.service
    systemctl enable virt${drv}d{,-ro,-admin}.socket
  done
  $ for drv in qemu network nodedev nwfilter secret storage
  do
    systemctl start virt${drv}d{,-ro,-admin}.socket
  done

Summary of commands to check if virtnetworkd is up:

# systemctl is-active virtnetworkd.socket
active
# systemctl is-active virtnetworkd.service
active

Links:

  1. https://www.berrange.com/posts/2020/02/04/libvirt-split-of-the-monolithic-libvirtd-daemon/
  2. https://libvirt.org/daemons.html#switching-to-modular-daemons
stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

cfergeau commented 2 years ago

Summary of commands:

$ systemctl stop libvirtd.service $ systemctl stop libvirtd{,-ro,-admin,-tcp,-tls}.socket $ systemctl disable libvirtd.service $ systemctl disable libvirtd{,-ro,-admin,-tcp,-tls}.socket $ for drv in qemu interface network nodedev nwfilter secret storage do systemctl unmask virt${drv}d.service systemctl unmask virt${drv}d{,-ro,-admin}.socket systemctl enable virt${drv}d.service systemctl enable virt${drv}d{,-ro,-admin}.socket done $ for drv in qemu network nodedev nwfilter secret storage do systemctl start virt${drv}d{,-ro,-admin}.socket done

These steps should ensure that '/var/run/libvirt/virtnetworkd-sock exists, (systemctl start virtnetworkd.socket), so error: Failed to connect socket to '/var/run/libvirt/virtnetworkd-sock': No such file or directory is unexpected. Unless the system from the bug reporter was not configured correctly for libvirt split daemons.

cfergeau commented 2 years ago

Summary of commands:

$ systemctl stop libvirtd.service $ systemctl stop libvirtd{,-ro,-admin,-tcp,-tls}.socket $ systemctl disable libvirtd.service $ systemctl disable libvirtd{,-ro,-admin,-tcp,-tls}.socket $ for drv in qemu interface network nodedev nwfilter secret storage do systemctl unmask virt${drv}d.service systemctl unmask virt${drv}d{,-ro,-admin}.socket systemctl enable virt${drv}d.service systemctl enable virt${drv}d{,-ro,-admin}.socket done $ for drv in qemu network nodedev nwfilter secret storage do systemctl start virt${drv}d{,-ro,-admin}.socket done

These steps should ensure that '/var/run/libvirt/virtnetworkd-sock exists, (systemctl start virtnetworkd.socket), so error: Failed to connect socket to '/var/run/libvirt/virtnetworkd-sock': No such file or directory is unexpected. Unless the system from the bug reporter was not configured correctly for libvirt split daemons.

tabowling commented 8 months ago

Red Hat documents this really well here to ensure all of the various packages are installed and services enabled. It also provides a good description of qemu/libvirt/kvm. Chapter 2. Enabling virtualization