crc-org / crc

CRC is a tool to help you run containers. It manages a local OpenShift 4.x cluster, Microshift or a Podman VM optimized for testing and development purposes
https://crc.dev
Apache License 2.0
1.26k stars 240 forks source link

Getting exception upon CRC setup - error: internal error: Network is already in use by interface crc" #3768

Closed srigitsri closed 1 year ago

srigitsri commented 1 year ago

General information

CRC version

# Put `crc version` output here
CRC version 2.12.0+74565a6
OpenShift version 4.11.18
Podman version 4.2.0  
## CRC status
```bash

# Put `crc status --log-level debug` output here
Machine does not exist. Use 'crc start to create it```

## CRC config
```bash
# Put `crc config view` output here

Host Operating System

# Put the output of `cat /etc/os-release` in case of Linux
# put the output of `sw_vers` in case of Mac
# Put the output of `systeminfo` in case of Windows

Steps to reproduce

just ran 'crc setup' and got the above mentioned exception -- Please note it was working earlier , and suddenly this problem cropped up with no change as such.

Expected

Actual

Logs

Before gather the logs try following if that fix your issue

$ crc delete -f
$ crc cleanup
$ crc setup
$ crc start --log-level debug

Please consider posting the output of crc start --log-level debug on http://gist.github.com/ and post the link in the issue.

anjannath commented 1 year ago

please provide the debug logs from the setup command crc setup --log-level debug

this message error: internal error: Network is already in use by interface crc comes from libvirt when trying to create a libvirt virtual network that uses a network address that clashes with an existing one.

also provide the o/p of sudo virsh net-list --all and brctl show

srigitsri commented 1 year ago

Debug logs

level=debug msg="Checking if crc-http.socket has the expected content"

level=debug msg="Checking if crc-vsock.socket is running"

level=debug msg="Running 'systemctl --user status crc-vsock.socket'"

level=debug msg="crc-vsock.socket is listening"

level=debug msg="Checking if crc-vsock.socket has the expected content"

level=info msg="Checking if systemd-networkd is running"

level=debug msg="Checking if systemd-networkd.service is running"

level=debug msg="Running 'systemctl status systemd-networkd.service'"

level=debug msg="Command failed: exit status 4"

level=debug msg="stdout: "

level=debug msg="stderr: Unit systemd-networkd.service could not be found.\n"

level=debug msg="systemd-networkd.service is not running"

level=info msg="Checking if NetworkManager is installed"

level=debug msg="Checking if 'nmcli' is available"

level=debug msg="'nmcli' was found in /usr/bin/nmcli"

level=info msg="Checking if NetworkManager service is running"

level=debug msg="Checking if NetworkManager.service is running"

level=debug msg="Running 'systemctl status NetworkManager.service'"

level=debug msg="NetworkManager.service is already running"

level=info msg="Checking if /etc/NetworkManager/conf.d/crc-nm-dnsmasq.conf exists"

level=warning msg="Skipping above check..."

level=info msg="Checking if /etc/NetworkManager/dnsmasq.d/crc.conf exists"

level=warning msg="Skipping above check..."

level=info msg="Checking if libvirt 'crc' network is available"

level=debug msg="Checking if libvirt 'crc' network exists"

level=debug msg="Running 'virsh --connect qemu:///system net-info crc'"

level=debug msg="Checking if libvirt 'crc' definition is up to date"

level=debug msg="Running 'virsh --connect qemu:///system net-dumpxml --inactive crc'"

level=debug msg="libvirt 'crc' network has the expected value"

level=info msg="Checking if libvirt 'crc' network is active"

level=debug msg="Checking if libvirt 'crc' network is active"

level=debug msg="Running 'virsh --connect qemu:///system net-info crc'"

level=debug msg="Libvirt crc network is not active"

level=info msg="Starting libvirt 'crc' network"

level=debug msg="Starting libvirt 'crc' network"

level=debug msg="Running 'virsh --connect qemu:///system net-start crc'"

level=debug msg="Command failed: exit status 1"

level=debug msg="stdout: \n"

level=debug msg="stderr: error: Failed to start network crc\nerror: internal error: Network is already in use by interface crc\n"

Failed to start libvirt 'crc' network

exit status 1: error: Failed to start network crc

error: internal error: Network is already in use by interface crc

sudo virsh net-list --all

Name      State      Autostart   Persistent

----------------------------------------------

crc       inactive   yes         yes

default   active     yes         yes
praveenkumar commented 1 year ago

@srigitsri looks like you already have crc network so might try crc cleanup which remove that network.

srigitsri commented 1 year ago
INFO Removing vsock configuration
INFO Removing 'crc' network from libvirt
INFO Removing /etc/NetworkManager/dnsmasq.d/crc.conf file
INFO Removing /etc/NetworkManager/conf.d/crc-nm-dnsmasq.conf file
INFO Removing crc daemon systemd socket units
INFO Removing crc daemon systemd service
INFO Removing crc's virtual machine
INFO Removing crc libvirt storage pool
INFO Removing hosts file records added by CRC
INFO Removing pull secret from the keyring
INFO Removing older logs
INFO Removing CRC Machine Instance directory
INFO Removing crc executable symlink
Failed to destroy libvirt 'crc' network

And when I execute crc setup , then get the same exception

srigitsri commented 1 year ago

Is there a specific command to delete this network

srigitsri commented 1 year ago

Since cleanup is not removing existing network, any alternative

praveenkumar commented 1 year ago

@srigitsri in that case you can try manually removing the network using virsh command, but I want to know why it is not removing can you share the output of crc cleanup --log-level debug ?

$ sudo virsh net-destroy crc
$ sudo virsh net-undefine crc
srigitsri commented 1 year ago

crc cleanup –log-level debug

crc cleanup --log-level debug

DEBU CRC version: 2.12.0+74565a6

DEBU OpenShift version: 4.11.18

DEBU Podman version: 4.2.0

DEBU Running 'crc cleanup'

DEBU Checking if systemd-resolved.service is running

DEBU Running 'systemctl status systemd-resolved.service'

DEBU Command failed: exit status 3

DEBU stdout: * systemd-resolved.service - Network Name Resolution

Loaded: loaded (/usr/lib/systemd/system/systemd-resolved.service; disabled; v endor preset: disabled)

Active: inactive (dead)

 Docs: man:systemd-resolved.service(8)

       https://www.freedesktop.org/wiki/Software/systemd/resolved

       https://www.freedesktop.org/wiki/Software/systemd/writing-network-con                                                                                        figuration-managers

       https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-cl                                                                                        ients

DEBU stderr:

INFO Removing vsock configuration

INFO Removing 'crc' network from libvirt

DEBU Removing libvirt 'crc' network

DEBU Running 'virsh --connect qemu:///system net-info crc'

DEBU Running 'virsh --connect qemu:///system net-destroy crc'

DEBU Command failed: exit status 1

DEBU stdout:

DEBU stderr: error: Failed to destroy network crc

error: Requested operation is not valid: network 'crc' is not active

DEBU exit status 1 : error: Failed to destroy network crc

error: Requested operation is not valid: network 'crc' is not active

DEBU Failed to destroy libvirt 'crc' network

INFO Removing /etc/NetworkManager/dnsmasq.d/crc.conf file

DEBU Checking if 'nmcli' is available

DEBU 'nmcli' was found in /usr/bin/nmcli

INFO Removing /etc/NetworkManager/conf.d/crc-nm-dnsmasq.conf file

DEBU Checking if 'nmcli' is available

DEBU 'nmcli' was found in /usr/bin/nmcli

INFO Removing crc daemon systemd socket units

DEBU Removing crc daemon systemd socket units

DEBU Running 'systemctl --user stop crc-http.socket'

DEBU Running 'systemctl --user stop crc-vsock.socket'

INFO Removing crc daemon systemd service

DEBU Removing crc daemon systemd service

DEBU Running 'systemctl --user stop crc-daemon.service'

INFO Removing crc's virtual machine

DEBU Running 'virsh --connect qemu:///system domstate crc'

DEBU Command failed: exit status 1

DEBU stdout:

DEBU stderr: error: failed to get domain 'crc'

INFO Removing crc libvirt storage pool

DEBU Running 'virsh --connect qemu:///system pool-info crc'

DEBU Command failed: exit status 1

DEBU stdout:

DEBU stderr: error: failed to get pool 'crc'

error: Storage pool not found: no storage pool with matching name 'crc'

DEBU exit status 1 : error: failed to get pool 'crc'

error: Storage pool not found: no storage pool with matching name 'crc'

INFO Removing hosts file records added by CRC

DEBU Running '/ocpuser/ocpuser/.crc/bin/crc-admin-helper-linux clean .crc.testin g .apps-crc.testing'

INFO Removing pull secret from the keyring

INFO Removing older logs

INFO Removing CRC Machine Instance directory

DEBU Deleting machines directory

INFO Removing crc executable symlink

Failed to destroy libvirt 'crc' network

srigitsri commented 1 year ago

After running those teo virsh command, still the same exception

praveenkumar commented 1 year ago

@srigitsri what is the output of following command?

$ sudo virsh net-list
$ sudo virsh net-undefine crc
$ sudo virsh net-list
srigitsri commented 1 year ago

sudo virsh net-list

Name State Autostart Persistent


default active yes yes

sudo virsh net-undefine crc

error: failed to get network 'crc'

error: Network not found: no network with matching name 'crc'

sudo virsh net-list

Name State Autostart Persistent


default active yes yes

praveenkumar commented 1 year ago

@srigitsri which means there is no crc network so now crc setup and crc start should work as expected.

srigitsri commented 1 year ago

Even after destroy, undefine command, it continues to give exception, is it possible to connect over teams/webex to troubleshoot

praveenkumar commented 1 year ago

@srigitsri can you share your mail ID, will try to send you the invite?

praveenkumar commented 1 year ago

@srigitsri looks like github hide the email ID if I put foo@gmail.com but I should use foo AT gmail DOT com

praveenkumar commented 1 year ago

During debugging session looks like the RHEL-8 image is used by @srigitsri is provided by IT and had some wired modification which doesn't allow to delete the crc interface from ifconfig -a even when from libvirt side we destroyed and undefine the crc network. So I need to manually delete the crc interface using nmcli command

$ sudo nmcli -f bridge con delete crc

After that crc setup works as expected and also crc start . Closing this issue.

BensFat commented 6 months ago

check the name of the interface "crc" if exist with : #ifconfig The remove it : #sudo ip link delete crc

crc setup