Open indresh28 opened 2 years ago
The VM fails to start because of
Message='unsupported configuration: unable to open vhost-vsock device'
Maybe something special needs to be done for this device to be available on ubuntu? Hopefully its kernel is not too old for vhost-vsock support?
Thanks @cfergeau for the the response, this is my kernel details: Linux 5.4.0-104-generic #118-Ubuntu SMP Wed Mar 2 19:02:41 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux. Could you help me to understand what configuration changes needed.?
I am facing the same issue. what is the solution?
I ran a quick test on an up to date ubuntu 20.04, downloaded latest crc, ran crc setup
, and I did not get any issues related to vhost-vsock
, so more details about crc setup
, crc config view
, ... would be useful in making progress.
test-server:/etc$ crc setup
INFO Using bundle path /home/securify/.crc/cache/crc_libvirt_4.10.14_amd64.crcbundle
INFO Checking if running as non-root
INFO Checking if running inside WSL2
INFO Checking if crc-admin-helper executable is cached
INFO Checking for obsolete admin-helper executable
INFO Checking if running on a supported CPU architecture
INFO Checking minimum RAM requirements
INFO Checking if crc executable symlink exists
INFO Checking if Virtualization is enabled
INFO Checking if KVM is enabled
INFO Checking if libvirt is installed
INFO Checking if user is part of libvirt group
INFO Checking if active user/process is currently part of the libvirt group
INFO Checking if libvirt daemon is running
INFO Checking if a supported libvirt version is installed
INFO Checking if crc-driver-libvirt is installed
INFO Checking crc daemon systemd service
INFO Setting up crc daemon systemd service
INFO Checking crc daemon systemd socket units
INFO Checking if AppArmor is configured
INFO Checking if vsock is correctly configured
INFO Checking if CRC bundle is extracted in '$HOME/.crc'
INFO Checking if /home/securify/.crc/cache/crc_libvirt_4.10.14_amd64.crcbundle exists
Your system is correctly setup for using CRC. Use 'crc start' to start the instance
test-server:/etc$ crc start
INFO Checking if running as non-root
INFO Checking if running inside WSL2
INFO Checking if crc-admin-helper executable is cached
INFO Checking for obsolete admin-helper executable
INFO Checking if running on a supported CPU architecture
INFO Checking minimum RAM requirements
INFO Checking if crc executable symlink exists
INFO Checking if Virtualization is enabled
INFO Checking if KVM is enabled
INFO Checking if libvirt is installed
INFO Checking if user is part of libvirt group
INFO Checking if active user/process is currently part of the libvirt group
INFO Checking if libvirt daemon is running
INFO Checking if a supported libvirt version is installed
INFO Checking if crc-driver-libvirt is installed
INFO Checking crc daemon systemd socket units
INFO Checking if AppArmor is configured
INFO Checking if vsock is correctly configured
INFO Loading bundle: crc_libvirt_4.10.14_amd64...
INFO Starting CRC VM for OpenShift 4.10.14...
Error starting machine: Error in driver during machine start: virError(Code=67, Domain=10, Message='unsupported configuration: unable to open vhost-vsock device')
test-server:/etc$ crc config view
- consent-telemetry : no
- network-mode : user
test-server:/etc$
test-server:/etc$ systemctl status libvirtd
● libvirtd.service - Virtualization daemon
Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2022-06-28 11:39:35 +03; 5h 0min ago
TriggeredBy: ● libvirtd-admin.socket
● libvirtd-ro.socket
● libvirtd.socket
Docs: man:libvirtd(8)
https://libvirt.org
Main PID: 60891 (libvirtd)
Tasks: 17 (limit: 32768)
Memory: 16.1M
CGroup: /system.slice/libvirtd.service
└─60891 /usr/sbin/libvirtd
Jun 28 16:38:49 test-server libvirtd[60891]: End of file while reading data: Input/output error
Jun 28 16:38:50 test-server libvirtd[60891]: unsupported configuration: unable to open vhost-vsock device
Jun 28 16:38:50 test-server libvirtd[60891]: End of file while reading data: Input/output error
test-server:/etc$
test-server:/etc$ uname -a
Linux MesutGL-1-82 5.4.0-121-generic #137-Ubuntu SMP Wed Jun 15 13:33:07 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
test-server:/etc$ more /etc/os-release
NAME="Ubuntu"
VERSION="20.04.4 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.4 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
This was reported a while back, and was related to nested virtualization, I don't know if this is also your case
https://github.com/code-ready/crc/issues/2130
I'd check /dev/vhost*
, lsmod |grep vsock
, libvirt logs for the VM in /var/log/libvirt/qemu/crc.log
, ... to try to understand why your system is having issues with vsock. sudo virsh edit crc
+ sudo virsh start crc
can be a quick way to experiment with changes to see if they help.
You can also use system networking instead of user, in which case I don't htink crc will try to use vsock.
test-server:/etc$ virsh start crc
error: Failed to start domain crc
error: unsupported configuration: unable to open vhost-vsock device
test-server:/etc$ ls -al /dev/vhost*
crw------- 1 root root 10, 238 Jun 28 10:16 /dev/vhost-net
crw------- 1 root root 10, 241 Jun 28 10:16 /dev/vhost-vsock
test-server:/etc$ lsmod |grep vsock
vmw_vsock_virtio_transport_common 32768 0
vmw_vsock_vmci_transport 32768 2
vsock 36864 4 vmw_vsock_virtio_transport_common,vmw_vsock_vmci_transport
vmw_vmci 69632 2 vmw_balloon,vmw_vsock_vmci_transport
test-server:/etc$ sudo tail -n 20 /var/log/libvirt/qemu/crc.log
2022-06-28 06:29:30.446+0000: shutting down, reason=failed
2022-06-28 06:29:47.861+0000: shutting down, reason=failed
2022-06-28 06:31:20.326+0000: shutting down, reason=failed
2022-06-28 06:59:27.129+0000: shutting down, reason=failed
2022-06-28 07:18:19.214+0000: shutting down, reason=failed
2022-06-28 07:44:25.074+0000: shutting down, reason=failed
2022-06-28 07:50:40.867+0000: shutting down, reason=failed
2022-06-28 08:02:11.203+0000: shutting down, reason=failed
test-server:/etc$ crc cleanup
INFO Removing vsock configuration
INFO Removing 'crc' network from libvirt
INFO Removing /etc/NetworkManager/dispatcher.d/99-crc.sh file
INFO Cleaning up AppArmor configuration
INFO Removing crc daemon systemd socket units
INFO Removing crc daemon systemd service
INFO Removing crc's virtual machine
INFO Removing crc libvirt storage pool
INFO Removing hosts file records added by CRC
INFO Removing pull secret from the keyring
INFO Removing older logs
INFO Removing CRC Machine Instance directory
INFO Removing crc executable symlink
unexpected AppArmor template file /etc/apparmor.d/libvirt/TEMPLATE.qemu, cannot configure it automatically
exit status 1
test-server:/etc$
$ crc config set network-mode system
Network mode changed. Please run `crc cleanup` and `crc setup`.
test-server:/etc$ crc cleanup
INFO Removing vsock configuration
INFO Removing 'crc' network from libvirt
INFO Removing /etc/NetworkManager/dispatcher.d/99-crc.sh file
INFO Cleaning up AppArmor configuration
INFO Removing crc daemon systemd socket units
INFO Removing crc daemon systemd service
INFO Removing crc's virtual machine
INFO Removing crc libvirt storage pool
INFO Removing hosts file records added by CRC
INFO Removing pull secret from the keyring
INFO Removing older logs
INFO Removing CRC Machine Instance directory
INFO Removing crc executable symlink
unexpected AppArmor template file /etc/apparmor.d/libvirt/TEMPLATE.qemu, cannot configure it automatically
exit status 1
test-server:/etc$ crc setup
INFO Using bundle path /home/securify/.crc/cache/crc_libvirt_4.10.14_amd64.crcbundle
INFO Checking if running as non-root
INFO Checking if running inside WSL2
INFO Checking if crc-admin-helper executable is cached
INFO Checking for obsolete admin-helper executable
INFO Checking if running on a supported CPU architecture
INFO Checking minimum RAM requirements
INFO Checking if crc executable symlink exists
INFO Creating symlink for crc executable
INFO Checking if Virtualization is enabled
INFO Checking if KVM is enabled
INFO Checking if libvirt is installed
INFO Checking if user is part of libvirt group
INFO Checking if active user/process is currently part of the libvirt group
INFO Checking if libvirt daemon is running
INFO Checking if a supported libvirt version is installed
INFO Checking if crc-driver-libvirt is installed
INFO Checking crc daemon systemd service
INFO Setting up crc daemon systemd service
INFO Checking crc daemon systemd socket units
INFO Setting up crc daemon systemd socket units
INFO Checking if AppArmor is configured
INFO Updating AppArmor configuration
INFO Using root access: Updating AppArmor configuration
INFO Using root access: Changing permissions for /etc/apparmor.d/libvirt/TEMPLATE.qemu to 644
INFO Checking if systemd-networkd is running
`Network configuration with systemd-networkd is not supported`. Perhaps you can try this new network mode: https://github.com/code-ready/crc/wiki/VPN-support--with-an--userland-network-stack
test-server:/etc$
<domain type='kvm'>
<name>crc</name>
<uuid>48867536-d812-41a6-91d5-77c64b670cbe</uuid>
<memory unit='KiB'>9437184</memory>
<currentMemory unit='KiB'>9437184</currentMemory>
<vcpu placement='static'>4</vcpu>
<os>
<type arch='x86_64' machine='pc-q35-4.2'>hvm</type>
<boot dev='hd'/>
<bootmenu enable='no'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
</features>
<cpu mode='host-passthrough' check='none'>
<feature policy='disable' name='rdrand'/>
</cpu>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' io='threads'/>
<source file='/home/testuser/.crc/machines/crc/crc.qcow2'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
</disk>
<controller type='usb' index='0' model='qemu-xhci'>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</controller>
<controller type='sata' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pcie-root'/>
<controller type='pci' index='1' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='1' port='0x10'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='2' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='2' port='0x11'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
</controller>
<controller type='pci' index='3' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='3' port='0x12'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
</controller>
<controller type='pci' index='4' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='4' port='0x13'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
</controller>
<controller type='pci' index='5' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='5' port='0x14'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
</controller>
<serial type='stdio'>
<target type='isa-serial' port='0'>
<model name='isa-serial'/>
</target>
</serial>
<console type='stdio'>
<target type='serial' port='0'/>
</console>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' port='-1' autoport='yes'>
<listen type='address'/>
</graphics>
<video>
<model type='cirrus' vram='16384' heads='1' primary='yes'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
</video>
<memballoon model='none'/>
<rng model='virtio'>
<backend model='random'>/dev/urandom</backend>
<address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
</rng>
<vsock model='virtio'>
<cid auto='yes'/>
<address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
</vsock>
</devices>
</domain>
and also the server is a Ubuntu on ESX environment.
Yeah, this is what vmw_vsock_vmci_transport
indicates. Unfortunately, we don't have ubuntu on vmware test setups, and we don't recommend using nested virtualization with crc.
test-server:/etc$ ls -al /dev/vhost*
crw------- 1 root root 10, 241 Jun 28 10:16 /dev/vhost-vsock
Looks like the content of /etc/udev/rules.d/99-crc-vsock.rules
needs to be adjusted to set permissions to 0660
for that file. maybe changing them manually will be enough, maybe you'll need crc config set skip-check-vsock true
.
for editing rules, I don't have something like rules files:
test-server:/etc$ sudo find / -iname *crc-vsock*
/home/testuser/.config/systemd/user/crc-vsock.socket
/home/testuser/.config/systemd/user/default.target.wants/crc-vsock.socket
find: ‘/proc/1099498’: No such file or directory
find: ‘/proc/1099499’: No such file or directory
test-server:/etc$
crc config set skip-check-vsock true
didn't help.
I'm not planning to use this system for production, but I need to make some developments. so I am suffering.
/etc/udev/rules.d/99-crc-vsock.rules
is checked/created by INFO Checking if vsock is correctly configured
, running with --log-level debug
will give a few more details about what it's doing.
crc config set skip-check-vsock true
didn't help.
I don't know what 'didn't help' means, but it needs to be done in combination with adjusting /dev/vsock
permissions, this config option by itself is not useful.
I was facing the same problem. I could resolve the issue by issuing the following commands:
crc cleanup sudo rmmod vmw_vsock_vmci_transport sudo modprobe vhost_vsock
crc setup crc start
Hope this helps someone
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Same issue here on Ubuntu 23.10. https://github.com/crc-org/crc/issues/3183#issuecomment-1217841493 did not work for me.
Fixed after manual install of virtiofsd
package.
sudo apt install virtiofsd -y
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 23.10"
NAME="Ubuntu"
VERSION_ID="23.10"
VERSION="23.10 (Mantic Minotaur)"
VERSION_CODENAME=mantic
ID=ubuntu
ID_LIKE=debian
....
$ crc version
CRC version: 2.30.0+b6532a
OpenShift version: 4.14.3
Podman version: 4.4.4
Any solution, I have the same problem but nothing
I can see at least 2 issues in this bug report, one about a nested virtualization setup running ubuntu, which we don't support, and another one about the need to install virtiofsd on ubuntu for crc to work. I don't know which of these 2 issues you are having, or if this is a 3rd one.
Same issue here on Ubuntu 23.10. #3183 (comment) did not work for me.
Fixed after manual install of
virtiofsd
package.sudo apt install virtiofsd -y
Host OS
$ cat /etc/os-release PRETTY_NAME="Ubuntu 23.10" NAME="Ubuntu" VERSION_ID="23.10" VERSION="23.10 (Mantic Minotaur)" VERSION_CODENAME=mantic ID=ubuntu ID_LIKE=debian ....
CRC version
$ crc version CRC version: 2.30.0+b6532a OpenShift version: 4.14.3 Podman version: 4.4.4
installing virtiofsd package also helped me! thanks for calling this out!
Same issue here on Ubuntu 23.10. #3183 (comment) did not work for me. Fixed after manual install of
virtiofsd
package.sudo apt install virtiofsd -y
Host OS
$ cat /etc/os-release PRETTY_NAME="Ubuntu 23.10" NAME="Ubuntu" VERSION_ID="23.10" VERSION="23.10 (Mantic Minotaur)" VERSION_CODENAME=mantic ID=ubuntu ID_LIKE=debian ....
CRC version
$ crc version CRC version: 2.30.0+b6532a OpenShift version: 4.14.3 Podman version: 4.4.4
installing virtiofsd package also helped me! thanks for calling this out! installing virtiofsd package also helped me!
General information
crc setup
before starting it (Yes/No)? yesCRC version
CRC status
CRC config
Host Operating System
Steps to reproduce
Expected
crc should start and service should be accessible
Actual
crc failed to start
Logs
This issue talks about the ubuntu supported out of the box: https://github.com/code-ready/crc/issues/3074#issuecomment-1072028144