Closed huapox closed 1 year ago
debug info
[root@host23-69 ~]# virt-host-validate
QEMU: 正在检查 for hardware virtualization : PASS
QEMU: 正在检查 if device /dev/kvm exists : PASS
QEMU: 正在检查 if device /dev/kvm is accessible : PASS
QEMU: 正在检查 if device /dev/vhost-net exists : PASS
QEMU: 正在检查 if device /dev/net/tun exists : PASS
QEMU: 正在检查 for cgroup 'memory' controller support : PASS
QEMU: 正在检查 for cgroup 'memory' controller mount-point : PASS
QEMU: 正在检查 for cgroup 'cpu' controller support : PASS
QEMU: 正在检查 for cgroup 'cpu' controller mount-point : PASS
QEMU: 正在检查 for cgroup 'cpuacct' controller support : PASS
QEMU: 正在检查 for cgroup 'cpuacct' controller mount-point : PASS
QEMU: 正在检查 for cgroup 'cpuset' controller support : PASS
QEMU: 正在检查 for cgroup 'cpuset' controller mount-point : PASS
QEMU: 正在检查 for cgroup 'devices' controller support : PASS
QEMU: 正在检查 for cgroup 'devices' controller mount-point : PASS
QEMU: 正在检查 for cgroup 'blkio' controller support : PASS
QEMU: 正在检查 for cgroup 'blkio' controller mount-point : PASS
QEMU: 正在检查 for device assignment IOMMU support : PASS
QEMU: 正在检查 if IOMMU is enabled by kernel : WARN (IOMMU appears to be disabled in kernel. Add intel_iommu=on to kernel cmdline arguments)
LXC: 正在检查 用于 Linux >= 2.6.26 : PASS
LXC: 正在检查 for namespace ipc : PASS
LXC: 正在检查 for namespace mnt : PASS
LXC: 正在检查 for namespace pid : PASS
LXC: 正在检查 for namespace uts : PASS
LXC: 正在检查 for namespace net : PASS
LXC: 正在检查 for namespace user : PASS
LXC: 正在检查 for cgroup 'memory' controller support : PASS
LXC: 正在检查 for cgroup 'memory' controller mount-point : PASS
LXC: 正在检查 for cgroup 'cpu' controller support : PASS
LXC: 正在检查 for cgroup 'cpu' controller mount-point : PASS
LXC: 正在检查 for cgroup 'cpuacct' controller support : PASS
LXC: 正在检查 for cgroup 'cpuacct' controller mount-point : PASS
LXC: 正在检查 for cgroup 'cpuset' controller support : PASS
LXC: 正在检查 for cgroup 'cpuset' controller mount-point : PASS
LXC: 正在检查 for cgroup 'devices' controller support : PASS
LXC: 正在检查 for cgroup 'devices' controller mount-point : PASS
LXC: 正在检查 for cgroup 'blkio' controller support : PASS
LXC: 正在检查 for cgroup 'blkio' controller mount-point : PASS
LXC: 正在检查 if device /sys/fs/fuse/connections exists : PASS
[root@host23-69 ~]# dmesg | grep kvm
[root@host23-69 ~]# lscpu | grep -q Virtualization || echo "No Virtualization"
No Virtualization
[root@host23-69 ~]# lsmod | grep kvm
kvm_intel 188644 0
kvm 621480 1 kvm_intel
irqbypass 13503 1 kvm
iommu=pt intel_iommu=on
, then reboot[root@host23-69 ~]# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto spectre_v2=retpoline rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet iommu=pt intel_iommu=on"
GRUB_DISABLE_RECOVERY="true"
[root@host23-69 ~]# grub2-mkconfig -o /boot/grub2/grub.cfg
#reboot
[root@host23-69 ~]# virter vm run --name alma-8-hello --id 100 --wait-ssh alma-8
INFO[0000] Create host key
INFO[0000] Define VM
INFO[0000] Create boot volume
INFO[0000] Create cloud-init volume
INFO[0000] Add DHCP entry from 52:54:00:00:00:64 to 192.168.122.100
INFO[0001] Start VM
FATA[0001] Failed to start VM 100: could not create (start) domain: 不支持的配置:this QEMU does not support 'virtio' video device
error with centos7 bare-metal:
You may be the first to try with such an old host distribution. I recommend using a newer distribution if possible.
this QEMU does not support 'virtio' video device
This is not configurable yet (see #4). You could remove the corresponding lines and build virter yourself:
diff --git a/internal/virter/libvirtxml.go b/internal/virter/libvirtxml.go
index b3b7179..099c26e 100644
--- a/internal/virter/libvirtxml.go
+++ b/internal/virter/libvirtxml.go
@@ -207,11 +207,6 @@ func (v *Virter) vmXML(poolName string, vm VMConfig, mac string, meta *VMMeta) (
Graphics: []lx.DomainGraphic{
{VNC: vncGraphics},
},
- // For some reason, debian stretch doesn't boot without a video card. The virtio model seems to stable
- // enough, even for multi-arch scenarios.
- Videos: []lx.DomainVideo{
- {Model: lx.DomainVideoModel{Type: "virtio"}},
- },
MemBalloon: &lx.DomainMemBalloon{
Model: "virtio",
Alias: &lx.DomainAlias{
@JoelColledge Thx a lot.
I rebuild the code, got next err:
[root@host23-69 ~]# virter vm run --name alma-8-hello-sam --id 101 --wait-ssh alma-8
INFO[0000] Create host key
INFO[0000] Define VM
INFO[0000] Create boot volume
INFO[0000] Create cloud-init volume
INFO[0000] Add DHCP entry from 52:54:00:00:00:65 to 192.168.122.101
INFO[0000] Start VM
FATA[0002] Failed to start VM 101: could not create (start) domain: 内部错误:process exited while connecting to monitor: qemu-kvm: -machine q35,accel=kvm,usb=off,dump-guest-core=off: Unsupported machine type
Use -machine help to list supported machines!
find the code here, maybe my old-sys not supported?
//internal/virter/arch.go
func (c *CpuArch) Machine() string {
switch c.get() {
case CpuArchAMD64:
return "q35"
case CpuArchARM64:
return "virt"
case CpuArchPPC64LE:
return "pseries"
case CpuArchS390x:
return "s390-ccw-virtio"
default:
return ""
}
}
pve-debian10
, with libvirt installed then got another network error(seems it's before the vm-starting)11:50:27 root@pve03 virter ?.uname -a
Linux pve03 5.4.106-1-pve #1 SMP PVE 5.4.106-1 (Fri, 19 Mar 2021 11:08:47 +0100) x86_64 GNU/Linux
# 11:47:29 root@pve03 virter ?.virter vm run --name alma-8-hello --id 100 --wait-ssh alma-8
INFO[0000] Create host key
INFO[0000] Define VM
INFO[0000] Create boot volume
INFO[0000] Create cloud-init volume
INFO[0000] Add DHCP entry from 52:54:00:00:00:64 to 192.168.122.100
FATA[0000] Failed to start VM 100: could not add DHCP entry: Requested operation is not valid: network is not running
find the code here, maybe my old-sys not supported?
Looks like it. You can find the supported machines with virsh capabilities
. Look at the <machine>
entries. For comparison I have:
<arch name='x86_64'>
...
<machine canonical='pc-q35-6.2' maxCpus='288'>q35</machine>
You will need to change the code to use something your host supports.
Out of interest: How did you discover virter and how do you intend to use it?
[root@host23-69 ~]# virsh capabilities |egrep "<arch|q35" -n
6: <arch>x86_64</arch>
158: <arch name='alpha'>
173: <arch name='armv7l'>
223: <arch name='cris'>
237: <arch name='i686'>
244: <machine maxCpus='255'>pc-q35-1.6</machine>
245: <machine maxCpus='255'>pc-q35-1.5</machine>
252: <machine maxCpus='255'>pc-q35-1.4</machine>
256: <machine maxCpus='255'>pc-q35-2.0</machine>
257: <machine canonical='pc-q35-2.0' maxCpus='255'>q35</machine>
260: <machine maxCpus='255'>pc-q35-1.7</machine>
290: <arch name='lm32'>
...
508: <arch name='x86_64'>
515: <machine maxCpus='255'>pc-q35-1.6</machine>
516: <machine maxCpus='255'>pc-q35-1.5</machine>
523: <machine maxCpus='255'>pc-q35-1.4</machine>
528: <machine maxCpus='255'>pc-q35-2.0</machine>
529: <machine canonical='pc-q35-2.0' maxCpus='255'>q35</machine>
531: <machine maxCpus='255'>pc-q35-1.7</machine>
559: <arch name='xtensa'>
577: <arch name='xtensaeb'>
12:02:56 root@pve03 virter ?.virsh capabilities |egrep "<arch|q35" -n
6: <arch>x86_64</arch>
152: <arch name='armv6l'>
255: <arch name='armv7l'>
358: <arch name='aarch64'>
462: <arch name='i686'>
481: <machine maxCpus='288'>pc-q35-5.1</machine>
483: <machine maxCpus='288'>pc-q35-5.2</machine>
484: <machine canonical='pc-q35-5.2' maxCpus='288'>q35</machine>
...
532: <arch name='x86_64'>
551: <machine maxCpus='288'>pc-q35-5.1</machine>
553: <machine maxCpus='288'>pc-q35-5.2</machine>
554: <machine canonical='pc-q35-5.2' maxCpus='288'>q35</machine>
--
You will need to change the code to use something your host supports.
seems q35, both supported.
cent7 ...
Not sure why qemu doesn't like the machine type, then. Let's concentrate on the Debian 10 host.
Requested operation is not valid: network is not running
Looks like the default network is not started by default on this distribution. See the commands here: https://wiki.debian.org/KVM#Libvirt_default_network
Okay, network error resolved with:
1053 2023-03-07 00:09:22 virsh --connect=qemu:///system net-start default ####
# 12:10:03 root@pve03 virter ?.ip a |grep inet |grep 122
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
12:40:28 root@pve03 images → vt network ls
Name Forward-Type IP-Range Domain DHCP Bridge
default (virter default) nat 192.168.122.1/24 192.168.122.2-192.168.122.254 virbr0
Could not open '/var/lib/libvirt/images/virter:work:alma-8-hello': Permission denied
# 12:11:23 root@pve03 ~ → virter vm run --name alma-8-hello --id 100 --wait-ssh alma-8
INFO[0000] Create host key
INFO[0000] Define VM
INFO[0000] Create boot volume
INFO[0000] Create cloud-init volume
INFO[0000] Add DHCP entry from 52:54:00:00:00:64 to 192.168.122.100
INFO[0001] Start VM
FATA[0003] Failed to start VM 100: could not create (start) domain: internal error: process exited while connecting to monitor: qemu-system-x86_64: -realtime mlock=off: warning: '-realtime mlock=...' is deprecated, please use '-overcommit mem-lock=...' instead
2023-03-06T16:11:28.339334Z qemu-system-x86_64: -drive file=/var/lib/libvirt/images/virter:work:alma-8-hello,format=qcow2,if=none,id=drive-virtio-disk0,discard=unmap: Could not open '/var/lib/libvirt/images/virter:work:alma-8-hello': Permission denied
# 12:15:04 root@pve03 virter ?.ll /var/lib/libvirt/images/
total 541M
-rw-r--r-- 1 libvirt-qemu libvirt-qemu 540M Mar 5 04:56 virter:layer:sha256:9505239cbaf56fae0abc93a2dba312606540014fcc01e74f196edac0b5fbe783
-rw-r--r-- 1 root root 193K Mar 5 04:56 virter:tag:alma-8
-rw------- 1 root root 193K Mar 7 00:11 virter:work:alma-8-hello
-rw------- 1 root root 44K Mar 7 00:11 virter:work:alma-8-hello-cidata
#qemu,libvirt's pkgs/version
12:11:28 root@pve03 ~ → dpkg -l |grep qemu
ii libproxmox-backup-qemu0 1.0.3-1 amd64 Proxmox Backup Server client library for QEMU
ii pve-qemu-kvm 5.2.0-6 amd64 Full virtualization on x86 hardware
ii qemu-server 6.4-1 amd64 Qemu Server Tools
12:25:40 root@pve03 ~ → dpkg -l |grep libvirt
ii libvirt-clients 5.0.0-4+deb10u1 amd64 Programs for the libvirt library
ii libvirt-daemon 5.0.0-4+deb10u1 amd64 Virtualization daemon
ii libvirt-daemon-system 5.0.0-4+deb10u1 amd64 Libvirt daemon configuration files
ii libvirt0:amd64 5.0.0-4+deb10u1 amd64 library for interfacing with different virtualization systems
--
Out of interest: How did you discover virter and how do you intend to use it?
find at github libvirt
filtered by golang; I'm intend to use this for Pressure test
BareMetal> VM> KubeEdge(edgecore).
I'm with poor net-quality to github, apoligize for my slow response
chmod 777
12:36:26 root@pve03 images → virter vm run --name alma-8-hello-sam --id 101 --wait-ssh alma-8
INFO[0000] Create host key
INFO[0000] Define VM
INFO[0000] Create boot volume
INFO[0000] Create cloud-init volume
INFO[0000] Add DHCP entry from 52:54:00:00:00:65 to 192.168.122.101
INFO[0001] Start VM
FATA[0002] Failed to start VM 101: could not create (start) domain: internal error: qemu unexpectedly closed the monitor: qemu-system-x86_64: -realtime mlock=off: warning: '-realtime mlock=...' is deprecated, please use '-overcommit mem-lock=...' instead
2023-03-06T16:36:42.978783Z qemu-system-x86_64: -drive file=/var/lib/libvirt/images/virter:work:alma-8-hello-sam,format=qcow2,if=none,id=drive-virtio-disk0,discard=unmap: Could not open '/var/lib/libvirt/images/virter:work:alma-8-hello-sam': Permission denied
12:36:43 root@pve03 images → ll
total 541M
-rw-r--r-- 1 libvirt-qemu libvirt-qemu 540M Mar 5 04:56 virter:layer:sha256:9505239cbaf56fae0abc93a2dba312606540014fcc01e74f196edac0b5fbe783
-rw-r--r-- 1 root root 193K Mar 5 04:56 virter:tag:alma-8
-rwxrwxrwx 1 root root 193K Mar 7 00:33 virter:work:alma-8-hello*
-rwxrwxrwx 1 root root 44K Mar 7 00:33 virter:work:alma-8-hello-cidata*
-rw------- 1 root root 193K Mar 7 00:36 virter:work:alma-8-hello-sam
-rw------- 1 root root 44K Mar 7 00:36 virter:work:alma-8-hello-sam-cidata
12:36:55 root@pve03 images →chmod 777 /var/lib/libvirt/images/virter:work:alma-8-hello*
but how could I restart a VM which is existed?
12:35:30 root@pve03 images → vt vm -h
Virtual machine related subcommands.
Usage:
virter vm [command]
Available Commands:
commit Commit a virtual machine
cp Copy files and directories from and to VM
exec Run provisioning steps on VMs
exists Check whether a VM exists
host-key Get the host key for a VM
rm Remove virtual machines
run Start a virtual machine with a given image
ssh Run an interactive ssh shell in a VM
Flags:
-h, --help help for vm
I've tried with chmod/chown files to libvirt/images or the disk files, then use virsh -c qemu:///system "start alma-8-hello"
but not work for me (newbie to libvirt)
01:21:23 root@pve03 images → virsh -c qemu:///system "list --all"
Id Name State
-----------------------------------
- alma-8-hello shut off
- alma-8-hello-sam shut off
01:22:09 root@pve03 images → virsh -c qemu:///system "start alma-8-hello"
error: Failed to start domain alma-8-hello
error: internal error: qemu unexpectedly closed the monitor: qemu-system-x86_64: -realtime mlock=off: warning: '-realtime mlock=...' is deprecated, please use '-overcommit mem-lock=...' instead
2023-03-06T17:22:45.664025Z qemu-system-x86_64: -drive file=/var/lib/libvirt/images/virter:work:alma-8-hello,format=qcow2,if=none,id=drive-virtio-disk0,discard=unmap: Could not open '/var/lib/libvirt/images/virter:work:alma-8-hello': Permission denied
01:22:45 root@pve03 images → chmod 777 /var/lib/libvirt/images/virter:work:alma-8-hello
01:23:11 root@pve03 images → ll
total 541M
-rw-r--r-- 1 root root 540M Mar 5 04:56 virter:layer:sha256:9505239cbaf56fae0abc93a2dba312606540014fcc01e74f196edac0b5fbe783
-rw-r--r-- 1 libvirt-qemu libvirt-qemu 193K Mar 5 04:56 virter:tag:alma-8
-rwxrwxrwx 1 root root 193K Mar 7 00:33 virter:work:alma-8-hello*
-rwxrwxrwx 1 root root 44K Mar 7 00:33 virter:work:alma-8-hello-cidata*
-rw------- 1 root root 193K Mar 7 01:21 virter:work:alma-8-hello-sam
-rw------- 1 root root 44K Mar 7 01:21 virter:work:alma-8-hello-sam-cidata
01:24:12 root@pve03 images →virsh -c qemu:///system "start alma-8-hello"o
error: Failed to start domain alma-8-hello
error: internal error: qemu unexpectedly closed the monitor: qemu-system-x86_64: -realtime mlock=off: warning: '-realtime mlock=...' is deprecated, please use '-overcommit mem-lock=...' instead
2023-03-06T17:24:36.192369Z qemu-system-x86_64: -drive file=/var/lib/libvirt/images/virter:work:alma-8-hello,format=qcow2,if=none,id=drive-virtio-disk0,discard=unmap: Could not open '/var/lib/libvirt/images/virter:work:alma-8-hello': Permission denied
01:24:36 root@pve03 images → chown libvirt-qemu:libvirt-qemu virter:work:alma-8-hello*
01:25:01 root@pve03 images → ll
total 541M
-rw-r--r-- 1 root root 540M Mar 5 04:56 virter:layer:sha256:9505239cbaf56fae0abc93a2dba312606540014fcc01e74f196edac0b5fbe783
-rw-r--r-- 1 libvirt-qemu libvirt-qemu 193K Mar 5 04:56 virter:tag:alma-8
-rwxrwxrwx 1 libvirt-qemu libvirt-qemu 193K Mar 7 00:33 virter:work:alma-8-hello*
-rwxrwxrwx 1 libvirt-qemu libvirt-qemu 44K Mar 7 00:33 virter:work:alma-8-hello-cidata*
-rw------- 1 libvirt-qemu libvirt-qemu 193K Mar 7 01:21 virter:work:alma-8-hello-sam
-rw------- 1 libvirt-qemu libvirt-qemu 44K Mar 7 01:21 virter:work:alma-8-hello-sam-cidata
01:25:02 root@pve03 images →virsh -c qemu:///system "start alma-8-hello"
error: Failed to start domain alma-8-hello
error: internal error: qemu unexpectedly closed the monitor: qemu-system-x86_64: -realtime mlock=off: warning: '-realtime mlock=...' is deprecated, please use '-overcommit mem-lock=...' instead
2023-03-06T17:25:08.440474Z qemu-system-x86_64: -drive file=/var/lib/libvirt/images/virter:work:alma-8-hello,format=qcow2,if=none,id=drive-virtio-disk0,discard=unmap: Could not open '/var/lib/libvirt/images/virter:work:alma-8-hello': Permission denied
01:25:08 root@pve03 images → ll ../
total 20K
drwx--x--x 2 root root 4.0K Dec 5 2019 boot/
drwxr-xr-x 2 root root 4.0K Mar 7 01:25 dnsmasq/
drwxr-xr-x 2 root root 4.0K Mar 7 01:21 images/
ref: https://superuser.com/questions/298426/kvm-image-failed-to-start-with-virsh-permission-denied
a easy solution would be to edit the /etc/libvirt/qemu.conf file and uncommenting the following lines:
User = "root"
group = "root"
Don't forget to restart libvirtd
# 01:32:21 root@pve03 images → cat /etc/libvirt/qemu.conf |egrep "^user|^group" -n
442:user = "root"
446:group = "root"
may AppArmor?
but how could I restart a VM which is existed?
Just remove it with virter vm rm ...
and then start it again. For more fine-grained control you would have to use virsh.
may AppArmor?
Sounds like it. See the README: https://github.com/LINBIT/virter#apparmor
@JoelColledge Thx again~
--
both tested ok with a container which contains qemu+libvirt
ref: https://github.com/Speedy37/docker-libvirtd
# 23.69-cent7 ok
# 23.72-pve ok:
[root@host23-69 ct-virt]# docker run --privileged -v $(pwd)/run:/var/run/libvirt -v /var/lib/libvirt:/var/lib/libvirt ghcr.io/speedy37/docker-libvirtd/libvirtd:main
ln -s /root/ct-virt/run/libvirt-sock /run/libvirt/
2.15.0/barge.qcow2
)[root@host23-69 virter]# pwd
/root/.local/share/virter
[root@host23-69 virter]# cat images.toml
...
[bargee-215]
url= "https://ghproxy.com/https://github.com/bargees/barge-packer/releases/download/2.15.0/barge.qcow2"
root@pve2372:~/.local/share/virter# virter vm run --name bargee215-sam105 --id 105 --wait-ssh bargee-215
bargee-215 pull done [==========================================================================================] 15.88MiB / 15.88MiB
virter:layer:sha256:fbdd buffer layer done [============================================================================] 15.88MiB / 15.88MiB
virter:layer:sha256:fbdd upload layer done [============================================================================] 15.88MiB / 15.88MiB
INFO[0013] Create host key
INFO[0013] Define VM
INFO[0013] Create boot volume
INFO[0013] Create cloud-init volume
INFO[0013] Add DHCP entry from 52:54:00:00:00:69 to 192.168.122.105
INFO[0015] Start VM
INFO[0015] Wait for VM to get ready
root@pve2372:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0355de6332a2 ghcr.io/speedy37/docker-libvirtd/libvirtd:main "/usr/bin/supervisor…" 7 minutes ago Up 7 minutes frosty_davinci
root@pve2372:~# docker exec -it 0355 bash
bash-5.1# virsh list
Id Name State
----------------------------------
1 bargee215-sam105 running
bash-5.1# apk add openssh-client
bash-5.1# ssh bargee@192.168.122.105
The authenticity of host '192.168.122.105 (192.168.122.105)' can't be established.
ECDSA key fingerprint is SHA256:3Rqkzbhd3LfyXxwKwlW2EST/Jwpas7JxUHZl7jt5770.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.122.105' (ECDSA) to the list of known hosts.
bargee@192.168.122.105's password:
Welcome to Barge 2.15.0, Docker version 1.10.3, build 662b14f
[bargee@barge ~]$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[bargee@barge ~]$
/close
Glad to hear that it is working. It is interesting to see how you are using the tool.
error with centos7 bare-metal: