search

Arc-Compute / LibVF.IO

A vendor neutral GPU multiplexing tool driven by VFIO & YAML.
GNU Affero General Public License v3.0
790 stars 39 forks source link

Debian bookwrom support #68

Open k-u-s opened 1 year ago

k-u-s commented 1 year ago

Tried following setup guide on clean installation of Debian 12 Bookworm RC 1 and did not go as planned. I manage to start VM but display driver seams to not work.

First think I needed to change was to remove qemu package from apt install cause its no longer available in newer sources but since its dummy package I do not think it causes any problems (but its needs to be done other wise installation of other packages also fails)

Then first step went as expected after restart got small res screen. Starting again got prompts as mentioned in guide (plus telemetry prompt from choosonim) but after restarting display was still in small res (and could not be changed). Creating vm works but do not shows nvidia graphic device in windows Device Manager.

Are there any additional requirements from host to make it work? If you think it can help I can re run those steps to get full logs from installation.

Hardware: Intel i9-9900K Nvidia Gefroce RTX 2080

k-u-s commented 1 year ago

Tested it a little bit more and after downgrading kernel from 6.1 to 5.10 after fresh setup, I manged to finish both steps and get normal resolutions and multiple screen working again. I did not get a vm to recognize gpu yet but will try to check that later.

k-u-s commented 1 year ago

Further testing unfortunately did not allow to get working vm with gpu. Set up went as followed:

After last step first there was blue screen then after restart or two graphic card disappeared and did not work anymore. Last detail that came to my mind is that when shutting down VM when msg "INFO: Installing to base kernel" appeared I waited a few minutes then ctrl+c to stop it cause even when i left it for few hours it did not completed.

Logs from terminal - arcd create ~ arcd create VmWinGeneric.yaml ./Downloads/Win10_22H2_EnglishInternational_x64.iso 200 [2023-04-18 11:11:10] - INFO: Executing: (exec: "echo", args: @["\'d41d7b0b-0cc6-4243-a435-37ed48e34a97\'", ">", "/sys/class/mdev_bus/0000:01:00.0/mdev_supported_types/nvidia-1/create"]) [2023-04-18 11:11:10] - INFO: Executing: (exec: "qemu-img", args: @["create", "-f", "qcow2", "-o", "preallocation=metadata", "/home/kus/.local/libvf.io/live/e2eea016-fe7e-4a58-885b-c4dde39140b0", "200G"]) Formatting '/home/kus/.local/libvf.io/live/e2eea016-fe7e-4a58-885b-c4dde39140b0', fmt=qcow2 cluster_size=65536 extended_l2=off preallocation=metadata compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2023-04-18 11:11:11] - INFO: Executing: (exec: "/bin/qemu-system-x86_64", args: @["-D", "/home/kus/.local/libvf.io/logs/qemu/e2eea016-fe7e-4a58-885b-c4dde39140b0-session.txt", "-no-hpet", "-mem-prealloc", "-uuid", "e2eea016-fe7e-4a58-885b-c4dde39140b0", "-machine", "pc-q35-4.2,accel=kvm,usb=off,vmport=off,dump-guest-core=off", "-cpu", "host,ss=on,vmx=on,pcid=on,-hypervisor,arat=on,tsc-adjust=on,umip=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaveopt=on,pdpe1gb=on,ibpb=on,amd-stibp=on,amd-ssbd=on,skip-l1dfl-vmentry=on,pschange-mc-no=on,hv-vapic,hv_time,hv-spinlocks=0x1fff,hv-vendor-id=null,kvm=off,topoext=on", "-rtc", "clock=host,base=localtime", "-m", "8192", "-smp", "cores=4,threads=1,sockets=1", "-hda", "/home/kus/.local/libvf.io/live/e2eea016-fe7e-4a58-885b-c4dde39140b0", "--enable-kvm", "-device", "vfio-pci,id=hostdev0,sysfsdev=/sys/bus/mdev/devices/d41d7b0b-0cc6-4243-a435-37ed48e34a97,display=off", "-device", "rtl8139,netdev=net0", "-netdev", "user,id=net0,hostfwd=tcp::2222-:22", "-net", "user", "-net", "nic", "-device", "vhost-vsock-pci,guest-cid=11467", "-qmp", "unix:/tmp/sockets/e2eea016-fe7e-4a58-885b-c4dde39140b0/main.sock,server,nowait", "-qmp", "unix:/tmp/sockets/e2eea016-fe7e-4a58-885b-c4dde39140b0/master.sock,server,nowait", "-cdrom", "./Downloads/Win10_22H2_EnglishInternational_x64.iso"]) ~ qemu-system-x86_64: -device vfio-pci,id=hostdev0,sysfsdev=/sys/bus/mdev/devices/d41d7b0b-0cc6-4243-a435-37ed48e34a97,display=off: warning: vfio d41d7b0b-0cc6-4243-a435-37ed48e34a97: Could not enable error recovery for the device qemu-system-x86_64: -device vhost-vsock-pci,guest-cid=11467: warning: device is modern-only, but for backward compatibility legacy is allowed [2023-04-18 11:11:14] - INFO: Executing: (exec: "/usr/bin/chgrp", args: @["kvm", "/tmp/sockets/e2eea016-fe7e-4a58-885b-c4dde39140b0/main.sock", "/tmp/sockets/e2eea016-fe7e-4a58-885b-c4dde39140b0/master.sock"]) [2023-04-18 11:11:14] - INFO: Executing: (exec: "/usr/bin/chmod", args: @["g+rwx", "/tmp/sockets/e2eea016-fe7e-4a58-885b-c4dde39140b0/main.sock", "/tmp/sockets/e2eea016-fe7e-4a58-885b-c4dde39140b0/master.sock"]) [2023-04-18 11:11:14] - INFO: Connecting to the socket [2023-04-18 11:11:14] - WARN: QMP has not been tested for this version [2023-04-18 11:11:14] - DEBUG: Sending to socket: {"execute":"qmp_capabilities"} [2023-04-18 11:11:14] - DEBUG: Received from socket: {"return": {}} [2023-04-18 11:11:14] - DEBUG: Response: (event: return) [2023-04-18 11:13:13] - DEBUG: Received from socket: {"timestamp": {"seconds": 1681809193, "microseconds": 358481}, "event": "RESET", "data": {"guest": true, "reason": "guest-reset"}} [2023-04-18 11:13:13] - DEBUG: Response: (event: INVALID, errorMessage: "Not applicable") [2023-04-18 11:13:13] - DEBUG: Received from socket: {"timestamp": {"seconds": 1681809193, "microseconds": 418270}, "event": "RESET", "data": {"guest": true, "reason": "guest-reset"}} [2023-04-18 11:13:13] - DEBUG: Response: (event: INVALID, errorMessage: "Not applicable") [2023-04-18 11:14:18] - DEBUG: Received from socket: {"timestamp": {"seconds": 1681809258, "microseconds": 511460}, "event": "RESET", "data": {"guest": true, "reason": "guest-reset"}} [2023-04-18 11:14:18] - DEBUG: Response: (event: INVALID, errorMessage: "Not applicable") [2023-04-18 11:14:18] - DEBUG: Received from socket: {"timestamp": {"seconds": 1681809258, "microseconds": 540317}, "event": "RESET", "data": {"guest": true, "reason": "guest-reset"}} [2023-04-18 11:14:18] - DEBUG: Response: (event: INVALID, errorMessage: "Not applicable") [2023-04-18 11:14:31] - DEBUG: Received from socket: {"timestamp": {"seconds": 1681809271, "microseconds": 386237}, "event": "RTC_CHANGE", "data": {"offset": 0, "qom-path": "/machine/unattached/device[10]"}} [2023-04-18 11:14:31] - DEBUG: Response: (event: INVALID, errorMessage: "Not applicable") [2023-04-18 11:14:32] - DEBUG: Received from socket: {"timestamp": {"seconds": 1681809271, "microseconds": 728301}, "event": "RTC_CHANGE", "data": {"offset": 0, "qom-path": "/machine/unattached/device[10]"}} [2023-04-18 11:14:32] - DEBUG: Response: (event: INVALID, errorMessage: "Not applicable") [2023-04-18 11:14:55] - DEBUG: Received from socket: {"timestamp": {"seconds": 1681809295, "microseconds": 569289}, "event": "RTC_CHANGE", "data": {"offset": 0, "qom-path": "/machine/unattached/device[10]"}} [2023-04-18 11:14:55] - DEBUG: Response: (event: INVALID, errorMessage: "Not applicable") [2023-04-18 11:14:56] - DEBUG: Received from socket: {"timestamp": {"seconds": 1681809295, "microseconds": 569444}, "event": "RTC_CHANGE", "data": {"offset": 0, "qom-path": "/machine/unattached/device[10]"}} [2023-04-18 11:14:56] - DEBUG: Response: (event: INVALID, errorMessage: "Not applicable") [2023-04-18 11:16:23] - DEBUG: Received from socket: {"timestamp": {"seconds": 1681809383, "microseconds": 106937}, "event": "RESET", "data": {"guest": true, "reason": "guest-reset"}} [2023-04-18 11:16:23] - DEBUG: Response: (event: INVALID, errorMessage: "Not applicable") [2023-04-18 11:16:23] - DEBUG: Received from socket: {"timestamp": {"seconds": 1681809383, "microseconds": 142869}, "event": "RESET", "data": {"guest": true, "reason": "guest-reset"}} [2023-04-18 11:16:23] - DEBUG: Response: (event: INVALID, errorMessage: "Not applicable") qemu: Slirp: Failed to send packet, ret: -1 [2023-04-18 11:30:14] - DEBUG: Received from socket: {"timestamp": {"seconds": 1681810214, "microseconds": 80200}, "event": "RESET", "data": {"guest": true, "reason": "guest-reset"}} [2023-04-18 11:30:14] - DEBUG: Response: (event: INVALID, errorMessage: "Not applicable") [2023-04-18 11:30:14] - DEBUG: Received from socket: {"timestamp": {"seconds": 1681810214, "microseconds": 113475}, "event": "RESET", "data": {"guest": true, "reason": "guest-reset"}} [2023-04-18 11:30:14] - DEBUG: Response: (event: INVALID, errorMessage: "Not applicable") [2023-04-18 11:36:48] - DEBUG: Received from socket: {"timestamp": {"seconds": 1681810608, "microseconds": 377869}, "event": "SUSPEND_DISK"} [2023-04-18 11:36:48] - DEBUG: Response: (event: INVALID, errorMessage: "Not applicable") [2023-04-18 11:36:48] - DEBUG: Received from socket: {"timestamp": {"seconds": 1681810608, "microseconds": 377946}, "event": "SHUTDOWN", "data": {"guest": true, "reason": "guest-shutdown"}} [2023-04-18 11:36:48] - DEBUG: Response: (event: SHUTDOWN) [2023-04-18 11:36:48] - INFO: Cleaning up VM [2023-04-18 11:36:48] - INFO: Executing: (exec: "echo", args: @["\'1\'", ">", "/sys/bus/mdev/devices/d41d7b0b-0cc6-4243-a435-37ed48e34a97/remove"]) [2023-04-18 11:36:48] - INFO: Deleted MDEV: hostdev0 [2023-04-18 11:36:48] - INFO: Executing: (exec: "/bin/rm", args: @["-rf", "/home/kus/.local/libvf.io/lime/e2eea016-fe7e-4a58-885b-c4dde39140b0"]) [2023-04-18 11:36:48] - INFO: Running teardown command lists [2023-04-18 11:36:48] - INFO: Cleaned up VM [2023-04-18 11:36:48] - INFO: Save e2eea016-fe7e-4a58-885b-c4dde39140b0 VM: false [2023-04-18 11:36:48] - INFO: Installing to base kernel
Logs from terminal - arcd start ~ arcd start VmWinGeneric.yaml --safe-mode [2023-04-18 11:38:01] - INFO: Executing: (exec: "echo", args: @["\'6910ebbc-0c06-4aa7-9b6e-c419ca8d1cc3\'", ">", "/sys/class/mdev_bus/0000:01:00.0/mdev_supported_types/nvidia-1/create"]) [2023-04-18 11:38:01] - INFO: Executing: (exec: "/bin/qemu-system-x86_64", args: @["-D", "/home/kus/.local/libvf.io/logs/qemu/2c27f4af-5e6d-41b0-9f0f-9556a262ff30-session.txt", "-no-hpet", "-mem-prealloc", "-device", "ivshmem-plain,id=shmem0,memdev=ivshmem_kvmfr", "-object", "memory-backend-file,id=ivshmem_kvmfr,mem-path=/dev/shm/kvmfr-2c27f4af-5e6d-41b0-9f0f-9556a262ff30,size=128M,share=yes", "-device", "ivshmem-plain,id=shmem1,memdev=ivshmem_kvmsr", "-object", "memory-backend-file,id=ivshmem_kvmsr,mem-path=/dev/shm/kvmsr-2c27f4af-5e6d-41b0-9f0f-9556a262ff30,size=2M,share=yes", "-uuid", "2c27f4af-5e6d-41b0-9f0f-9556a262ff30", "-machine", "pc-q35-4.2,accel=kvm,usb=off,vmport=off,dump-guest-core=off", "-cpu", "host,ss=on,vmx=on,pcid=on,-hypervisor,arat=on,tsc-adjust=on,umip=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaveopt=on,pdpe1gb=on,ibpb=on,amd-stibp=on,amd-ssbd=on,skip-l1dfl-vmentry=on,pschange-mc-no=on,hv-vapic,hv_time,hv-spinlocks=0x1fff,hv-vendor-id=null,kvm=off,topoext=on", "-rtc", "clock=host,base=localtime", "-m", "8192", "-smp", "cores=4,threads=1,sockets=1", "-hda", "/home/kus/.local/libvf.io/kernel/windows.arc", "--enable-kvm", "-device", "vfio-pci,id=hostdev0,sysfsdev=/sys/bus/mdev/devices/6910ebbc-0c06-4aa7-9b6e-c419ca8d1cc3,display=off", "-device", "rtl8139,netdev=net0", "-netdev", "user,id=net0,hostfwd=tcp::2222-:22", "-net", "user", "-net", "nic", "-device", "vhost-vsock-pci,guest-cid=82104", "-qmp", "unix:/tmp/sockets/2c27f4af-5e6d-41b0-9f0f-9556a262ff30/main.sock,server,nowait", "-qmp", "unix:/tmp/sockets/2c27f4af-5e6d-41b0-9f0f-9556a262ff30/master.sock,server,nowait"]) ~ qemu-system-x86_64: -device vfio-pci,id=hostdev0,sysfsdev=/sys/bus/mdev/devices/6910ebbc-0c06-4aa7-9b6e-c419ca8d1cc3,display=off: warning: vfio 6910ebbc-0c06-4aa7-9b6e-c419ca8d1cc3: Could not enable error recovery for the device qemu-system-x86_64: -device vhost-vsock-pci,guest-cid=82104: warning: device is modern-only, but for backward compatibility legacy is allowed [2023-04-18 11:38:04] - INFO: Executing: (exec: "/usr/bin/chgrp", args: @["kvm", "/tmp/sockets/2c27f4af-5e6d-41b0-9f0f-9556a262ff30/main.sock", "/tmp/sockets/2c27f4af-5e6d-41b0-9f0f-9556a262ff30/master.sock", "/dev/shm/kvmfr-2c27f4af-5e6d-41b0-9f0f-9556a262ff30", "/dev/shm/kvmsr-2c27f4af-5e6d-41b0-9f0f-9556a262ff30"]) [2023-04-18 11:38:04] - INFO: Executing: (exec: "/usr/bin/chmod", args: @["g+rwx", "/tmp/sockets/2c27f4af-5e6d-41b0-9f0f-9556a262ff30/main.sock", "/tmp/sockets/2c27f4af-5e6d-41b0-9f0f-9556a262ff30/master.sock", "/dev/shm/kvmfr-2c27f4af-5e6d-41b0-9f0f-9556a262ff30", "/dev/shm/kvmsr-2c27f4af-5e6d-41b0-9f0f-9556a262ff30"]) [2023-04-18 11:38:04] - INFO: Connecting to the socket [2023-04-18 11:38:04] - WARN: QMP has not been tested for this version [2023-04-18 11:38:04] - DEBUG: Sending to socket: {"execute":"qmp_capabilities"} [2023-04-18 11:38:04] - DEBUG: Received from socket: {"return": {}} [2023-04-18 11:38:04] - DEBUG: Response: (event: return) [2023-04-18 11:38:26] - DEBUG: Received from socket: {"timestamp": {"seconds": 1681810706, "microseconds": 406361}, "event": "RESET", "data": {"guest": true, "reason": "guest-reset"}} [2023-04-18 11:38:26] - DEBUG: Response: (event: INVALID, errorMessage: "Not applicable") [2023-04-18 11:38:26] - DEBUG: Received from socket: {"timestamp": {"seconds": 1681810706, "microseconds": 442101}, "event": "RESET", "data": {"guest": true, "reason": "guest-reset"}} [2023-04-18 11:38:26] - DEBUG: Response: (event: INVALID, errorMessage: "Not applicable") qemu: vfio_region_read(6910ebbc-0c06-4aa7-9b6e-c419ca8d1cc3:region0+0x0, 4) failed: Bad address qemu: vfio_region_read(6910ebbc-0c06-4aa7-9b6e-c419ca8d1cc3:region0+0xa00, 4) failed: Bad address qemu: vfio_region_read(6910ebbc-0c06-4aa7-9b6e-c419ca8d1cc3:region0+0x4, 4) failed: Bad address [2023-04-18 11:40:06] - DEBUG: Received from socket: {"timestamp": {"seconds": 1681810806, "microseconds": 578535}, "event": "RESET", "data": {"guest": true, "reason": "guest-reset"}} [2023-04-18 11:40:06] - DEBUG: Response: (event: INVALID, errorMessage: "Not applicable") [2023-04-18 11:40:06] - DEBUG: Received from socket: {"timestamp": {"seconds": 1681810806, "microseconds": 634095}, "event": "RESET", "data": {"guest": true, "reason": "guest-reset"}} [2023-04-18 11:40:06] - DEBUG: Response: (event: INVALID, errorMessage: "Not applicable") qemu: vfio_region_read(6910ebbc-0c06-4aa7-9b6e-c419ca8d1cc3:region0+0x0, 4) failed: Bad address [2023-04-18 11:40:53] - DEBUG: Received from socket: {"timestamp": {"seconds": 1681810853, "microseconds": 638854}, "event": "RESET", "data": {"guest": true, "reason": "guest-reset"}} [2023-04-18 11:40:53] - DEBUG: Response: (event: INVALID, errorMessage: "Not applicable") [2023-04-18 11:40:53] - DEBUG: Received from socket: {"timestamp": {"seconds": 1681810853, "microseconds": 669018}, "event": "RESET", "data": {"guest": true, "reason": "guest-reset"}} [2023-04-18 11:40:53] - DEBUG: Response: (event: INVALID, errorMessage: "Not applicable") [2023-04-18 11:41:09] - DEBUG: Received from socket: {"timestamp": {"seconds": 1681810869, "microseconds": 375886}, "event": "RESET", "data": {"guest": true, "reason": "guest-reset"}} [2023-04-18 11:41:09] - DEBUG: Response: (event: INVALID, errorMessage: "Not applicable") [2023-04-18 11:41:09] - DEBUG: Received from socket: {"timestamp": {"seconds": 1681810869, "microseconds": 411485}, "event": "RESET", "data": {"guest": true, "reason": "guest-reset"}} [2023-04-18 11:41:09] - DEBUG: Response: (event: INVALID, errorMessage: "Not applicable") qemu: vfio_region_read(6910ebbc-0c06-4aa7-9b6e-c419ca8d1cc3:region0+0x0, 4) failed: Bad address [2023-04-18 11:41:55] - DEBUG: Received from socket: {"timestamp": {"seconds": 1681810915, "microseconds": 159323}, "event": "RESET", "data": {"guest": true, "reason": "guest-reset"}} [2023-04-18 11:41:55] - DEBUG: Response: (event: INVALID, errorMessage: "Not applicable") [2023-04-18 11:41:55] - DEBUG: Received from socket: {"timestamp": {"seconds": 1681810915, "microseconds": 192772}, "event": "RESET", "data": {"guest": true, "reason": "guest-reset"}} [2023-04-18 11:41:55] - DEBUG: Response: (event: INVALID, errorMessage: "Not applicable") qemu: vfio_region_read(6910ebbc-0c06-4aa7-9b6e-c419ca8d1cc3:region0+0x0, 4) failed: Bad address qemu: vfio_region_read(6910ebbc-0c06-4aa7-9b6e-c419ca8d1cc3:region0+0xa00, 4) failed: Bad address qemu: vfio_region_read(6910ebbc-0c06-4aa7-9b6e-c419ca8d1cc3:region0+0x4, 4) failed: Bad address [2023-04-18 11:45:49] - DEBUG: Received from socket: {"timestamp": {"seconds": 1681811149, "microseconds": 267174}, "event": "POWERDOWN"} [2023-04-18 11:45:49] - DEBUG: Response: (event: POWERDOWN) [2023-04-18 11:45:53] - DEBUG: Received from socket: {"timestamp": {"seconds": 1681811153, "microseconds": 890335}, "event": "SUSPEND_DISK"} [2023-04-18 11:45:53] - DEBUG: Response: (event: INVALID, errorMessage: "Not applicable") [2023-04-18 11:45:53] - DEBUG: Received from socket: {"timestamp": {"seconds": 1681811153, "microseconds": 890400}, "event": "SHUTDOWN", "data": {"guest": true, "reason": "guest-shutdown"}} [2023-04-18 11:45:53] - DEBUG: Response: (event: SHUTDOWN) [2023-04-18 11:45:54] - INFO: Cleaning up VM [2023-04-18 11:45:54] - INFO: Executing: (exec: "echo", args: @["\'1\'", ">", "/sys/bus/mdev/devices/6910ebbc-0c06-4aa7-9b6e-c419ca8d1cc3/remove"]) [2023-04-18 11:45:54] - INFO: Deleted MDEV: hostdev0 [2023-04-18 11:45:54] - INFO: Executing: (exec: "/bin/rm", args: @["-rf", "/dev/shm/kvmfr-2c27f4af-5e6d-41b0-9f0f-9556a262ff30", "/dev/shm/kvmsr-2c27f4af-5e6d-41b0-9f0f-9556a262ff30", "/home/kus/.local/libvf.io/lime/2c27f4af-5e6d-41b0-9f0f-9556a262ff30"]) [2023-04-18 11:45:54] - INFO: Running teardown command lists [2023-04-18 11:45:54] - INFO: Cleaned up VM [2023-04-18 11:45:54] - INFO: Save 2c27f4af-5e6d-41b0-9f0f-9556a262ff30 VM: true [2023-04-18 11:45:54] - INFO: Installing to base kernel
VmWinGeneric.yaml ```yaml %YAML 1.2 %TAG !n! tag:nimyaml.org,2016: --- !n!custom:Config name: "Win Generic VM" startintro: true startapp: false installOs: "none" nographics: true spice: true showhypervisor: false introspect: "looking-glass" shareddir: !!null ~ spicePort: 5900 sshPort: 2222 connectivity: exposedPorts: [] container: kernel: windows.arc state: [] initialSize: 20 iso: !!null ~ cpus: cores: 4 sockets: 1 threads: 1 ramAlloc: 8192 gpus: - - maxVRam: 4000 - minVRam: 1000 - gpuType: sysfsdev - mdevType: "nvidia-1" - parentPort: !!null ~ - devId: hostdev0 nics: [] root: ~/.local/libvf.io sudo: false commands: [] startupCommands: [] teardownCommands: [] installCommands: [] appCommands: [] ```