search

ipaqmaster / vfio

A script for easy pci and usb passthrough along with disks, iso's and other useful flags for quick tinkering with less of a headache. I use it for VM gaming and other PCI/LiveCD/PXE/VM/RawImage testing given the script's accessibility.
GNU General Public License v3.0
189 stars 11 forks source link

in my system (ubuntu 18.04) the command : find /sys/class/iommu/ -type l gives the wrong output. #4

Closed Marietto2008 closed 3 years ago

Marietto2008 commented 3 years ago

root@loziomario-Z390-AORUS-PRO:/home/loziomario/Scrivania/new-passT/ipaqmaster# ./main -image myWin10VM.qcow2 -imageformat raw -iso elementaryos-5.1-stable.20200814.iso -iso virtio-win-0.1.190.iso

This is a DRY run. Please specify -run to actually run IOMMU isn't loaded? For an Intel machine you must add 'intel_iommu=on' to your kernel boot options in your bootloader. If you're on an AMD machine instead add these two: 'iommu=pt iommu=1'

If you've already done this and rebooted, your host may not be compatible or some relevant bios settings could be disabled.

root@loziomario-Z390-AORUS-PRO:/home/loziomario/Scrivania/new-passT/ipaqmaster# find /sys/class/iommu

/sys/class/iommu/

root@loziomario-Z390-AORUS-PRO:/home/loziomario/Scrivania/new-passT/ipaqmaster# find /sys/class/iommu/ -type l ---> nothing

ipaqmaster commented 3 years ago

Possibly a difference between distributions. If you do a ls -alh /sys/class/iommu/ does it spit anything out from in there at all?

Marietto2008 commented 3 years ago

on ubuntu 21.04 :

root@Z390-AORUS-PRO:/home/ziomario# find /sys/class/iommu/ -type l /sys/class/iommu/dmar0 /sys/class/iommu/dmar1

ipaqmaster commented 3 years ago

That command returns two entries, I'm not sure how the "IOMMU isn't loaded" check kicked in seeing that directory is clearly populated. Will try to reproduce in ubuntu 21.04 later this week.

ipaqmaster commented 3 years ago

Ok I gave it a go this evening as below:

  1. I installed Ubuntu 21.04 onto a 120G sata SSD I have available with ZFS as the installation option.

  2. Added iommu=pt iommu=1 to my boot options as I'm on an AMD machine.

  3. I made a 100GB virtual block device in ZFS using sudo zfs create rpool/win10 -s -V100G for the VM to use

  4. I git-cloned the repo and as a quick test ran ~/vfio/main -image /dev/zvol/rpool/win10 -imageformat raw -iso ~/Downloads/Win10_21H1_English_x64.iso -iso ~/Downloads/virtio-win-0.1.189.iso -bios /usr/share/ovmf/OVMF.fd -pci 'NVIDIA|0b:00.3|AX200' -run and I was able to boot into the VM in a low driverless resolution mode just fine, install it to the 100G volume using the VirtIO drivers and eventually install the nvidia driver for the full experience.

I don't think I did anything special in this case however I did have to manually specify -bios /usr/share/ovmf/OVMF.fd given the path is different to the Archlinux system I had written this for. Unfortunately this answer still doesn't make it suitable for multi-gpu scenarios as mentioned in your other github issues but in this particular instance and combined with the output you shared from /sys/class/iommu which should not have flipped any error.. I've been unable to replicate your problem.

Marietto2008 commented 3 years ago

its easy. Your script should ask if the main OS is ubuntu or Arch,because according with the OS,there are minor changes to apply. Probably it should ask even which version of ubuntu is running,because there are little differences between versions. Anyway man. This is is a little problem. In another issue I talked about the fact that your script doesn't interact well with the final user. Its too much confusive. I would like to start here :

qemu-system-x86_64 -machine q35,accel=kvm,kernel_irqchip=on -enable-kvm -m 15986 -cpu host,kvm=off \ -smp sockets=1,cores=8,threads=2 -drive if=pflash,format=raw,unit=0,readonly,file=/usr/share/OVMF/OVMF_CODE.fd \ -drive file=budgieVM.qcow2,if=none,discard=on,id=drive1,format=qcow2 \ -device virtio-blk-pci,drive=drive1,id=virtio1 \ -device virtio-net-pci,netdev=mynet \ -netdev tap,id=mynet,ifname=tap0,script=no,downscript=no,vhost=on \ -device ich9-intel-hda,id=sound0,bus=pcie.0,addr=0x4 \ -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \ -device virtio-balloon-pci,id=balloon0,bus=pcie.0,addr=0x8 \ -device qemu-xhci,id=xhci \ -device usb-tablet,bus=xhci.0 \ -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pcie.0,addr=0x7 \ -chardev spicevmc,name=vdagent,id=vdagent \ -device virtserialport,nr=1,bus=virtio-serial0.0,chardev=vdagent,name=com.redhat.spice.0 \ -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pcie.0,addr=0x5 -device vfio-pci,host=0000:01:00.0,id=hostdev0,bus=pcie.0,addr=0x9 \ -device vfio-pci,host=0000:01:00.1,id=hostdev1,bus=pcie.0,addr=0xa \ -device vfio-pci,host=0000:01:00.2,id=hostdev2,bus=pcie.0,addr=0xb \ -device vfio-pci,host=0000:01:00.3,id=hostdev3,bus=pcie.0,addr=0xc \

as you see,between those commands,there are constants and variables. The most important variables are the PCI address. I would write a script that can grab automatically the correct PCI values (in this case they are : 0x9 ; 0xa ; 0xb ; 0xc) ; as well as the -m value,the number of threads,sockets,the different paths used,like the path of the OVMF file and it puts them automatically inside the qemu command lines. The point is that to the user should be asked very few questions. Do u want to write a script like the one that I have explained ? It would be useful for me and you. I'm not sure if it's included in your "main" script.

ipaqmaster commented 3 years ago

I don't think making the script prompt the user is a good idea. I'd rather have it autodetect what needs to be done. But with that, I wouldn't want to make exceptions for every single distribution out there making the script more cumbersome than it already is to cater for every directory path and filename difference that OVMF seems to experience distro to distro.

In my tests last night adding -bios /usr/share/ovmf/OVMF.fd to the arguments was enough to get me running completely fine and past that issue on Ubuntu 20.08. I've installed Ubuntu 18.04 and will test that tonight also to check for any breaking issues.

The qemu arguments you've provided seem regular and using -pci '01:00' in this vfio script would've caught all 4 sub devices of that one PCI device. You can try this without the -run flag and it should show what matched for fine-tuning. You can also use parts of the device name, model or manufacturer in the pci arguments of this script if you don't have any other devices plugged in that conflict and do not need to be passed through.

Marietto2008 commented 3 years ago

We should look for the costants,case by case.

on Arch :

/usr/share/ovmf/OVMF.fd

on Ubuntu 21.04 :

/usr/share/OVMF/OVMF_CODE.fd

the costant values are : /usr/share/ and .fd. So,the bash script should check what's written after /usr/share/,until .fd. It will find the missing path. it doesnt care what is that,the costant will be "added" to the missing path,just until the second constant : .fd. and there you go.

Marietto2008 commented 3 years ago

I need to understand how to grab these pci address using the proper bash command :

0x7 ; 0x9 ; 0xa ; 0xb ; 0xc.

I tried random values,but they didn't work. Sometimes it says that that pci slot is already used. In this specific case I could try from 0x1 to 0x9 and from 0xa to 0xz regardless of the errors ?

ipaqmaster commented 3 years ago

Testing with recent commits I've been able to use the script in single and dual GPU scenarios just fine on both Archlinux, Ubuntu 21.04 and Ubuntu 18.04 after installing the dependencies and adding the relevant iommu boot options. I've been unable to replicate your initial comments on the /sys/class/iommu outputs being different. As for prefilled OVMF path checks, I'm happy leaving the single check in at the moment as it really should be specified. Perhaps I'll add a lookup table or dynamic search in future however it isn't planned.

I don't understand what your last comment regarding these pci address values has to do with these issues however with some resource constraints I'm unable to help with anything outside the scope of this project at the moment.