search

intel / gvt-linux

Other
509 stars 95 forks source link

OVMF / UEFI support? #23

Closed DocMAX closed 5 years ago

DocMAX commented 6 years ago

When will it come for VT-d? What are the problems at the moment? What has to be modified? Will this ever work? Thanks.

TerrenceXu commented 6 years ago

GVT-d also support OVMT / UEFI VM, thanks. ☺

From: DocMAX [mailto:notifications@github.com] Sent: Friday, March 23, 2018 6:46 AM To: intel/gvt-linux gvt-linux@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: [intel/gvt-linux] OVMF / UEFI support? (#23)

When will it come for VT-d? What are the problems at the moment? What has to be modified? Will this ever work? Thanks.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/intel/gvt-linux/issues/23, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AYv0NMJHPuC4NBahfXZUuetYY3qx_UIsks5thCmrgaJpZM4S36PI.

jscinoz commented 6 years ago

Is support for OVMF + GVT on the roadmap? At present it does not appear to work with either a Q35 or I440FX VM using OVMF (SeaBIOS does work, with both machine types)

DocMAX commented 6 years ago

any news?

TerrenceXu commented 6 years ago

Do you means GVT-g? GVT-g has support OVMF for a long time.

DocMAX commented 6 years ago

the classic gvt-d.... (non virtual gpu). doesnt work with ovmf... screen stays black. seabios works.

can somebody confirm that intel gpu passthrough is working with OVMF UEFI bios???

DocMAX commented 6 years ago

Any updates please?

hyuan3 commented 5 years ago

Display doesn't work with OVMF bios. It's because display needs OpRegion which is not available to guest in OVMF bios scenario. Please use seabios or UPT mode.

CuriousTommy commented 5 years ago

Apparently it is possible to use the display on OVMF. Going from what this user has said, you can do the following:

Disclaimer: I haven't tested this.

DocMAX commented 5 years ago

thanks. works with gvt-g but not with gvt-d.

CuriousTommy commented 5 years ago

@DocMAX Did you only need the gvt_uefi.rom or did you also have to patch the OVMF file?

Also were you also able to see the OVMF screen on boot up, or did the screen only initialize after booting into the OS?

DocMAX commented 5 years ago

just the rom is enough. you only see a picture AFTER the guest operating system has initialized the display with the driver.

so this works for gvt-g: -device vfio-pci,sysfsdev=/sys/bus/pci/devices/0000\:00\:02.0/59c57b6c-13ee-4055-846e-e4d3dc55d389,romfile=/opt/vm/bios/vbios/vbios_gvt_uefi.rom,display=on,x-igd-opregion=on

but this (gvt-d) is NOT working (black screen) -device vfio-pci,host=$IOMMU_GPU,romfile=/opt/vm/bios/vbios/vbios_gvt_uefi.rom

special paramaters like x-vga=on dont work either.

HouQiming commented 5 years ago

@DocMAX Can you try GVT-d with a patched OVMF (attached) with neither romfile=foo or rombar=0 on the vfio-pci device? My gut feeling is that GVT-d needs both the OpRegion patch and the original VBIOS. So it won't do to apply the OpRegion patch as a rom file or a kernel patch.

ovmf.zip

DocMAX commented 5 years ago

thank you, but the patched ovmf doesn't work :-(. i'm on a thinkpad x1 carbon 2015 with broadwell cpu by the way.

TerrenceXu commented 5 years ago

@DocMAX, Canyou share your whole boot up GVT-d VM command?

DocMAX commented 5 years ago

Sure, here are my full lines including kernel parameters:

intel_iommu=on iommu=pt

IOMMU_GPU="0000:00:02.0"
IOMMU_GPU_AUDIO="0000:00:1b.0"

vfiobind() {
    modprobe vfio-pci
    for dev in $IOMMU_GPU $IOMMU_GPU_AUDIO; do
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
            echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
    done
}

vfiobind

qemu-system-x86_64
-nodefaults -display none -vga none -parallel none -serial none
-monitor unix:/run/qemu/win-monitor.sock,server,nowait
-cpu host,kvm=off,hv_vendor_id=nvidia43fix -smp $(nproc),sockets=1,cores=4,threads=1 -enable-kvm
-drive file=/opt/vm/qcow2/win.qcow2,if=virtio
-netdev bridge,br=br0,id=net0 -device virtio-net-pci,netdev=net0,mac=52:54:00:00:00:10
-m 4G
-drive if=pflash,format=raw,readonly,file=/opt/vm/bios/ovmf-patched/ovmf.fd
-device vfio-pci,host=$IOMMU_GPU
-device ich9-intel-hda -device hda-duplex
-object input-linux,id=kbd1,evdev=/dev/input/by-path/platform-i8042-serio-0-event-kbd,grab_all=on,repeat=on
-object input-linux,id=mouse1,evdev=/dev/input/by-path/platform-i8042-serio-1-event-mouse
DocMAX commented 5 years ago

OK, so can at least somebody confirm that GVT-d is not yet working with OVMF Bios? Even with the patches?

elmeyer commented 5 years ago

@HouQiming thanks for posting this file; I've made some success with it and a freshly built qemu from git-master using -device vfio-pci-nohotplug,display=on,ramfb=on,... Can you share exactly which patches were applied to build the file you uploaded?

HouQiming commented 5 years ago

I built it with the opregion patch at:

https://bugzilla.tianocore.org/attachment.cgi?id=168

Did you make success using GVT-d or GVT-g? My own attempt at GVT-d also failed like DocMAX.

On Mon, Mar 18, 2019 at 6:01 PM Lars Meyer notifications@github.com wrote:

@HouQiming https://github.com/HouQiming thanks for posting this file; I've made some success with it and a freshly built qemu from git-master using -device vfio-pci-nohotplug,display=on,ramfb=on,... Can you share exactly which patches were applied to build the file you uploaded?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/intel/gvt-linux/issues/23#issuecomment-473844330, or mute the thread https://github.com/notifications/unsubscribe-auth/AHrIF_ggWeoRRkRhr6KfpC3IuAjqD5hlks5vX2PhgaJpZM4S36PI .

elmeyer commented 5 years ago

I'm using GVT-g. Is the kernel patch you posted mainlined in any
version yet? Did you make no modifications to edk2/OVMF itself?

I built it with the opregion patch at:

https://bugzilla.tianocore.org/attachment.cgi?id=168

Did you make success using GVT-d or GVT-g? My own attempt at GVT-d also failed like DocMAX.

On Mon, Mar 18, 2019 at 6:01 PM Lars Meyer notifications@github.com wrote:

@HouQiming https://github.com/HouQiming thanks for posting this file; I've made some success with it and a freshly built qemu from git-master using -device vfio-pci-nohotplug,display=on,ramfb=on,... Can you share exactly which patches were applied to build the file you uploaded?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/intel/gvt-linux/issues/23#issuecomment-473844330, or
mute the thread https://github.com/notifications/unsubscribe-auth/AHrIF_ggWeoRRkRhr6KfpC3IuAjqD5hlks5vX2PhgaJpZM4S36PI .

-- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/intel/gvt-linux/issues/23#issuecomment-473845498

DocMAX commented 5 years ago

with ramfb=on and gvt-g i get seg fault after os switches to its driver

HouQiming commented 5 years ago

@elmeyer the kernel patch is never mainlined. It's more of a temporary debugging patch. The kernel patch, the OVMF patch, and the VBIOS file posted on reddit are all equivalent. You can patch it in any of the three places. In the end I decided to do it in OVMF.

@DocMAX The seg fault you mention is a QEMU bug I fixed a while ago. You can check my fork at https://github.com/HouQiming/qemu/

Edit: for the sake of completeness - I also uploaded my patched OVMF to https://github.com/HouQiming/edk2

DocMAX commented 5 years ago

HouQiming: OK, but i think i will stay with the release from the arch repo. I think it's not worth the effort just to see the boot screen. Hoping the patch will come to mainstream.

DocMAX commented 5 years ago

Is this topic dead?

HouQiming commented 5 years ago

I did some additional survey afterwards. A few sources indicate that you need a good deal of stuff to make native display work (UEFI or BIOS), including opregion passthrough, VBIOS, ISA stuff, etc:

https://openxt.atlassian.net/wiki/spaces/DC/pages/29851668/PCI-DM-HELPER+and+IGD+Passthrough#PCI-DM-HELPERandIGDPassthrough-PCIpass-through:VGAbioscopy

With all the complications and a working VT-g setup, I'm no longer interested in setting up VT-d. Out of curiosity, why do you insist on VT-d?

On Wed, Apr 10, 2019 at 3:52 AM DocMAX notifications@github.com wrote:

Is this topic dead?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/intel/gvt-linux/issues/23#issuecomment-481409859, or mute the thread https://github.com/notifications/unsubscribe-auth/AHrIF39ck356W5cYb474vLTKYLMsb5CDks5vfO-SgaJpZM4S36PI .

DocMAX commented 5 years ago

I insist to get the 2.560 x 1.440 fullscreen resolution. VT-g is limited to 1920x1080.

HouQiming commented 5 years ago

Well, I also fixed the resolution issue myself. I've been using VT-g at native resolution on my Surface Pro for quite a while. You can use my kernel at: https://github.com/HouQiming/linux-gpdwin2

And write your monitor EDID to the gvt_edid file in the iGPU's sysfs before booting the VM. I have a set of patches for the ramfb resolution, too.

If the maintainers are interested, I can submit a patch. Though I brute-forced quite a few things.

DocMAX commented 5 years ago

I'm talking about:

[x1c i915-GVTg_V4_4]# cat description 
low_gm_size: 128MB
high_gm_size: 512MB
fence: 4
resolution: 1920x1200
weight: 4
HouQiming commented 5 years ago

@DocMAX That thing is completely unrelated to the real deal. Just fix the EDID and the resolution fixes itself.

lulinfan commented 5 years ago

@HouQiming, thanks for the ovmf patch, I use it with GVT-g + dmabuf+ramfb, it works. however when I reboot the windows OS, the screen become black, and QEMU print some log:

ramfb_fw_cfg_write: 1024x768 @ 0x7e70d000
ramfb_fw_cfg_write: 262144x196608 @ 0x7e791000
!!!! X64 Exception Type - 0E(#PF - Page-Fault)  CPU Apic ID - 00000000 !!!!
ExceptionData - 0000000000000003  I:0 R:0 U:0 W:1 P:1 PK:0 S:0
RIP  - 000000007FB2B5EA, CS  - 0000000000000038, RFLAGS - 0000000000010206
RAX  - 0000000000000000, RCX - 0000000BFFDE4400, RDX - 000000007E791000
RBX  - 0000003000000000, RSP - 000000007FF31AD0, RBP - 000000007E791000
RSI  - 000000007FF31BB0, RDI - 000000007F000000
R8   - 0000000000000000, R9  - 0000000000030000, R10 - 0000000000000004
R11  - 0000000000000008, R12 - 0000003000000000, R13 - 000000007FB2B801
R14  - 000000007FF31DD0, R15 - 000000007F2DD018
DS   - 0000000000000030, ES  - 0000000000000030, FS  - 0000000000000030
GS   - 0000000000000030, SS  - 0000000000000030
CR0  - 0000000080010033, CR2 - 000000007F000000, CR3 - 000000007FC01000
CR4  - 0000000000000668, CR8 - 0000000000000000
DR0  - 0000000000000000, DR1 - 0000000000000000, DR2 - 0000000000000000
DR3  - 0000000000000000, DR6 - 00000000FFFF0FF0, DR7 - 0000000000000400
GDTR - 000000007FBEEA98 0000000000000047, LDTR - 0000000000000000
IDTR - 000000007F820018 0000000000000FFF,   TR - 0000000000000000
FXSAVE_STATE - 000000007FF31730
!!!! Find image based on IP(0x7FB2B5EA) /home/hqm/edk2/Build/OvmfX64/RELEASE_GCC5/X64/OvmfPkg/QemuRamfbDxe/QemuRamfbDxe/DEBUG/QemuRamfbDxe.dll (ImageBase=000000007FB2A000, EntryPoint=000000007FB2B154) !!!!

I have to kill the QEMU process and restart it. Host: i5-6350HQ CPU, Ubuntu 18.04 with kernel 4.17, qemu 3.1.0. Guest OS: windows7 64bit

HouQiming commented 5 years ago

@lulinfan This problem could have been caused by my other OVMF patch which allows you to set ramfb resolution from QEMU command line. I fixed it in my patched QEMU:

https://github.com/HouQiming/qemu

which locks the ramfb resolution after initialization and rejects invalid resolutions.

lulinfan commented 5 years ago

@HouQiming, I merge your qemu patch "ramfb resolution locking" to qemu3.1.0, but still got black screen when reboot the Windows 10 guest whith a log "ramfb_fw_cfg_write: >4K, change rejected" .. I also tried to download your qemu and compiled it, but i found no ramfb support for vfio-pci...

HouQiming commented 5 years ago

Come to think of it, one needs to lift the lock whenever the guest is rebooted. I'll check later. Did my patch fix the debug messages, though?

My fork was based on an older version of QEMU which uses the old ramfb syntax -device vfio-pci-ramfb,foo instead of -device vfio-pci,ramfb=on,foo.

lulinfan commented 5 years ago

I tried -device vfio-pci-ramfb,foo with your qemu and ovmf download from https://github.com/intel/gvt-linux/issues/23#issuecomment-468899624, still black screen when win10 reboot, and got a "ramfb_fw_cfg_write: >4K, change rejected" log, I have to Ctrl+C to kill the process..

root@ubuntu:/opt# /usr/bin/qemu-system-x86_64 -m 2048  -M pc -name test_guest -hda /opt/lessons/thin/win10_uefi.qcow2 -bios /home/stack/ovmf2.fd -enable-kvm -k en-us -machine kernel_irqchip=on -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -cpu host -device piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -readconfig /home/stack/intel/usb.cfg -device usb-tablet,id=tablet,bus=uhci.0,port=1 -soundhw hda -device vfio-pci-ramfb,sysfsdev=/sys/bus/pci/devices/0000:00:02.0/59c57b6c-13ee-4055-846e-e4d3dc55d389,rombar=0,display=on,x-igd-opregion=on -vga none -display gtk,gl=on

(qemu-system-x86_64:3974): dbind-WARNING **: 20:02:47.524: Error retrieving accessibility bus address: org.freedesktop.DBus.Error.ServiceUnknown: The name org.a11y.Bus was not provided by any .service files
audio: Could not init `oss' audio driver

(qemu-system-x86_64:3974): Gdk-CRITICAL **: 20:02:47.847: gdk_window_get_scale_factor: assertion 'GDK_IS_WINDOW (window)' failed

(qemu-system-x86_64:3974): Gdk-CRITICAL **: 20:02:47.847: gdk_window_get_origin: assertion 'GDK_IS_WINDOW (window)' failed

(qemu-system-x86_64:3974): Gdk-CRITICAL **: 20:02:47.847: gdk_window_get_scale_factor: assertion 'GDK_IS_WINDOW (window)' failed

(qemu-system-x86_64:3974): Gdk-CRITICAL **: 20:02:47.847: gdk_window_get_origin: assertion 'GDK_IS_WINDOW (window)' failed

(qemu-system-x86_64:3974): Gdk-CRITICAL **: 20:02:47.849: gdk_window_get_scale_factor: assertion 'GDK_IS_WINDOW (window)' failed

(qemu-system-x86_64:3974): Gdk-CRITICAL **: 20:02:47.849: gdk_window_get_origin: assertion 'GDK_IS_WINDOW (window)' failed

(qemu-system-x86_64:3974): Gdk-CRITICAL **: 20:02:47.852: gdk_window_get_scale_factor: assertion 'GDK_IS_WINDOW (window)' failed

(qemu-system-x86_64:3974): Gdk-CRITICAL **: 20:02:47.852: gdk_window_get_origin: assertion 'GDK_IS_WINDOW (window)' failed
ramfb_fw_cfg_write: 1280x720 @ 0x7e70d000
ramfb_fw_cfg_write: 327680x-805175296 @ 0x7e791000
ramfb_fw_cfg_write: >4K, change rejected
^Cqemu-system-x86_64: terminating on signal 2
root@ubuntu:/opt# qemu-system-x86_64 --version
QEMU emulator version 2.12.92 (v2.12.0-2554-gf50aebfa6e-dirty)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
root@ubuntu:/home/stack#
HouQiming commented 5 years ago

@lulinfan Just made a new commit that unlocks the ramfb during reset. Guest reboot works for me now.

lulinfan commented 5 years ago

@HouQiming, it work now, thanks

lulinfan commented 5 years ago

I still got strange black screen when I add -smp sockets=1,cores=4,threads=1 or -smp sockets=1,cores=2,threads=2 to start the win10 guest, only works with -cpu host and no -smp,foo , and I tried different combition of sockets/cores/threads, all failed. I can only use 1 CPU in guest..

Host: i5-6350HQ CPU

HouQiming commented 5 years ago

I've been using -cpu host and -smp just fine. Anything on dmesg or qemu output when you get a black screen? Or do your system get slower before it crashes or anything?

On Tue, Apr 30, 2019 at 11:14 AM lulinfan notifications@github.com wrote:

I still got strange black screen when I add -smp sockets=1,cores=4,threads=1 or -smp sockets=1,cores=2,threads=2 to start the win10 guest, only works with -cpu host and no -smp,foo , and I tried different combition of sockets/cores/threads, all failed. I can only use 1 CPU in guest..

Host: i5-6350HQ CPU

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/intel/gvt-linux/issues/23#issuecomment-487809740, or mute the thread https://github.com/notifications/unsubscribe-auth/AB5MQF4SHWEQTYFIHR7WV5LPS62PTANCNFSM4EW7UPEA .

lulinfan commented 5 years ago

@HouQiming, the qemu from your QEMU works, version 3.1.0 that I download from qemu git show me the black screen. no crash and no qmeu out, only black screen, I can see ramfb work during the boot, when vGPU driver load it become black screen.

HouQiming commented 5 years ago

I patched the display subsystem quite heavily, so it could be that.

The most important patch is adding a glBindTexture before the glTexSubImage2D used for ramfb upload. It seems to screw up the driver the frame QEMU switches from ramfb to dmabuf.

On Tue, Apr 30, 2019 at 8:19 PM lulinfan notifications@github.com wrote:

@HouQiming https://github.com/HouQiming, the qemu from your QEMU works, version 3.1.0 that I download from qemu git show me the black screen. no crash and no qmeu out, only black screen, I can see ramfb work during the boot, when vGPU driver load it become black screen.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/intel/gvt-linux/issues/23#issuecomment-487929696, or mute the thread https://github.com/notifications/unsubscribe-auth/AB5MQF3QAOHKDZ5QTT2LZW3PTA2MZANCNFSM4EW7UPEA .

marcel-apf commented 5 years ago

@HouQiming can you please upstream the 'glBindTexture' change? It also fixes some strange artifacts appearing when the Windows guest turns off the monitor or resolution changes. Does it affects the performance somehow (in other words, is there a reason not to do it)?

I can upstream it myself, but I have no OpenGL knowledge and I cannot explain why is not enough to add the binding only in the 'initialization' code.

HouQiming commented 5 years ago

@marcel-apf Can you give me a TL;DR version of "How to submit a patch to QEMU"? Last time I tried to upstream something, I read the wall of text, used git to send an email and got no response at all, which was discouraging.

From my OpenGL experience, I believe the glBindTexture overhead is negligible. But your suspicion is right, only binding during initialization is the performance-correct behavior and the very fact it's needed in the inner loop indicates a bug elsewhere. Most likely there is some unwanted interaction between GL / EGL code paths (e.g. GTK / dmabuf / ... bound a different texture in some unexpected control flow twist), and certain checks are omitted in the display driver (e.g. out-of-bound texture upload not blocked, which can be exploitable).

Since QEMU is more security critical than performance critical. I'd say it's better to call glBindTexture before every texture upload, maybe reset all relevant GL states in every function, just to make sure. If the guest could meaningfully manipulate the GL control flow (e.g. by strategically timed resolution / pixel format changes), a texture bug could lead to a VM breakout.

marcel-apf commented 5 years ago

@HouQiming, in order to up-stream the patch :

  1. Spend some time to explain the exact issue and how the patch solves the problem.
  2. Use git to prepare the patch, the procedure is standard, the same as sending a kernel patch.
  3. Run ./scripts/checkpatch.pl on your patch and ensure no errors are reported.
  4. Run ./scripts/get_maintainer.pl to obtain the specific maintainers list.
  5. Send it to qemu-devel@nongnu.org, cc-ing the maintainers from the above step. Please add me to cc, so you will have at least one feedback :)
HouQiming commented 5 years ago

Thanks for the info! Patch sent.

On Sun, May 5, 2019 at 9:15 PM Marcel Apfelbaum notifications@github.com wrote:

@HouQiming https://github.com/HouQiming regarding sending the path to QEMU:

  1. Use git to prepare the patch, the procedure is standard, the same as sending a kernel patch.
  2. Run ./scripts/checkpatch.pl on your patch and ensure no errors are reported.
  3. Run ./scripts/get_maintainer.pl to obtain the specific maintainers list.
  4. Send it to qemu-devel@nongnu.org, cc-ing the maintainers from the above step. Please add me to cc, so you will have at least one feedback :)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/intel/gvt-linux/issues/23#issuecomment-489425494, or mute the thread https://github.com/notifications/unsubscribe-auth/AB5MQF33GF6KXQJM52SD263PT3MV3ANCNFSM4EW7UPEA .

marcel-apf commented 5 years ago

@HouQiming, appreciated!

lulinfan commented 5 years ago

@HouQiming, I pathed your glBindTexture patch for -smp foo=, the first time I get black screen, but when I try the sencond time to boot the guest, it works.. But now i find a new problem that when use win10 as guest with qemu from your git, the resolution does not fit the monitor display, I can't see a full windows srcren on monitor, the resolution of the monitor, Host ,guest, all of them is at 1920*1080. but for win7 guest, it works perfect.

HouQiming commented 5 years ago

@lulinfan First, my qemu patches have nothing to do with fixing GVT resolution. It needs a kernel patch: https://github.com/HouQiming/gvt-linux/

The patch replaces the default 1920x1200 resolution with a more conventional 1920x1080. But if you want anything else, you need a user-space script that writes your monitor's EDID to the interface exposed under your vGPU.

That said, can you give more details about what exactly is happening to your Win 10 guest? I can't decipher what you said in your post...

lulinfan commented 5 years ago

I upload the picture, I will try your gvt-linux patch later. win10.zip

HouQiming commented 4 years ago

@DocMAX Update: I tried GVT-d a few days ago. Turns out that the OVMF patch works partially for GVT-d, but:

TheCatFelix commented 4 years ago

I was successful too in getting GVT-d to work with OVMF using the "magic" vbios_gvt_uefi.rom file. I've been struggling with this for a week, so here a photo/screenshot as proof for anybody who is going through the same endeavor.

image_from_ios - Copy

adambmedent commented 4 years ago

I can also confirm that the magic rom file made all the difference in the world for me as well.

Now have my HD 630 passed through to a VM running OVMF. Physical display is working 100%!