Need AArch64 / ARM64 versions

[Googulator]

Microsoft is now building Windows 10 and Windows Server for ARM64 machines. To successfully run these in qemu, virtio drivers are needed for the ARM64 platform.

[YanVugenfirer]

It is on our roadmap. In a meanwhile, could you please share how and what versions of Windows Server you try to run on QEMU, so we will align our efforts.

Thanks.

[vrozenfe]

While it is not on our radmap, it is on our wishlist :) I believe we can give it a try as soon as we can run ARM Windows VM image on QEMU.

Best, Vadim.

[sameehj]

Hi @Googulator,

We are currently attempting to run Windows Server on top of qemu and didn't manage to fully boot Windows. In case you made any progress on that can you please share the it with us so we can port the drivers ASAP.

[Googulator]

In qemu v2.11-rc1, Windows 10 version 1709 (build 16299) for "ARM64", as Microsoft calls it, can now install and boot using this UEFI firmware image: https://github.com/Googulator/edk2/releases/tag/AHCI

Extract the contents of the ISO into a 5GB VHDX or other qemu-compatible image format, create a 2nd, empty VHDX of about 50GB, then run qemu like this (change "-smp 8" if you don't want to emulate that many cores): qemu-system-aarch64 -M virt -m 8192 -cpu cortex-a57 -smp 8 -bios QEMU_EFI.fd -device VGA -device nec-usb-xhci -device usb-kbd -device usb-tablet -drive file=50gb.vhdx,id=install,if=none -netdev user,id=network -device virtio-net,netdev=network -device ahci,id=ahci -device ide-hd,drive=install,bus=ahci.0 -drive file=5gb.vhdx,id=dvd,if=none -device ahci,id=ahci2 -device ide-hd,drive=dvd,bus=ahci2.0

Walk through setup as you would on an x86 system.

Probably won't work under KVM on real ARM hardware, as it depends on the emulated VGA framebuffer, which is fundamentally incompatible with KVM. Unfortunately, virtio-gpu can't be used as an alternative because Windows requires runtime graphics capability through UEFI GOP after ExitBootServices(), which means it can't run on PixelBltOnly screens like the one provided by virtio-gpu.

EDIT: 2.11-rc1 is required, earlier versions have some bug related to memory management(?) that cause Windows to blackscreen early during boot.

[Googulator]

Unrelated issue, but the reason why you have to first extract the image to a VHDX is that simply using the ISO as an ide-cd device causes the qemu host process to crash when you press a key at the prompt to boot from the ISO.

[sameehj]

Hi @Googulator ,

Thank you very much for the instructions and the very detailed explanation. I am following the steps you provided in attempt to boot Windows from the vhdx image but I think I am not preparing the vhdx image from the iso appropriately. Can you please shed some light on how you did it? (tools, special configurations, etc..)

[oscarbg]

@Googulator +1 can you post steps to prepare vhdx from ISO.. seems my virtual disk may be not bootable or badly created..

[huoqianyu]

@sameehj , Hello friends, Today I found that a user called "LightBulbFun" in forum "BetaArchive" successfully installed Windows 10 ARM64 usinig an older firmware. You can see this link https://www.betaarchive.com/forum/viewtopic.php?f=62&t=37172&start=50 for more information. By following his guide, I successfully installed Windows 10 build 16299.15.

P.S. The screenshot shows a small "HELLO WORLD" program created by Visual Basic 6, and that may means x86 program can run in ARM64 win10.

[sameehj]

@huoqianyu,

That's great! We'll be trying to follow his instructions and update you on the progress =)

[falkor2k15]

Any news on the KVM incompatibility issue and any efforts that are being taken to get around it? Is it something that will eventually be fixed in Qemu-system-aarch64?

What is the "emulated VGA framebuffer" exactly and why would this cause an incompatibility with KVM? Surely there must be a way around it?

[YanVugenfirer]

As was mentioned in the previous comment by @huoqianyu it is possible to run Windows on QEMU now.

[falkor2k15]

Yes, but it runs too slow, and the KVM acceleration problem is stopping it from running fast on ARM64 hardware. So my question is specifically about the KVM incompatibility: is it a problem with Qemu or KVM or Windows? Any efforts being made to resolve it? Is there a way around it?

[vrozenfe]

Can you give a try to Linux arm64 guest and see if it performs better than Windows?

[falkor2k15]

The ARM64 guests all have graphics turned off using the -nographic option - so I don't think they can be used with VGA output - for example see this: https://wiki.ubuntu.com/ARM64/QEMU

KVM may also work with no graphics, hence the compatibility problem has something to do with the graphics, but it hasn't been discussed anywhere.

[vrozenfe]

Headless configuration should work for Windows Server as well. But we might be able to build ARM64 version of virtio-gpu dod driver to give it a try.

[falkor2k15]

I don't know where to find the Windows Server 2016 on ARM ISO - I think it was given to selected people only. If you could please build a new graphics device to get KVM to work in conjunction with Qemu-System-Aarch64 + Windows 10 on ARM then that would be great - I'm all ears and waiting day-by-day to test this!

[driver1998]

Of cause you can boot Windows without VGA output, but that will only be a black screen, the OS should still be working though. And if drivers for virtio-gpu is available, we may be able to use that.

[falkor2k15]

There are currently no drivers available - they would have to be made for Windows on ARM; however, that would not be a solution, but a workaround to the problem of no VGA support in Qemu coupled with KVM compatibility issues. I can only assume this is some kind of deliberate joke to put out headless server scripts and hold back on allowing the public to visually experience modern hardware virtualization. A blind man should be grateful just to walk and talk - even though he cannot see where he's going?

[Bullseye300]

For those who have problems by installing QEMU. I found a ready to run image with Windows pre-installed: https://virtuallyfun.com/wordpress/2018/03/22/windows-10-arm64-on-qemu/

I think that needs to be added in the start script (Driver still missing):

-net nic,model=virtio ^ -net user ^

[YanVugenfirer]

You are welcomed to check the branch: https://github.com/virtio-win/kvm-guest-drivers-windows/tree/arm64

Instructions for build machine (using community edition of VS2017): https://github.com/virtio-win/kvm-guest-drivers-windows/wiki/Building-the-drivers#coming-soon-building-virtio-win-drivers-including-arm64-using-visual-studio-2017

Soon will be integrated to master and we will distribute the binaries.

Best regards, Yan.

[falkor2k15]

Thanks Yan! Is there a pre-built binary I could use to test with qemu-system-aarch64/KVM?

[ybendito]

@falkor2k15 I can send built binaries (signed with test signature). Which ones you want to try? (This is not an official build with complete tree, so I'll pick up ones you're interested in)

[falkor2k15]

@ybendito, yes please!! :) Is there a VGA one you could send me as that's the main problem with the qemu-system-aarch64 and KVM. For example; is there a VGA QXL? -vga qxl And how about this one? -net nic,model=virtio Apparently, this ARM64 stuff is meant to be quite easy to build using the Latest Visual Studio (2017): https://youtu.be/vdYIaUeZnqc?t=1047 However, I'm not a dev, so I wouldn't know. Finally, if you do manage to build at least a VGA driver then how might we import them into a qemu-system-aarch64 script? Thanks and looking forward to it!

My email is gilius2k15@gmail.com

I'm not sure if you guys are aware of this new QEMU machine (xilinx-zynq-a9) and new ARM CPU (A72)? https://www.xilinx.com/support/documentation/sw_manuals/xilinx2017_4/ug1169-xilinx-qemu.pdf

Would be good to test with a53, a57 and a72.

Edit: I don't think the QXL one is a virtio driver. I think it's just called -vga virtio or -virtio-vga or virtio-gpu

[ybendito]

@falkor2k15 qxl-wddm-dod is not a part of virtio-win, we did not build it for ARMx and currently we do not have an estimation for that. virtio-net driver is available for trial: https://drive.google.com/open?id=1fBBysiREXap4KSG2ImaY5IUp_yLIDt2x

[falkor2k15]

Looking promising... Just need to try to install this windows so I can get to the desktop - might need to switch to Linux host... Qemu doesn't like my i7.

[ybendito]

@falkor2k15 I think you play with some drivers set that is not related to our build, I've sent only netkvm.

[driver1998]

I am testing Windows ARM64 on KVM, could you please send me the viostor bits? @ybendito

[ybendito]

@driver1998 On which platform(chipset)? I do not think you can use viostor at Windows setup, for that the driver shall be signed by Microsoft. But you can add virtio disk later if you enable test signature. https://drive.google.com/open?id=1FxOxFoONccJYdTaUnVs_YQ145O_FePZF

[driver1998]

BCM2837 @ Raspberry Pi 3B, not the best choice of course, but it does work and it is the only thing I have. You can use DISM to install Windows manually then inject drivers (which allows testsign drivers), thus bypass Windows Setup.

[falkor2k15]

That block level driver was custom made a long time ago - I just added the NetKVM to my existing drivers disk.

This is not looking good though - I can't seem to get a working Internet setup using -net nic,model=virtio:

Edit: I got "vlan 0 is not connected to host network", so I will try -net nic,model=virtio,vlan=1 maybe? Edit: apparently, the -net nic has been depreciated, so I need to look at netdev switches instead. Looks like it's a toss up between these two: -netdev user,id=n0 -device virtio-net,netdev=n0 -netdev user,id=n0 -device virtio-net-pci,netdev=n0

[ybendito]

@falkor2k15 With yellow bang on the device / driver it is hard to get it working. What is the error code in the device manager? Did you enable test signing?

[falkor2k15]

@ybendito it says the driver is unsigned and I can't get a network adapter to appear using various switches.

How to enable test signing? Bcdedit.exe -set TESTSIGNING ON Is it just this?

[ybendito]

@falkor2k15 yes, + reboot

[falkor2k15]

Looking much better now: network discovery appeared; exclamation mark gone; adapter appeared; now have an IP address, but no Internet for the some reason But I did get this error when I tried to troubleshoot the connection: http://wiki-errors.com/wiki-errors.php?wiki=0x800B0101 Also, the sfc /scannow could not fix everything with this version. BTW, I'm using this version of QEMU from May 2018: https://qemu.weilnetz.de/w64/2018/qemu-w64-setup-20180519.exe (all later versions have a “ACPI BIOS ERROR” using the custom EFI/VGA firmware)

We are also restricted to earlier versions of Windows 10 on ARM because of some other issue with HAL or some such: Edit: will try to create a new user profile.

Edit: I set a static IP and it works!!!

So now I need to find out if it's possible to RDP/VNC to this guest (from Linux) and whether KVM works with nographics...? That will be really difficult to test - will need to use Gentoo or OpenSUSE on the Raspberry Pi. I don't understand whether the VGA issue with KVM would affect any kind of graphics output or just from displaying directly through QEMU? Perhaps RDPing is not going to help.

Is there definitely no alternative VGA driver we could try in the virtio suite besides the aforementioned QXL?

Edit: I got RDP to work; here's my full script:

qemu-system-aarch64.exe ^ -accel tcg,thread=multi ^ -M virt ^ -cpu cortex-a57 ^ -smp 3 ^ -m 4G ^ -pflash QEMU_EFI.img ^ -pflash QEMU_VARS.img ^ -device VGA ^ -device nec-usb-xhci ^ -device usb-kbd ^ -device usb-mouse ^ -device usb-tablet ^ -device virtio-blk,drive=system ^ -drive if=none,id=system,format=raw,file=system.vhdx ^ -netdev user,hostfwd=tcp::5555-:3389,id=n0 -device virtio-net,netdev=n0

You have to allow blank passwords over RDP via secpol.msc: Open Local Policies -> Security Options Change Accounts: Limit local account use of blank passwords to console from Enabled to Disabled

You can then RDP from Windows (intel) to ARM guest using 127.0.0.1:5555

[falkor2k15]

I managed to get the Windows 10 on ARM guest to run without VGA output and was able to RDP into it still, so now I'm preparing for the KVM test... (there's a rumour that KVM works only with VNC and not RDP; am unsure)

These three lines will need to be changed from: -cpu cortex-a57 ^ -device VGA ^ -accel tcg,thread=multi ^

To: -cpu cortex-a53 ^ OR -cpu host -nographic ^ -cpu host -enable-kvm ^

If that works - will be a miracle! :)

Somehow, though, I doubt the gatekeepers would have made it that easy for us... ;)

EDIT: Seems to work in principle, but there was a lot of QEMU-KVM errors with version 1.11. Am trying Qemu version 3, but this has a different issue based on the old custom UEFI firmware where we get an "ACPI Bios Error" BSOD. There's no KVM errors, so maybe we just need a new QEMU_EFI.img and QEMU_VARS.img - whether it be from Qemu's own EFI (if working) or from Tianocore: https://github.com/tianocore/edk2 Prebuilt binaries here maybe? How to use them? http://snapshots.linaro.org/components/kernel/leg-virt-tianocore-edk2-upstream/latest/QEMU-AARCH64/

[driver1998]

@falkor2k15 Here is the configuration I use for KVM: Raspberry Pi 3B, ArchLinuxARM qemu-3.0.0 release source, no changes, compiled myself latest Linaro UEFI snapshot (the link you provided) So far I can get WinPE to boot, but KVM crashes when trying to boot a Windows install on a harddisk image. (Tried HDD through ich9-usb-ehci, nec-usb-xhci and virtio, same thing)

traces if you want:

[alarm@alarm ~]$ qemu-system-aarch64 -M virt-2.12 -accel kvm -cpu cortex-a53 -smp 2 -m 768 -drive file=/mnt/usb/hdd.vhd,id=hdd,if=none -bios ~/QEMU_EFI_1.fd -device ramfb -device ich9-usb-ehci1 -device usb-kbd -device usb-storage,drive=hdd -vnc :0
Could not open option rom 'vgabios-ramfb.bin': No such file or directory
ramfb_fw_cfg_write: 800x600 @ 0x6bcec000
ramfb_fw_cfg_write: 800x600 @ 0x6bcec000
error: kvm run failed Function not implemented
PC=fffff80151b52840  SP=ffffd388881edff0
X00=ffffc30f771f37d0 X01=0000000000000004 X02=0000000000000001 X03=fffff80151b530e8
X04=0000000000000018 X05=0000000000000000 X06=000000000000001c X07=0000000000000080
X08=ffffac80360da000 X09=0000000000000100 X10=ffffd388881edf98 X11=0000000000000000
X12=ffffd388881edf98 X13=ffffc30f77180094 X14=ffffc30f77180088 X15=000000000000001c
X16=00007b289058984f X17=ffff585ff7c64971 X18=fffff801b4b10000 X19=0000000000000000
X20=ffffc30f771f37d0 X21=00000000000001e0 X22=0000000000000000 X23=0000000000000000
X24=ffffc30f771f31a0 X25=ffffc30f771f3020 X26=ffffd388881ee0e0 X27=0000000000000000
X28=0000000000000000 X29=ffffd388881edff0 X30=fffff80151b51ebc
PSTATE=60000044 -ZC- EL1t

If you use -device ramfb instead of -device VGA, you can get a display output on newer, unmodified UEFI builds.

@ybendito By the way, your viostor build is working (tested on softmmu only).

[falkor2k15]

@driver1998 , that's really helpful regarding the ramfb device!!! :) Cheers for that... I got the display working with the latest linaro bios (even made the 2 flash files to save EFI settings) + ramfb.

Those errors are the same KVM errors I got because you are using virt-2.12. I think they are fixed in the latest virt (3) to match the version of Qemu, but I am still getting the ACPI BIOS BSOD even with the latest firmware when using -virt-3.0.

[driver1998]

@falkor2k15 By the way, what is the hardware of your host?

[falkor2k15]

I am currently testing Win10 i7-8700K side-by-side with Raspberry Pi 3b+ running Gentoo ARM64 with a KVM-enabled kernel.

I just got passed the ACPI bios error using virt-2.9 so need to test that next with KVM enabled.

[falkor2k15]

Same problem: One day we'll get a WOA KVM - just not this Christmas... now that we got the ramfb driver though - could always test some ARM64 Linux distros to see if they also suffer from KVM errors or not.

Anyway, at least the virtio drivers are working!

[mariobalanica]

WinPE 10 build 17134 running under QEMU KVM on the Raspberry Pi 3 B (Linux 4.19.4-1-ARCH):

It seems that only USB EHCI controllers work (usb-ehci and ich9-usb-ehci1). xHCI controllers (nec-usb-xhci/qemu-xhci) will cause Windows to hang on the spinning dots.

[falkor2k15]

@mariobalanica, were you able to get full Windows 10 w/ KVM running yet using the right USB controller? I've already eliminated -no-acpi (causes a different problem) and turning off hibernation powercfg /h off (hasn't helped either) but then it did go on to complain about the disk driver.

[falkor2k15]

Does anyone know the correct commands for EHCI to be able to install windows? This doesn't work: -device usb-ehci,id=ehci ^ -device usb-storage,bus=ehci.0,drive=usbstick ^ -drive if=none,id=usbstick,file=system.vhdx ^

Same problem with: -device ich9-usb-ehci1,id=ehci1 ^ -device usb-storage,bus=ehci1.0,drive=usbstick ^ -drive if=none,id=usbstick,file=system.vhdx ^

EDIT: seems we have to use DISM first to get past this initial hurdle? EDIT 2: seems we need to keep it as a virtio block device, but just change the controller. There's also USB mass storage device drivers for ARM64, but I cannot find them anywhere.

[falkor2k15]

As @driver1998 said it's the same KVM issue with all the USB controllers. However, since WinPE boots, we should be able to test out portable apps like the new VLC for Windows 10 on ARM and see how it performs on Pi hardware:

[driver1998]

It seems that KVM always crash at around here (build 17134):

0: kd> g
Breakpoint 0 hit
pci!PciProgramMsiXInterrupt+0xc4:
fffff802`5ace8fbc b8004548 str         w8,[x10],#4
0: kd> t

KVM crashes right after I step over this instruction...

Here is what dmesg said...

[12454.505444] kvm [2037]: load/store instruction decoding not implemented

[ybendito]

@driver1998 Do you run it on ARM? If yes, looks like platform or qemu/KVM problem with MSIX. Do you have other PCI devices with MSIX (they usually have negative IRQ)? You can try disabling the msix for pci-virtionet device in the qemu command line.

[driver1998]

@ybendito, yes, and it only happens when I am booting a full install instead of a PE image.

By the way, how to "disable msix"?

[driver1998]

I normally have these devices in my commandline: ramfb for display ich9-usb-ehci1 with usb-kbd and usb-tablet viostor

[ybendito]

@driver1998 (disable MSIX) I hope 'vectors=0' as virtio-net parameter can disable it

[driver1998]

@ybendito Tried that, didn't work.