search

utmapp / UTM

Virtual machines for iOS and macOS
https://getutm.app
Apache License 2.0
26.98k stars 1.35k forks source link

UTM with HVF on Intel causes BSOD in Windows guests (IRQL_NOT_LESS_OR_EQUAL) #2368

Closed conath closed 3 years ago

conath commented 3 years ago

When running a VM with -accel hvf, a Windows guest will reproducibly crash with a blue screen error IRQL_NOT_LESS_OR_EQUAL. This issue duplicates #2303.

In the case of a Windows 7 and 8.1 guest, the BSOD is confirmed to occur during the first phase of setup.

In the case of a Windows 10 guest, the BSOD occurs repeatably during the first phase of setup or, if the OS is already installed, at least once after the OOBE.

Configuration

Workaround: before installing Windows in the VM, in UTM Preferences, enable "Force slower emulation even when hypervisor is available". Then start the VM and install Windows (it will take about twice as long as expected). After Windows is installed, shut down the VM and disable "Force slower emulation even when hypervisor is available".

conath commented 3 years ago

Unfortunately, the workaround mentioned is only a temporary fix. Even with a fully installed Windows guest, as soon as certain (as of yet unknown) actions are made in the guest OS, it crashes to BSOD again. For example, in the following screen recording I attempt to install the latest VirtIO Guest Tools in my Windows 10 VM, running with HVF.

https://user-images.githubusercontent.com/12073163/110122533-0a019200-7dc0-11eb-93f9-5a4c9c386ee5.mov

osy commented 3 years ago

@conath can you try this build with updated QEMU? https://github.com/utmapp/UTM/actions/runs/634437375

conath commented 3 years ago

A Windows 7 x86 install in a x86_64 q35 machine worked using HVF and that build. Going to try a Windows 10 x64 install next.

conath commented 3 years ago

Windows 10 20H2 x64 install crashed with a new BSOD KMODE_EXCEPTION_NOT_HANDLED.

QEMU arguments 
qemu-system-x86_64 -L /…/qemu -S -qmp tcp:127.0.0.1:4444,server,nowait -vga none -spice port…off
-device qxl-vga -cpu host -smp cpus=4,sockets=1,cores=4,threads=1 -machine q35, -accel hvf -accel tcg,tb-size=1024 -global PIIX4_PM.disable_s3=1 -global ICH9-LPC.disable_s3=1 -boot order=d -m 4096 -name "Win 10 x64" -device usb-ehci -device usb-tablet -device usb-mouse -device usb-kbd -device virtio-blk-pci,drive=drive0,bootindex=0 -drive "if=none,media=disk,id=drive0,file=/…/Win 10 x64.utm/Images/disk-0.qcow2,cache=writethrough" -drive if=none,media=cdrom,id=drive1 -nic none -uuid 87E35857-5001-4E66-B2E9-1068E964BBD0 -rtc base=localtime 
-device ide-drive,drive=drive1
conath commented 3 years ago

And booting with AHCI as drive interface (via custom arguments) crashes with IRQL_NOT_LESS_OR_EQUAL while booting the installer or while installing (not the same every time).

-device ahci,id=ahci
-device ide-hd,drive=drive0,bus=ahci.0
-device ide-cd,drive=drive1,bus=ahci.1
conath commented 3 years ago

Windows XP install crashes in the text mode part when it loads the HAL. The stop code 0x000000a5 points to an ACPI issue and the first parameter (0x000000011) means "The system cannot enter ACPI mode".

(0x000000011, Parameter2, Parameter3, Parameter4):

The system cannot enter ACPI mode. There are many > reasons for this, including: ▪ The system cannot initialize the AML interpreter. ▪ The system cannot find the Root System > Description table. ▪ The system cannot allocate a critical driver. ▪ The system cannot load the Root System > Description table. ▪ The system cannot load device descriptor blocks. ▪ The system cannot connect an interrupt vector. ▪ The SCI_EN (system control interrupt enable > request) cannot be set (see 0x00000001). ▪ The ACPI Table checksum is incorrect. ACPI is a hierarchical arrangement of tables, each one > building upon the next to define the complete capabilities > of the system and of every device in the system. ACPI > starts by looking for the Root System Description table, > which points to the next table, which points to the next > table, and so on. Usually, the 0x000000011 error occurs > because these tables are damaged or missing.

Trying the suggestion at the top of the linked support article, which suggests disabling the ACPI driver by pressing F7 before the drivers load, causes the VM to reboot immediately when it does start loading the drivers.

conath commented 3 years ago

The 0x000000A5 (0x…11, …) happens with the pc (1996) system. When using the q35 system, I get a different "parameter 1": 0x000000A5 (0x…02, …), indicating an ACPI root PCI resource failure.

conath commented 3 years ago

During more recent testing with Windows Server 2016 and Windows XP Pro X64 Edition, I have discovered that using the q35 system version 2.9 or earlier gets rid of the BSODs! It seems that the newer q35 systems combined with the Hypervisor causes the issue.

Specifically, when installing XP and faced with the stopcode 0x000000A5, switching to q35-2.9 results in successful install (if you have AHCI drivers loaded, otherwise you get stopcode 7B due to inaccessible boot drive).

conath commented 3 years ago

The same BSOD DRIVER IRQL LESS OR EQUAL still occurs with q35 version 6.0 and Windows 11 (tested the leaked ISO 21996.1.210529-1541.co_release_CLIENT_CONSUMER_x64FRE_en-us.iso).

Windows11BSOD

conath commented 3 years ago

I have made a discovery: the BSOD IRQL_NOT_LESS_OR_EQUAL seems to coincide with the VM losing cursor focus. Just now, I moved my mouse out of the VM window and it BSOD'd. Not sure if it's a coincidence or a cause of the crash, but I tried reinstalling Windows while not moving my cursor out of the Window and it didn't crash.

Update: I can reproduce this reliably now. @osy any ideas why this might be happening or is this qemu/hvf territory?

Update 2: it seems like it also crashes if the mouse focused in the VM and not moved for a while.

Update 3: it just crashed while I was interacting with it, so no, this mouse movement and focus behavior is purely coincidental.

osy commented 3 years ago

@conath Is this still an issue on latest version? Or is #2721 preventing this from being tested?

conath commented 3 years ago

Yes, same BSOD still occurs as of UTM 2.2.4 running pc-q35-6.1 on Intel MacBook Air with macOS 11.5.2, guest is Windows 21H1 X64 installer.

mpolden commented 3 years ago

Same issue here on same config as above comment, but a MacBook Pro instead of Air.

Are there any known workarounds? I just needed a Windows VM briefly and thought I might give UTM a shot, but it doesn't appear to work at all.

conath commented 3 years ago

@mpolden Unfortunately the only known workaround is to disable use of the Hypervisor, which means reducing the speed by at least half:

Workaround: before installing Windows in the VM, in UTM Preferences, enable "Force slower emulation even when hypervisor is available". Then start the VM and install Windows (it will take about twice as long as expected).

nicholasngai commented 3 years ago

Hi! I’ve been running QEMU directly from the CLI without using UTM and have been getting this issue regularly, as well.

I narrowed it down to -cpu flag I was using, and it turns out that using -cpu host was causing my issues. I switched to -cpu Cascadelake-Server (according to the recommendation to use the CPU matching your host CPU's generation) while keeping -accel hvf, and it works without any BSOD.

Does UTM default to -cpu host behavior? If it does, could you try switching it to some pre-defined QEMU CPU and see if the crashes still occur?

osy commented 3 years ago

That’s interesting. Yes it uses host by default but you can change the cpu type in the settings.

conath commented 3 years ago

I can confirm that changing from default to another choice (I used Skylake-Client) gets rid of the BSOD, but retains fast HVF performance. 😃

osy commented 3 years ago

Reopening since the workaround caused #3249

osy commented 3 years ago

WinDbg log of crash on Windows 11 startup.

0: kd> !analyze -v
Connected to Windows 10 22000 x64 target at (Fri Nov  5 15:35:39.605 2021 (UTC - 7:00)), ptr64 TRUE
Loading Kernel Symbols
.........

Press ctrl-c (cdb, kd, ntsd) or ctrl-break (windbg) to abort symbol loads that take too long.
Run !sym noisy before .reload to track down problems loading symbols.

......................................................
................................................................
............................
Loading User Symbols
...........

Press ctrl-c (cdb, kd, ntsd) or ctrl-break (windbg) to abort symbol loads that take too long.
Run !sym noisy before .reload to track down problems loading symbols.

.....................................................
.........
Loading unloaded module list
.............
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

KMODE_EXCEPTION_NOT_HANDLED (1e)
This is a very common bugcheck.  Usually the exception address pinpoints
the driver/function that caused the problem.  Always note this address
as well as the link date of the driver/image that contains this address.
Arguments:
Arg1: ffffffffc0000005, The exception code that was not handled
Arg2: fffff80632a30819, The address that the exception occurred at
Arg3: 0000000000000000, Parameter 0 of the exception
Arg4: ffffcf88055022df, Parameter 1 of the exception

Debugging Details:
------------------

Debugger extension ext.analyze execution is cancelled.
      PC: 00007ffd`4e57466c  VA: 00000032`2ba79305  R/W: 9  Parameter: 00000000`00000005
0: kd> kb
 # RetAddr               : Args to Child                                                           : Call Site
00 fffff806`32b63572     : fffff806`30c4afd0 fffff806`32802390 fffff806`30bda180 00000000`00000000 : nt!DbgBreakPointWithStatus
01 fffff806`32b62db1     : fffff806`00000003 fffff806`30c4afd0 fffff806`32a28710 00000000`0000001e : nt!KiBugCheckDebugBreak+0x12
02 fffff806`32a12c77     : fffff806`32600000 fffff806`30c4c668 fffff806`30c4c668 fffff806`32903cf9 : nt!KeBugCheck2+0xa71
03 fffff806`32b491fa     : 00000000`0000001e ffffffff`c0000005 fffff806`32a30819 00000000`00000000 : nt!KeBugCheckEx+0x107
04 fffff806`32a1c30f     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!HvlpVtlCallExceptionHandler+0x22
05 fffff806`329030e7     : fffff806`30c4bc60 fffff806`32a1a905 fffff806`30c3fd00 fffff806`30c4b760 : nt!RtlpExecuteHandlerForException+0xf
06 fffff806`32907031     : ffffffff`ffffffff fffff806`30c4c710 fffff806`30c4c710 fffff806`30c4be80 : nt!RtlDispatchException+0x2d7
07 fffff806`32a257ce     : 00000000`00000000 fffff806`30c4c630 fffff806`30c4c7b0 fffff806`32a257ce : nt!KiDispatchException+0x1b1
08 fffff806`32a217da     : 00000000`00000000 00000000`00000000 00000000`00000000 fffff806`32a2138f : nt!KiExceptionDispatch+0x10e
09 fffff806`32a30819     : 00000000`00000000 00000000`00000000 00000000`00000000 fffff806`30c4cf70 : nt!KiPageFault+0x41a
0a fffff806`329d6951     : fffff806`00000001 00000000`00000022 fffff806`30c3a000 fffff806`30c40000 : nt!ExpCenturyDpcRoutine$fin$0+0x35
0b fffff806`32a1c38f     : 00000000`00000022 fffff806`30c4cf00 fffff806`30c3fa60 fffff806`30c4ca00 : nt!_C_specific_handler+0x1a1
0c fffff806`3290367b     : 00000000`00000000 fffff806`30c3fa60 fffff806`329718bf fffff806`326df0ec : nt!RtlpExecuteHandlerForUnwind+0xf
0d fffff806`329d6895     : fffff806`326972ec fffff806`00000001 fffff806`30c3fa60 fffff806`30c40000 : nt!RtlUnwindEx+0x2db
0e fffff806`32a1c30f     : fffff806`30c3f738 fffff806`30c4d6e0 fffff806`30c4d1e0 fffff806`30c4d1e0 : nt!_C_specific_handler+0xe5
0f fffff806`329030e7     : fffff806`30c4d6e0 fffff806`329718bf fffff806`30c3fa30 fffff806`30c4d1e0 : nt!RtlpExecuteHandlerForException+0xf
10 fffff806`32907031     : ffffffff`ffffffff fffff806`30c3f7e0 fffff806`30c3f7e0 fffff806`30c4d900 : nt!RtlDispatchException+0x2d7
11 fffff806`32a13a82     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiDispatchException+0x1b1
12 fffff806`32a13a50     : fffff806`32a257c7 ffffcf88`2d1f3040 00000000`00000000 00000000`00000000 : nt!KxExceptionDispatchOnExceptionStack+0x12
13 fffff806`32a257c7     : ffffcf88`2d1f3040 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiExceptionDispatchOnExceptionStackContinue
14 fffff806`32a2138f     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiExceptionDispatch+0x107
15 fffff806`32a1d69d     : 00000000`00000202 fffff806`3285b42d fffff806`30bda180 ffffa781`3f69f1b8 : nt!KiGeneralProtectionFault+0x30f
16 fffff806`32a1d5dd     : ffffcf88`2d1f3040 00000000`00000006 00000000`00000004 fffff806`00000000 : nt!KiCustomRecurseRoutine2+0xd
17 fffff806`32a1db1d     : 00000000`00000000 00000000`00000000 00000000`00000000 ffffcf88`28226200 : nt!KiCustomRecurseRoutine1+0xd
18 fffff806`32a1da5d     : ffffcf88`25ac8b90 ffffcf88`2d1f3040 00000000`00000000 00000000`00000000 : nt!KiCustomRecurseRoutine0+0xd
19 fffff806`32a1da92     : 00000000`00000000 ffffcf88`2d1f3180 00000000`00000001 00000000`00000000 : nt!KiCustomRecurseRoutine9+0xd
1a fffff806`329718bf     : fffff806`30c3faf9 fffff806`328de492 fffff806`30bda180 00000000`00000002 : nt!KiCustomAccessRoutine9+0x22
1b fffff806`328dcc74     : 00000000`00000006 85a6c038`dd955577 ffffcf88`25aa9422 fffff806`30bda180 : nt!ExpCenturyDpcRoutine+0x9f
1c fffff806`328db264     : fffff806`30bda180 00000000`0000000f 00000000`4dbe64c3 fffff806`30be1088 : nt!KiProcessExpiredTimerList+0x204
1d fffff806`32a1a905     : 00000000`00000000 fffff806`30bda180 fffff806`332f7bd0 ffffcb88`574928e0 : nt!KiRetireDpcList+0x714
1e fffff806`32a1a6e0     : ffffffff`ffffffff fffff806`3292b07a 00000000`00000002 ffffcb88`57492590 : nt!KxRetireDpcList+0x5
1f fffff806`32a19f25     : ffffcb88`574928e0 fffff806`32a14dc1 00000000`00002710 00000000`00210131 : nt!KiDispatchInterruptContinue
20 fffff806`32a14dc1     : 00000000`00002710 00000000`00210131 fffff7c2`4000e598 fffff806`328d393f : nt!KiDpcInterruptBypass+0x25
21 fffff806`32950d1a     : fffff806`30bda180 ffffffff`ffffffff 00000000`00000000 ffffcb88`574928e0 : nt!KiInterruptDispatchNoLockNoEtw+0xb1
22 fffff806`32912137     : fffff806`30bda180 ffffcb88`574927f0 ffffcb88`574928e0 ffffcf88`2d9e8080 : nt!ExpUpdateTimerConfigurationWorker+0xaa
23 fffff806`3291283e     : 00000000`00000001 00000000`00000001 ffffcf88`2d8c4d00 ffffcb88`57492a88 : nt!KeGenericProcessorCallback+0x12b
24 fffff806`32913153     : 00000000`00000000 ffffcf88`2d97d080 ffff3c56`42722cfc ffffcf88`2d97d080 : nt!ExpUpdateTimerConfiguration+0x9a
25 fffff806`32d6aa34     : ffffcf88`2d97d080 00000000`00002710 00000000`944d0c01 000008c4`000a4a40 : nt!ExpUpdateTimerResolution+0x133
26 fffff806`32a25075     : ffffcf88`2d9e8080 000008c4`0007caf0 000008c4`000af400 00000000`00000000 : nt!NtSetTimerResolution+0x104
27 00007ff9`cf1e6a04     : 00007ff9`ce5abdcb 00000000`00000000 00000000`00000001 0000005d`28fff1f0 : nt!KiSystemServiceCopyEnd+0x25
28 00007ff9`ce5abdcb     : 00000000`00000000 00000000`00000001 0000005d`28fff1f0 000008c4`0007caf0 : ntdll!NtSetTimerResolution+0x14
29 00007ff9`9b449c9b     : aaaaaaaa`00002710 00000000`00000001 00000000`00000001 0000005d`28fff1f0 : KERNEL32!timeBeginPeriod+0xcb
2a 00007ff9`9b86843c     : 00000000`07c65b40 aaaaaaaa`aaaaaaaa 00000000`00003345 00000000`00000000 : msedge!Ordinal0+0x1449c9b
2b 00007ff9`9b865d0c     : 0000005d`28fff360 aaaaaaaa`aaaaaaaa 000008c4`000a1e00 00007ff9`a3befe40 : msedge!Ordinal0+0x186843c
2c 00007ff9`9c227c05     : 00000000`00000000 0000005d`28fff5f0 0000005d`28fff6f8 00007ff9`9b4d908b : msedge!Ordinal0+0x1865d0c
2d 00007ff9`9a04cb6e     : 000008c4`00048e78 00000000`00000000 00000000`00000000 0000be3a`0c38d78e : msedge!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x87da5
2e 00007ff9`9f685908     : 0000005d`28fff6f8 00007ff9`9b4d908b 0000005d`28fff5e8 00000000`00000000 : msedge!Ordinal0+0x4cb6e
2f 00007ff9`9cedab04     : 00000000`00000000 00000000`00000000 0000005d`28fff650 00007ff9`9c7523da : msedge!GetMainTargetServices+0x2447938
30 00007ff9`9c7521f0     : 0000005d`28fff6b0 00000000`0000000b 00007ff9`a4a8d3b8 00007ff9`9bcb8970 : msedge!ChromeMain+0x1be94
31 00007ff9`9cefbc9c     : 000008c4`00000191 00000000`00000001 00000000`00000000 00000000`00000000 : msedge!RelaunchChromeBrowserWithNewCommandLineIfNeeded+0x5b2390
32 00007ff9`ce5a54e0     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : msedge!ChromeMain+0x3d02c
33 00007ff9`cf14485b     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : KERNEL32!BaseThreadInitThunk+0x10
34 00000000`00000000     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x2b
osy commented 3 years ago

Suspect that PatchGuard is in play: https://www.codeproject.com/Articles/28318/Bypassing-PatchGuard-3

osy commented 3 years ago

Last log seemed to have been incorrect (PatchGuard was triggered independent of the issue). This I believe is more accurate:

*** Fatal System Error: 0x0000000a
                       (0xFFFFA70BDA1651EF,0x0000000000000002,0x0000000000000000,0xFFFFF80364E30CCD)

Break instruction exception - code 80000003 (first chance)

A fatal system error has occurred.
Debugger entered on first try; Bugcheck callbacks have not been invoked.

A fatal system error has occurred.

For analysis of this file, run !analyze -v
nt!DbgBreakPointWithStatus:
fffff803`64e1b1d0 cc              int     3
kd> !analyze -v
Connected to Windows 10 22000 x64 target at (Fri Nov  5 18:58:06.256 2021 (UTC - 7:00)), ptr64 TRUE
Loading Kernel Symbols
.......................

Press ctrl-c (cdb, kd, ntsd) or ctrl-break (windbg) to abort symbol loads that take too long.
Run !sym noisy before .reload to track down problems loading symbols.

........................................
................................................................
...........................
Loading User Symbols
........................

Press ctrl-c (cdb, kd, ntsd) or ctrl-break (windbg) to abort symbol loads that take too long.
Run !sym noisy before .reload to track down problems loading symbols.

........
Loading unloaded module list
........
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

IRQL_NOT_LESS_OR_EQUAL (a)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high.  This is usually
caused by drivers using improper addresses.
If a kernel debugger is available get the stack backtrace.
Arguments:
Arg1: ffffa70bda1651ef, memory referenced
Arg2: 0000000000000002, IRQL
Arg3: 0000000000000000, bitfield :
    bit 0 : value 0 = read operation, 1 = write operation
    bit 3 : value 0 = not an execute operation, 1 = execute operation (only on chips which support this level of status)
Arg4: fffff80364e30ccd, address which referenced memory

Debugging Details:
------------------

KEY_VALUES_STRING: 1

    Key  : Analysis.CPU.mSec
    Value: 4421

    Key  : Analysis.DebugAnalysisManager
    Value: Create

    Key  : Analysis.Elapsed.mSec
    Value: 590561

    Key  : Analysis.Init.CPU.mSec
    Value: 5968

    Key  : Analysis.Init.Elapsed.mSec
    Value: 162895

    Key  : Analysis.Memory.CommitPeak.Mb
    Value: 124

    Key  : WER.OS.Branch
    Value: co_release

    Key  : WER.OS.Timestamp
    Value: 2021-06-04T16:28:00Z

    Key  : WER.OS.Version
    Value: 10.0.22000.1

BUGCHECK_CODE:  a

BUGCHECK_P1: ffffa70bda1651ef

BUGCHECK_P2: 2

BUGCHECK_P3: 0

BUGCHECK_P4: fffff80364e30ccd

READ_ADDRESS:  ffffa70bda1651ef Nonpaged pool

PROCESS_NAME:  MoUsoCoreWorker.exe

TRAP_FRAME:  369dd70000000300 -- (.trap 0x369dd70000000300)
Unable to read trap frame at 369dd700`00000300

STACK_TEXT:  
fffff803`6176e578 fffff803`64f63572     : fffff803`6176e6e0 fffff803`64c02390 fffff803`616fc180 00000000`00000000 : nt!DbgBreakPointWithStatus
fffff803`6176e580 fffff803`64f62db1     : fffff803`00000003 fffff803`6176e6e0 fffff803`64e28710 00000000`0000000a : nt!KiBugCheckDebugBreak+0x12
fffff803`6176e5e0 fffff803`64e12c77     : 00000000`00000000 00000000`00000000 fffff803`64a9edec 00000000`00000000 : nt!KeBugCheck2+0xa71
fffff803`6176ed50 fffff803`64e256a9     : 00000000`0000000a ffffa70b`da1651ef 00000000`00000002 00000000`00000000 : nt!KeBugCheckEx+0x107
fffff803`6176ed90 fffff803`64e21800     : 000004f0`fffffb30 000004d0`fffffb30 00000000`00000021 00000000`00000000 : nt!KiBugCheckDispatch+0x69
fffff803`6176eed0 fffff803`64e30ccd     : fffff803`6176f900 fffff803`64d435a5 fffff803`61761850 fffff803`64e1d571 : nt!KiPageFault+0x440
fffff803`6176f060 fffff803`64dd684f     : fffff803`00000001 fffff803`64a9ee1c fffff803`6175c000 fffff803`61762000 : nt!IopTimerDispatch$filt$2+0x4d
fffff803`6176f0a0 fffff803`64e0e6ba     : fffff803`64a9ee1c fffff803`61761638 fffff803`617618d0 fffff803`6176f140 : nt!_C_specific_handler+0x9f
fffff803`6176f110 fffff803`64e1c30f     : fffff803`61761638 fffff803`6176f6e0 fffff803`6176f1e0 fffff803`64d98b40 : nt!_GSHandlerCheck_SEH+0x6a
fffff803`6176f140 fffff803`64d030e7     : fffff803`6176f6e0 fffff803`64d98b40 fffff803`617618a0 fffff803`6176f1e0 : nt!RtlpExecuteHandlerForException+0xf
fffff803`6176f170 fffff803`64d07031     : ffffffff`ffffffff fffff803`617616e0 fffff803`617616e0 fffff803`6176f900 : nt!RtlDispatchException+0x2d7
fffff803`6176f8d0 fffff803`64e13a82     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiDispatchException+0x1b1
fffff803`6176ffb0 fffff803`64e13a50     : fffff803`64e257c7 ffffa50b`ea2e72f0 00000000`7f0f0176 ffffa50b`ded17ac4 : nt!KxExceptionDispatchOnExceptionStack+0x12
fffff803`617614f8 fffff803`64e257c7     : ffffa50b`ea2e72f0 00000000`7f0f0176 ffffa50b`ded17ac4 00000000`00000001 : nt!KiExceptionDispatchOnExceptionStackContinue
fffff803`61761500 fffff803`64e2138f     : 00000001`00000002 000000e8`00000001 00000000`00000001 fffff803`61761778 : nt!KiExceptionDispatch+0x107
fffff803`617616e0 fffff803`64e1d5dd     : ffffa50b`ded17b74 ffffa50c`06bd4c50 ffffa50b`da6fa170 ffffa50b`fc0e4290 : nt!KiGeneralProtectionFault+0x30f
fffff803`61761870 fffff803`64e1d612     : 00000000`00000000 00000000`00000000 00000000`00000000 ffffa50b`dad7b1a0 : nt!KiCustomRecurseRoutine1+0xd
fffff803`617618a0 fffff803`64d98b40     : fffff803`64f0da70 00000000`000262ce 00000000`00000000 00000000`00000000 : nt!KiCustomAccessRoutine1+0x22
fffff803`617618d0 fffff803`64de3f2b     : ffffa50b`dabfd1a0 00000000`00400a02 fffff803`64d98a70 00000000`01d7d2ec : nt!IopTimerDispatch+0xd0
fffff803`61761ad0 fffff803`64cdbdf1     : fffff803`61761e90 fffff803`61761e00 fffff803`616ff4c0 fffff803`00000002 : nt!KiSwInterruptDispatch+0xffb
fffff803`61761b00 fffff803`64cdadf2     : fffff803`00000000 00000000`00000000 00000000`26b1c859 fffff803`61703808 : nt!KiExecuteAllDpcs+0x491
fffff803`61761d00 fffff803`64e1a905     : 00000000`00000000 fffff803`616fc180 fffff803`656f7cf0 00000000`000000d0 : nt!KiRetireDpcList+0x2a2
fffff803`61761fb0 fffff803`64e1a6e0     : 00000000`0000062c fffff803`64d2b07a 00000000`00000000 ffff840c`369dd770 : nt!KxRetireDpcList+0x5
fffffd0d`fc00d430 fffff803`64e19f25     : 00000000`000000d0 fffff803`64e14dc1 00000000`011000d0 00000000`00000000 : nt!KiDispatchInterruptContinue
fffffd0d`fc00d460 fffff803`64e14dc1     : 00000000`011000d0 00000000`00000000 00000000`00000000 ffffc55c`b88a7ab7 : nt!KiDpcInterruptBypass+0x25
fffffd0d`fc00d470 fffff803`64ca584d     : 00000000`0000000c 00000000`00000000 369dd700`00000300 ffff840c`360089b4 : nt!KiInterruptDispatchNoLockNoEtw+0xb1
fffffd0d`fc00d600 fffff803`64c2f6e0     : ffff840c`2fc00380 00000000`000000ff 00000000`00001420 00000000`00000000 : nt!RtlpHpLfhSubsegmentFreeBlock+0x1bd
fffffd0d`fc00d680 fffff803`6546b819     : 01000000`00000000 00000000`00000000 fffffd0d`fc00d7a0 ffff840c`36008920 : nt!ExFreeHeapPool+0x340
fffffd0d`fc00d730 fffff803`650c54a9     : 00000000`00000000 fffffd0d`fc00d7a0 00000000`0000000f 00000000`00000010 : nt!ExFreePool+0x9
fffffd0d`fc00d760 fffff803`650c5dc8     : 00000000`0000007c ffff840c`369dd770 ffff840c`36008920 00000000`63416553 : nt!ObSetSecurityDescriptorInfo+0x129
fffffd0d`fc00d7d0 fffff803`650bb749     : ffff840c`369dd770 00000000`00000000 ffff840c`3041fda0 fffff803`650bbbd6 : nt!SeDefaultObjectMethod+0x308
fffffd0d`fc00d850 fffff803`650bb12c     : ffff840c`36008920 ffff840c`00000004 00000000`00000002 ffff840c`36008950 : nt!ObSetSecurityObjectByPointer+0x89
fffffd0d`fc00d8b0 fffff803`6514ba8e     : ffff840c`368b7060 00000000`00000002 fffffd0d`fc00db00 00000000`00000000 : nt!SepAppendAceToTokenObjectAcl+0x24c
fffffd0d`fc00d980 fffff803`64e25075     : ffffa50b`dc5dd080 00000045`ceafcf38 00000000`00000000 00000000`00000000 : nt!NtDuplicateToken+0x1fe
fffffd0d`fc00da70 00007fff`d9843b14     : 00007fff`d97e8e00 00000000`00000001 00000045`ceafd020 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x25
00000045`ceafcf18 00007fff`d97e8e00     : 00000000`00000001 00000045`ceafd020 00000000`00000000 00000000`00000000 : ntdll!NtDuplicateToken+0x14
00000045`ceafcf20 00007fff`d72da266     : 00000000`00000000 00000245`00000000 00000045`ceafd244 00007fff`d97c8b4a : ntdll!RtlCheckTokenMembershipEx+0x2c0
00000045`ceafd1b0 00007fff`c358a823     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000005 : KERNELBASE!CheckTokenMembership+0x26
00000045`ceafd1e0 00007fff`c3587fa7     : 00007fff`c3589ef0 00000045`ceafd398 00000245`601d8760 00000045`ceafd398 : wuapi!SecurityCheck+0xe3
00000045`ceafd2f0 00007fff`c3587f55     : 00007fff`c3589ef0 00007fff`c3589ef0 00000000`00000000 00007fff`c3580000 : wuapi!CUpdateDownloadContent::FinalConstruct+0x17
00000045`ceafd320 00007fff`c3576158     : 00000045`ceafd398 00000000`8007000e 00000245`601d8760 00000245`601d8760 : wuapi!ATL::CComObject<CUpdateDownloadContent>::CreateInstance+0x6d
00000045`ceafd370 00007fff`c35745bc     : 00000000`00000000 00000000`0000000b 00000000`00002857 00000245`601d98e8 : wuapi!CUpdateDownloadContentCollection::InitializeFromSerializer+0x7c
00000045`ceafd3d0 00007fff`c3576b06     : 00000245`5e74dbb0 00000000`00000000 00000000`00000000 00000245`5e74dbb0 : wuapi!CUpdate::DeserializeUpdate+0x53c
00000045`ceafd690 00007fff`c3583b89     : 00000245`5e74dbb0 00000045`ceafd778 00000245`5e74dbb0 00000245`5e74dbb0 : wuapi!CUpdate::Initialize+0x1a
00000045`ceafd6e0 00007fff`c3583af1     : 00000245`5e74dbb0 00000000`00000000 00000045`ceafd7b9 00000245`5f2c9048 : wuapi!CUpdateDeserializer::DeserializeUpdateInternal+0x75
00000045`ceafd710 00007ff7`e499fe90     : 00000245`5e74be70 00000245`5e8c9f40 00000045`ceafd7b9 00000245`5e8fe8c0 : wuapi!CUpdateDeserializer::DeserializeUpdate+0x41
00000045`ceafd740 00007ff7`e48db522     : 00000245`5e74be70 00000045`ceafdb49 00000245`5e8c9f30 00000245`5e74be70 : MoUsoCoreWorker!Windows::WU::WuProvider::Deserialize+0xd0
00000045`ceafd820 00007ff7`e48dc486     : 00000045`ceafdbd8 00000045`ceafdbd8 00000000`00000000 00000000`00000000 : MoUsoCoreWorker!UpdateManager::UpdateManager+0x952
00000045`ceafdab0 00007ff7`e49d7dd7     : 00000245`5df130b8 00000245`5df130b8 00000000`00000000 00000245`5df0b8f0 : MoUsoCoreWorker!UpdateManager::GetUpdateManager+0x22a
00000045`ceafdbb0 00007ff7`e48d1853     : 00000045`ceafdcd8 00000000`8007000e 00000245`5df0b8f0 00000045`ceafe050 : MoUsoCoreWorker!TaggedUpdateManager::Initialize+0xf7
00000045`ceafdc90 00007fff`d86823e3     : 00000000`00000004 00000045`ceafdce0 00000045`ceafe190 00000245`5df0b8f0 : MoUsoCoreWorker!MoUsoCoreWorker::GetUpdateManager+0xa3
00000045`ceafdcc0 00007fff`d86eb68c     : 00000045`ceafe180 00000245`5df31620 00000245`5df0b950 00000045`ceafe170 : RPCRT4!Invoke+0x73
00000045`ceafdd20 00007fff`d864e5b2     : 00000000`00000000 00000000`00000000 00000000`00000000 00007fff`d9207445 : RPCRT4!Ndr64StubWorker+0xb7c
00000045`ceafe3c0 00007fff`d9206155     : 00000000`00000000 00000045`ceafe5f0 00007fff`d3426770 00000245`5df0b950 : RPCRT4!NdrStubCall3+0xd2
00000045`ceafe420 00007fff`d86640a5     : 00000000`00000001 00000245`5df0b950 00000245`5defeef0 00000000`00000000 : combase!CStdStubBuffer_Invoke+0x65 [onecore\com\combase\ndr\ndrole\stub.cxx @ 1493] 
00000045`ceafe460 00007fff`d91ab17d     : 00000245`5defeef0 00000245`5df0e890 00005c1f`7f9c3cfc 00000000`00000000 : RPCRT4!CStdStubBuffer_Invoke+0x45
00000045`ceafe490 00007fff`d91aaf17     : 00000000`00000000 00000045`ceafe580 00000045`ceafe538 00000000`02000002 : combase!ObjectMethodExceptionHandlingAction<<lambda_c9f3956a20c9da92a64affc24fdd69ec> >+0x4d [onecore\com\combase\dcomrem\excepn.hxx @ 94] 
00000045`ceafe4f0 00007fff`d920b1b8     : 00000000`00000000 00000000`00000080 00000245`5df10b58 00000000`00000000 : combase!DefaultStubInvoke+0x257 [onecore\com\combase\dcomrem\channelb.cxx @ 1228] 
00000045`ceafe640 00007fff`d91f533b     : 00000000`00000000 00000045`ceafe760 00000045`ceafe718 00000000`00000000 : combase!SyncServerCall::StubInvoke+0x38 [onecore\com\combase\dcomrem\ServerCall.hpp @ 787] 
00000045`ceafe680 00007fff`d916ca71     : 00000000`00000001 00000245`5df1a2f0 00000245`5df1a440 00007fff`d91f6f63 : combase!ServerCall::ContextInvoke+0x46b [onecore\com\combase\dcomrem\ctxchnl.cxx @ 1425] 
00000045`ceafe960 00007fff`d91eb273     : 00000245`5df1a2f0 00000045`ceafea10 00000000`00000000 00000245`5df1a690 : combase!DefaultInvokeInApartment+0xc1 [onecore\com\combase\dcomrem\callctrl.cxx @ 3299] 
00000045`ceafe9b0 00007fff`d91e8999     : 00000045`ceaff000 00000245`5df19e80 00000001`00000001 00000245`5df1ba40 : combase!ComInvokeWithLockAndIPID+0xa13 [onecore\com\combase\dcomrem\channelb.cxx @ 2241] 
00000045`ceafeca0 00007fff`d8662c72     : 00000245`5df19e80 00000000`00000000 00000045`ceafefd4 00000245`5df1bb90 : combase!ThreadInvoke+0x1b9 [onecore\com\combase\dcomrem\channelb.cxx @ 7303] 
00000045`ceafeda0 00007fff`d862748f     : 00000045`ceafef00 00007fff`d91dc185 00000045`ceafefd4 00000245`5df19e80 : RPCRT4!DispatchToStubInCNoAvrf+0x22
00000045`ceafedf0 00007fff`d8627098     : 00000245`5df1b550 00000000`00000001 00000000`00000001 00000001`00000003 : RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0x1af
00000045`ceafeed0 00007fff`d86374b5     : 00000245`5df1b550 00000045`ceaff080 00000245`5df1ba40 00000000`00000000 : RPCRT4!RPC_INTERFACE::DispatchToStubWithObject+0x188
00000045`ceafef70 00007fff`d8636b07     : 00000000`0006fbbd 00000000`00000001 00000000`00000000 00000245`5df31620 : RPCRT4!LRPC_SCALL::DispatchRequest+0x175
00000045`ceaff040 00007fff`d863618b     : 00000000`00000000 00000245`5df12f80 00000245`00000000 00000000`00000000 : RPCRT4!LRPC_SCALL::HandleRequest+0x837
00000045`ceaff140 00007fff`d8635e61     : 00000000`00000000 00004f69`00000000 00000000`00000000 00000000`00000000 : RPCRT4!LRPC_SASSOCIATION::HandleRequest+0x24b
00000045`ceaff1c0 00007fff`d8635a97     : 00000245`5df39960 00000045`ceaff360 00000000`00000001 00000245`5df12f80 : RPCRT4!LRPC_ADDRESS::HandleRequest+0x181
00000045`ceaff260 00007fff`d863c179     : 00000000`00000000 00000245`5df31620 00000245`5df39a68 00000045`ceaff618 : RPCRT4!LRPC_ADDRESS::ProcessIO+0x897
00000045`ceaff3a0 00007fff`d97c1f90     : 00000000`00000000 00000045`ceaff438 00000045`ceaff618 00000000`00000000 : RPCRT4!LrpcIoComplete+0xc9
00000045`ceaff430 00007fff`d97b6c78     : 00000000`00000000 00000245`5df39b00 00000000`00000000 00000245`5df182d0 : ntdll!TppAlpcpExecuteCallback+0x280
00000045`ceaff4b0 00007fff`d88a54e0     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!TppWorkerThread+0x448
00000045`ceaff7a0 00007fff`d97a485b     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : KERNEL32!BaseThreadInitThunk+0x10
00000045`ceaff7d0 00000000`00000000     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x2b

SYMBOL_NAME:  nt!IopTimerDispatch$filt$2+4d

MODULE_NAME: nt

IMAGE_NAME:  ntkrnlmp.exe

STACK_COMMAND:  .thread ; .cxr ; kb

BUCKET_ID_FUNC_OFFSET:  4d

FAILURE_BUCKET_ID:  AV_nt!IopTimerDispatch$filt$2

OS_VERSION:  10.0.22000.1

BUILDLAB_STR:  co_release

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 10

FAILURE_ID_HASH:  {08052ca9-343b-0cad-b4c4-00f146cc0b1a}

Followup:     MachineOwner
---------
osy commented 3 years ago

So the crash is happening in PatchGuard handling but it's not a CRITICAL_STRUCTURE_CORRUPTION, so it's not PatchGuard integrity check failing but PatchGuard itself triggering a pagefault. It is reproducible with QEMU command line with HVF, so best guess is that there's a bug in HVF causing the failure.

osy commented 3 years ago

@conath I "fixed" this by removing the -cpu host from x86 hosts and I've tested it by running through Windows 10 setup twice with no issues. You can check as well.