search

basil00 / Divert

WinDivert: Windows Packet Divert
https://reqrypt.org/windivert.html
Other
2.32k stars 491 forks source link

DRIVER_IRQL_NOT_LESS_OR_EQUAL when handling IOCTL_WINDIVERT_STARTUP (null pointer access?) #330

Closed elmeyer closed 1 year ago

elmeyer commented 1 year ago

Hi, a customer using WinDivert 2.2.2 has provided a memory dump after encountering a BSOD with DRIVER_IRQL_NOT_LESS_OR_EQUAL. I have analyzed it in WinDbg and can provide the following results:

2: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high.  This is usually
caused by drivers using improper addresses.
If kernel debugger is available get stack backtrace.
Arguments:
Arg1: 0000000000000008, memory referenced
Arg2: 0000000000000002, IRQL
Arg3: 0000000000000000, value 0 = read operation, 1 = write operation
Arg4: fffff80d9ab8ab69, address which referenced memory

Debugging Details:
------------------

Page 5a00 not present in the dump file. Type ".hh dbgerr004" for details
Page 5a00 not present in the dump file. Type ".hh dbgerr004" for details
Page 5a00 not present in the dump file. Type ".hh dbgerr004" for details

KEY_VALUES_STRING: 1

    Key  : Analysis.CPU.mSec
    Value: 1359

    Key  : Analysis.DebugAnalysisManager
    Value: Create

    Key  : Analysis.Elapsed.mSec
    Value: 3167

    Key  : Analysis.IO.Other.Mb
    Value: 0

    Key  : Analysis.IO.Read.Mb
    Value: 2

    Key  : Analysis.IO.Write.Mb
    Value: 3

    Key  : Analysis.Init.CPU.mSec
    Value: 23578

    Key  : Analysis.Init.Elapsed.mSec
    Value: 4566727

    Key  : Analysis.Memory.CommitPeak.Mb
    Value: 144

    Key  : Bugcheck.Code.DumpHeader
    Value: 0xd1

    Key  : Bugcheck.Code.KiBugCheckData
    Value: 0xd1

    Key  : Bugcheck.Code.Register
    Value: 0xa

    Key  : WER.OS.Branch
    Value: rs1_release

    Key  : WER.OS.Timestamp
    Value: 2022-11-03T17:03:00Z

    Key  : WER.OS.Version
    Value: 10.0.14393.5501

FILE_IN_CAB:  MEMORY.DMP

BUGCHECK_CODE:  d1

BUGCHECK_P1: 8

BUGCHECK_P2: 2

BUGCHECK_P3: 0

BUGCHECK_P4: fffff80d9ab8ab69

READ_ADDRESS:  0000000000000008 

PROCESS_NAME:  ngs-pulsar-amd64.exe

TRAP_FRAME:  ffff93014fd39970 -- (.trap 0xffff93014fd39970)
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=0000000000000000 rbx=0000000000000000 rcx=fffff80d9ab92150
rdx=0000000000000000 rsi=0000000000000000 rdi=0000000000000000
rip=fffff80d9ab8ab69 rsp=ffff93014fd39b00 rbp=ffff93014fd39bc1
 r8=ffff93014fd39b30  r9=00000000000017a5 r10=fffff8010cc2ac40
r11=00000000000017a5 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei ng nz na pe nc
WinDivert64+0xab69:
fffff80d`9ab8ab69 48394808        cmp     qword ptr [rax+8],rcx ds:00000000`00000008=????????????????
Resetting default scope

STACK_TEXT:  
ffff9301`4fd39828 fffff801`0c577da9     : 00000000`0000000a 00000000`00000008 00000000`00000002 00000000`00000000 : nt!KeBugCheckEx
ffff9301`4fd39830 fffff801`0c5749e6     : ffff830e`a4f73330 ffff830e`a4f73330 00000000`0000000c fffff801`0c43d95b : nt!KiBugCheckDispatch+0x69
ffff9301`4fd39970 fffff80d`9ab8ab69     : ffff830e`b637a9c0 00350038`00330031 ffff8b00`00000036 00000000`00000018 : nt!KiPageFault+0x426
ffff9301`4fd39b00 fffff80d`9ab88c15     : 00000000`000000f0 00000000`00328b98 ffff9301`4fd39bc1 00000000`00000001 : WinDivert64+0xab69
ffff9301`4fd39b60 fffff80d`94e0b537     : ffff830e`d35ff750 00007cf1`2ca008a8 00000000`0012e489 00000000`00000000 : WinDivert64+0x8c15
ffff9301`4fd39c20 fffff80d`94e0a7aa     : ffff80e4`603e6a01 ffff80c0`72301f00 ffff80c0`60391808 fffff801`0c5fa9cd : Wdf01000!FxIoQueue::DispatchRequestToDriver+0x1b7 [d:\rs1\minkernel\wdf\framework\shared\irphandlers\io\fxioqueue.cpp @ 3322] 
ffff9301`4fd39cc0 fffff80d`94e062a2     : ffff8b00`1afa5c20 fffff80d`94e0a000 00000000`00000000 fffff80d`94e1c101 : Wdf01000!FxIoQueue::DispatchEvents+0x3aa [d:\rs1\minkernel\wdf\framework\shared\irphandlers\io\fxioqueue.cpp @ 3122] 
ffff9301`4fd39d90 fffff80d`94e027c9     : 00007cf1`2ca00800 00007cf1`2ca008a8 ffff830e`d35ff750 00000000`00000001 : Wdf01000!FxPkgIo::EnqueueRequest+0x362 [d:\rs1\minkernel\wdf\framework\shared\irphandlers\io\fxpkgio.cpp @ 697] 
ffff9301`4fd39e50 fffff80d`9ab84b7e     : 00000000`00000000 ffff830e`b74c7bf0 00000000`00000000 ffff830e`d35ff750 : Wdf01000!imp_WdfDeviceEnqueueRequest+0xc9 [d:\rs1\minkernel\wdf\framework\shared\core\fxdeviceapi.cpp @ 2003] 
ffff9301`4fd39eb0 fffff80d`94e05e4f     : ffff8b00`1afa5c20 fffff801`0c5faa50 ffff8b0f`d5fb25e8 019000bf`12d20867 : WinDivert64+0x4b7e
ffff9301`4fd39fa0 fffff80d`94e03a9b     : 0000000f`ffffff00 ffff8b0f`f1dad8f0 7fffffff`00000001 ffff830e`d35ff750 : Wdf01000!FxPkgIo::DispatchStep1+0x66f [d:\rs1\minkernel\wdf\framework\shared\irphandlers\io\fxpkgio.cpp @ 324] 
ffff9301`4fd3a060 fffff801`0c81a610     : 00000000`00000000 ffff9301`4fd3a4c0 00000000`00000000 ffff8b0f`00000000 : Wdf01000!FxDevice::DispatchWithLock+0x6fb [d:\rs1\minkernel\wdf\framework\shared\core\fxdevice.cpp @ 1430] 
ffff9301`4fd3a150 fffff801`0c8199b1     : ffff830e`bdc99ca0 ffff830e`a60326c0 ffff8b0f`f1dad8f0 ffff9301`4fd3a4c0 : nt!IopSynchronousServiceTail+0x1a0
ffff9301`4fd3a210 fffff801`0c818bb6     : ffff8b0f`d45b7d00 00000000`000012fc 00000000`00000000 00000057`75dff8d0 : nt!IopXxxControlFile+0xdf1
ffff9301`4fd3a360 fffff801`0c577603     : ffff80c0`602fff30 ffff80c0`603017f8 ffff31a8`725ce0ec 00000057`75dff9e0 : nt!NtDeviceIoControlFile+0x56
ffff9301`4fd3a3d0 00007ff9`ae285d74     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13
00000057`75dff758 00000000`00000000     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x00007ff9`ae285d74

SYMBOL_NAME:  WinDivert64+ab69

MODULE_NAME: WinDivert64

IMAGE_NAME:  WinDivert64.sys

IMAGE_VERSION:  2.0.0.0

STACK_COMMAND:  .cxr; .ecxr ; kb

BUCKET_ID_FUNC_OFFSET:  ab69

FAILURE_BUCKET_ID:  AV_WinDivert64!unknown_function

OS_VERSION:  10.0.14393.5501

BUILDLAB_STR:  rs1_release

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 10

FAILURE_ID_HASH:  {d6231627-20e0-32f9-9ad0-bdd7e0f5058a}

Followup:     MachineOwner
---------

Arg1 is what makes me think that this is a null pointer access.

Showing call parameters with the CodeMachine Kernel Debugger Extension DLL results in this output:

2: kd> !stack -p
Call Stack : 17 frames
## Stack-Pointer    Return-Address   Call-Site       
00 ffff93014fd39828 fffff8010c577da9 nt!KeBugCheckEx+0 
    Parameter[0] = 000000000000000a
    Parameter[1] = 0000000000000008
    Parameter[2] = 0000000000000002
    Parameter[3] = 0000000000000000
01 ffff93014fd39830 fffff8010c5749e6 nt!KiBugCheckDispatch+69 
    Parameter[0] = (unknown)       
    Parameter[1] = (unknown)       
    Parameter[2] = (unknown)       
    Parameter[3] = (unknown)       
02 ffff93014fd39970 fffff80d9ab8ab69 nt!KiPageFault+426 
    Parameter[0] = fffff80d9ab92150
    Parameter[1] = 0000000000000000
    Parameter[2] = ffff93014fd39b30
    Parameter[3] = 00000000000017a5
03 ffff93014fd39b00 fffff80d9ab88c15 WinDivert64+ab69 
    Parameter[0] = ffff830eb637a9c0
    Parameter[1] = (unknown)       
    Parameter[2] = (unknown)       
    Parameter[3] = (unknown)       
04 ffff93014fd39b60 fffff80d94e0b537 WinDivert64+8c15 
    Parameter[0] = 000074ffe505a3d8
    Parameter[1] = 00007cf12ca008a8
    Parameter[2] = 0000000000000018
    Parameter[3] = 0000000000000010
05 ffff93014fd39c20 fffff80d94e0a7aa Wdf01000!FxIoQueue::DispatchRequestToDriver+1b7 (perf)
    Parameter[0] = ffff8b001afa5c20
    Parameter[1] = ffff830ed35ff750
    Parameter[2] = (unknown)       
    Parameter[3] = (unknown)       
06 ffff93014fd39cc0 fffff80d94e062a2 Wdf01000!FxIoQueue::DispatchEvents+3aa (perf)
    Parameter[0] = (unknown)       
    Parameter[1] = (unknown)       
    Parameter[2] = (unknown)       
    Parameter[3] = (unknown)       
07 ffff93014fd39d90 fffff80d94e027c9 Wdf01000!FxPkgIo::EnqueueRequest+362 (perf)
    Parameter[0] = 00007cf12ca008a8
    Parameter[1] = ffff830eb74c7bf0
    Parameter[2] = ffff830ed35ff750
    Parameter[3] = (unknown)       
08 ffff93014fd39e50 fffff80d9ab84b7e Wdf01000!imp_WdfDeviceEnqueueRequest+c9 (perf)
    Parameter[0] = (unknown)       
    Parameter[1] = 0000000000000000
    Parameter[2] = 00007cf12ca008a8
    Parameter[3] = (unknown)       
09 ffff93014fd39eb0 fffff80d94e05e4f WinDivert64+4b7e 
    Parameter[0] = 0000000000000000
    Parameter[1] = 00007cf12ca008a8
    Parameter[2] = (unknown)       
    Parameter[3] = (unknown)       
0a ffff93014fd39fa0 fffff80d94e03a9b Wdf01000!FxPkgIo::DispatchStep1+66f (perf)
    Parameter[0] = ffff8b0fd5fb2460
    Parameter[1] = ffff8b0ff1dad8f0
    Parameter[2] = ffff8b0fd5fb25e8
    Parameter[3] = (unknown)       
0b ffff93014fd3a060 fffff8010c81a610 Wdf01000!FxDevice::DispatchWithLock+6fb (perf)
    Parameter[0] = ffff8b0019721e20
    Parameter[1] = ffff8b0ff1dad8f0
    Parameter[2] = (unknown)       
    Parameter[3] = (unknown)       
0c ffff93014fd3a150 fffff8010c8199b1 nt!IopSynchronousServiceTail+1a0 (perf)
    Parameter[0] = ffff8b0019721e20
    Parameter[1] = ffff8b0ff1dad8f0
    Parameter[2] = ffff830ebdc99ca0
    Parameter[3] = ffff830ea1c5f800
0d ffff93014fd3a210 fffff8010c818bb6 nt!IopXxxControlFile+df1 
    Parameter[0] = ffff830ebdc99ca0
    Parameter[1] = (unknown)       
    Parameter[2] = (unknown)       
    Parameter[3] = (unknown)       
0e ffff93014fd3a360 fffff8010c577603 nt!NtDeviceIoControlFile+56 
    Parameter[0] = (unknown)       
    Parameter[1] = (unknown)       
    Parameter[2] = (unknown)       
    Parameter[3] = (unknown)       
0f ffff93014fd3a3d0 00007ff9ae285d74 nt!KiSystemServiceCopyEnd+13 
    Parameter[0] = 00000000000012c8
    Parameter[1] = 00000000000012fc
    Parameter[2] = 0000000000000000
    Parameter[3] = 0000000000000000

The WDFREQUEST being dispatched is an IOCTL with the following parameters:

2: kd> !irp 0xffff8b0ff1dad8f0
Irp is active with 1 stacks 1 is current (= 0xffff8b0ff1dad9c0)
 Mdl=ffff830ea60326c0: System buffer=ffff830eb7385fc0: Thread ffff830ea1c5f800:  Irp stack trace.  
     cmd  flg cl Device   File     Completion-Context
>[IRP_MJ_DEVICE_CONTROL(e), N/A(0)]
            5  1 ffff8b0019721e20 ffff830ebdc99ca0 00000000-00000000    pending
           \Driver\WinDivert
            Args: 00000018 00000010 0x12e489 00000000
2: kd> !ioctldecode 0x12e489

Unknown IOCTL  : 0x12e489 

Device Type    : 0x12 (FILE_DEVICE_NETWORK)
Method         : 0x1 METHOD_IN_DIRECT 
Access         : FILE_READ_ACCESS FILE_WRITE_ACCESS 
Function       : 0x922

As per https://github.com/basil00/Divert/blob/v2.2.2/include/windivert_device.h#L309, the function is IOCTL_WINDIVERT_STARTUP. The only place I can see that makes sense for the function call at index 03 is windivert_reflect_open_event from https://github.com/basil00/Divert/blob/v2.2.2/sys/windivert.c#L3298.

I have extracted the context_s object pointed to from the parameter given to what is presumably windivert_reflect_open_event, and dumped it quick and dirty in Go (apologies for the nonexistent formatting):

{state:177 lock:0 device:0x7cf148b38408 object:0x7cf149c85708 process:0xffff8b0fd45b7800 flow_set:{Flink:0xffff830eb637a9e8 Blink:0xffff830eb637a9e8} flow_v4_callout_id:0 flow_v6_callout_id:0 work_queue:{Flink:0xffff830eb637aa00 Blink:0xffff830eb637aa00} packet_queue:{Flink:0xffff830eb637aa10 Blink:0xffff830eb637aa10} work_queue_length:0 packet_queue_length:0 packet_queue_maxlength:4096 packet_queue_size:0 packet_queue_maxsize:4194304 packet_queue_maxcounts:6624000 packet_queue_maxtime:2000 read_queue:0x7cf155e17fd8 worker:0x74f00fbe21d8 layer:0 flags:1 initialized:1 shutdown_recv:0 shutdown_send:0 shutdown_recv_enabled:1 priority:2162622465 priority16:2999 callout_guid:[{Data1:322597353 Data2:29879 Data3:4589 Data4:[146 1 239 196 186 149 13 199]} {Data1:322597355 Data2:29879 Data3:4589 Data4:[146 1 239 196 186 149 13 199]} {Data1:322597357 Data2:29879 Data3:4589 Data4:[146 1 239 196 186 149 13 199]} {Data1:322597359 Data2:29879 Data3:4589 Data4:[146 1 239 196 186 149 13 199]} {Data1:322597361 Data2:29879 Data3:4589 Data4:[146 1 239 196 186 149 13 199]} {Data1:322597363 Data2:29879 Data3:4589 Data4:[146 1 239 196 186 149 13 199]} {Data1:322597365 Data2:29879 Data3:4589 Data4:[146 1 239 196 186 149 13 199]} {Data1:322597367 Data2:29879 Data3:4589 Data4:[146 1 239 196 186 149 13 199]} {Data1:322597369 Data2:29879 Data3:4589 Data4:[146 1 239 196 186 149 13 199]} {Data1:322597371 Data2:29879 Data3:4589 Data4:[146 1 239 196 186 149 13 199]} {Data1:322597373 Data2:29879 Data3:4589 Data4:[146 1 239 196 186 149 13 199]} {Data1:322597375 Data2:29879 Data3:4589 Data4:[146 1 239 196 186 149 13 199]}] filter_guid:[{Data1:322597354 Data2:29879 Data3:4589 Data4:[146 1 239 196 186 149 13 199]} {Data1:322597356 Data2:29879 Data3:4589 Data4:[146 1 239 196 186 149 13 199]} {Data1:322597358 Data2:29879 Data3:4589 Data4:[146 1 239 196 186 149 13 199]} {Data1:322597360 Data2:29879 Data3:4589 Data4:[146 1 239 196 186 149 13 199]} {Data1:322597362 Data2:29879 Data3:4589 Data4:[146 1 239 196 186 149 13 199]} {Data1:322597364 Data2:29879 Data3:4589 Data4:[146 1 239 196 186 149 13 199]} {Data1:322597366 Data2:29879 Data3:4589 Data4:[146 1 239 196 186 149 13 199]} {Data1:322597368 Data2:29879 Data3:4589 Data4:[146 1 239 196 186 149 13 199]} {Data1:322597370 Data2:29879 Data3:4589 Data4:[146 1 239 196 186 149 13 199]} {Data1:322597372 Data2:29879 Data3:4589 Data4:[146 1 239 196 186 149 13 199]} {Data1:322597374 Data2:29879 Data3:4589 Data4:[146 1 239 196 186 149 13 199]} {Data1:322597376 Data2:29879 Data3:4589 Data4:[146 1 239 196 186 149 13 199]}] installed:[0 0 0 0 0 0 0 0 0 0 0 0] engine_handle:0xffff830ea1e3ac20 filter:0xffff830ea00f3880 filter_len:1 filter_flags:240 reflect:{entry:{Flink:<nil> Blink:<nil>} timestamp:0 data:{Timestamp:788090061452 ProcessId:13856 Layer:0 Flags:1 Priority:2999 _:[0 0 0 0 0 0]} open_event:{entry:{Flink:<nil> Blink:<nil>} context:0xffff830eb637a9c0 event:8 _:[0 0 0 0]} close_event:{entry:{Flink:<nil> Blink:<nil>} context:<nil> event:0 _:[0 0 0 0]} open:1 _:[0 0 0 0]}}

The dumped memory section holding this context_s object is attached. I cannot attach the entire memory dump both for privacy and size reasons, as it is 8 GB compressed and 24 GB uncompressed (the system has 768 GB of physical memory). In case it helps, here is the output of !vm:

Output of `!vm` ``` 2: kd> !vm Page File: \??\C:\pagefile.sys Current: 805306368 Kb Free Space: 805306360 Kb Minimum: 805306368 Kb Maximum: 846712400 Kb Physical Memory: 200966360 ( 803865440 Kb) Available Pages: 64282507 ( 257130028 Kb) ResAvail Pages: 65657869 ( 262631476 Kb) Locked IO Pages: 0 ( 0 Kb) Free System PTEs: 4295155904 (17180623616 Kb) Modified Pages: 18561 ( 74244 Kb) Modified PF Pages: 18560 ( 74240 Kb) Modified No Write Pages: 25 ( 100 Kb) NonPagedPool 0: 543 ( 2172 Kb) NonPagedPoolNx 0: 104934 ( 419736 Kb) NonPagedPool 1: 585 ( 2340 Kb) NonPagedPoolNx 1: 82906 ( 331624 Kb) NonPagedPool Usage: 1129 ( 4516 Kb) NonPagedPoolNx Usage: 1427253 ( 5709012 Kb) NonPagedPool Max: 4294967296 (17179869184 Kb) PagedPool 0: 110694 ( 442776 Kb) PagedPool 1: 55404 ( 221616 Kb) PagedPool 2: 155678 ( 622712 Kb) PagedPool Usage: 321776 ( 1287104 Kb) PagedPool Maximum: 4160749568 (16642998272 Kb) Processor Commit: 10273 ( 41092 Kb) Session Commit: 3550 ( 14200 Kb) Syspart SharedCommit 0 Shared Commit: 18227 ( 72908 Kb) Special Pool: 0 ( 0 Kb) Kernel Stacks: 35349 ( 141396 Kb) Pages For MDLs: 114866244 ( 459464976 Kb) ContigMem Pages: 0 ( 0 Kb) Pages For AWE: 0 ( 0 Kb) NonPagedPool Commit: 1505015 ( 6020060 Kb) PagedPool Commit: 321777 ( 1287108 Kb) Driver Commit: 19857 ( 79428 Kb) Boot Commit: 2359501 ( 9438004 Kb) SmallNonPagedPtesCommit: 0 ( 0 Kb) SlabAllocatorPages: 0 ( 0 Kb) SkPagesInUnchargedSlabs: 0 ( 0 Kb) System PageTables: 6030 ( 24120 Kb) VAD/PageTable Bitmaps: 10974 ( 43896 Kb) ProcessLockedFilePages: 14 ( 56 Kb) Pagefile Hash Pages: 0 ( 0 Kb) Sum System Commit: 119156811 ( 476627244 Kb) Total Private: 17448907 ( 69795628 Kb) Misc/Transient Commit: 72758 ( 291032 Kb) Committed pages: 136678476 ( 546713904 Kb) Commit limit: 402292952 ( 1609171808 Kb) Pid ImageName Commit SharedCommit Debt 35cc DcsAddMem.exe 65665288 Kb 1932 Kb 0 Kb 374c Dcsx.exe 621416 Kb 26636 Kb 0 Kb 151c MsMpEng.exe 458836 Kb 8756 Kb 0 Kb 3620 ngs-pulsar-amd64.exe 276536 Kb 8688 Kb 0 Kb 1640 java.exe 264656 Kb 6144 Kb 0 Kb 3ce4 dsm_om_connsvc64.exe 251112 Kb 6180 Kb 0 Kb 65cc rocketagent-x64.exe 199744 Kb 8712 Kb 0 Kb df8 svchost.exe 188220 Kb 11076 Kb 0 Kb b18 lsass.exe 137496 Kb 4520 Kb 0 Kb 1534 vmms.exe 136488 Kb 6324 Kb 0 Kb 39c8 DcsTmy.exe 123144 Kb 6400 Kb 0 Kb 2074 WmiPrvSE.exe 90996 Kb 8872 Kb 0 Kb 196c WmiPrvSE.exe 76960 Kb 9424 Kb 0 Kb 2eac powershell.exe 67716 Kb 6564 Kb 0 Kb 1934 clussvc.exe 59748 Kb 6132 Kb 0 Kb 3c74 Veeam.Backup.MountServic 43300 Kb 6200 Kb 0 Kb cc8 svchost.exe 40844 Kb 8596 Kb 0 Kb 2b30 WmiPrvSE.exe 39204 Kb 9020 Kb 0 Kb 1450 dsm_sa_datamgr64.exe 35816 Kb 6232 Kb 0 Kb 3fec Veeam.StandBy.Service.ex 35028 Kb 6492 Kb 0 Kb c84 svchost.exe 34984 Kb 9152 Kb 0 Kb b10 services.exe 29848 Kb 7000 Kb 0 Kb d40 svchost.exe 27392 Kb 8720 Kb 0 Kb cb8 svchost.exe 25416 Kb 6028 Kb 0 Kb 15b8 TeamViewer_Service.exe 24920 Kb 6148 Kb 0 Kb 2d54 rhs.exe 23424 Kb 2036 Kb 0 Kb 277c WmiPrvSE.exe 22208 Kb 8592 Kb 0 Kb cc0 svchost.exe 18288 Kb 8596 Kb 0 Kb 1500 svchost.exe 17872 Kb 6360 Kb 0 Kb 27a8 WmiPrvSE.exe 16528 Kb 8592 Kb 0 Kb 2d4c rhs.exe 16184 Kb 2036 Kb 0 Kb 7b70 vmwp.exe 15336 Kb 6016 Kb 0 Kb bc8 svchost.exe 15200 Kb 7008 Kb 0 Kb 6be0 vmwp.exe 14984 Kb 6016 Kb 0 Kb 5cbc vmwp.exe 14620 Kb 6016 Kb 0 Kb 7bcc vmwp.exe 14252 Kb 6016 Kb 0 Kb 6ff4 vmwp.exe 14144 Kb 6016 Kb 0 Kb 6494 vmwp.exe 14136 Kb 6016 Kb 0 Kb 13a4 VeeamHvIntegrationSvc.ex 14056 Kb 6128 Kb 0 Kb 2eec vmwp.exe 14032 Kb 6016 Kb 0 Kb 5c7c vmwp.exe 14028 Kb 6016 Kb 0 Kb 133c vmwp.exe 13908 Kb 6016 Kb 0 Kb 2e08 vmwp.exe 13892 Kb 6016 Kb 0 Kb 6d4c vmwp.exe 13672 Kb 6016 Kb 0 Kb 1dc0 NisSrv.exe 13604 Kb 2052 Kb 0 Kb 3ca8 vmwp.exe 13576 Kb 6016 Kb 0 Kb 6180 vmwp.exe 13516 Kb 6016 Kb 0 Kb 5d70 vmwp.exe 13156 Kb 6016 Kb 0 Kb 984 dwm.exe 13120 Kb 16756 Kb 0 Kb 6088 vmwp.exe 12756 Kb 6016 Kb 0 Kb 6cd4 vmwp.exe 12576 Kb 6016 Kb 0 Kb f10 KaseyaEndpoint.exe 12524 Kb 6124 Kb 0 Kb 6ab8 vmwp.exe 12448 Kb 6016 Kb 0 Kb 150c AgentMon.exe 12344 Kb 9500 Kb 0 Kb 7144 vmwp.exe 12300 Kb 6016 Kb 0 Kb 5628 vmwp.exe 12208 Kb 6016 Kb 0 Kb 7268 rundll32.exe 12152 Kb 9008 Kb 0 Kb 7264 rundll32.exe 11716 Kb 9008 Kb 0 Kb 6518 vmwp.exe 11672 Kb 6016 Kb 0 Kb 5e84 vmwp.exe 11592 Kb 6016 Kb 0 Kb 74e4 rundll32.exe 11588 Kb 9008 Kb 0 Kb 76d0 LogonUI.exe 11344 Kb 13804 Kb 0 Kb b88 svchost.exe 11308 Kb 6452 Kb 0 Kb 6568 vmwp.exe 11176 Kb 6016 Kb 0 Kb 5844 rundll32.exe 11044 Kb 8696 Kb 0 Kb 7494 rundll32.exe 10984 Kb 8688 Kb 0 Kb 236c vmcompute.exe 10908 Kb 2036 Kb 0 Kb 6bb8 rundll32.exe 10856 Kb 8688 Kb 0 Kb 5d28 rundll32.exe 10852 Kb 8696 Kb 0 Kb 2e10 rhs.exe 10784 Kb 2032 Kb 0 Kb b58 rundll32.exe 10712 Kb 8688 Kb 0 Kb 7750 rundll32.exe 10708 Kb 8688 Kb 0 Kb 723c rundll32.exe 10660 Kb 9008 Kb 0 Kb 71ec rundll32.exe 10568 Kb 8688 Kb 0 Kb 62b0 rundll32.exe 10564 Kb 8688 Kb 0 Kb 773c rundll32.exe 10460 Kb 8688 Kb 0 Kb 7608 rundll32.exe 10372 Kb 8696 Kb 0 Kb 6444 rundll32.exe 10364 Kb 8688 Kb 0 Kb 14e0 svchost.exe 10360 Kb 6172 Kb 0 Kb 2a70 rundll32.exe 10172 Kb 8696 Kb 0 Kb c8c svchost.exe 9912 Kb 1940 Kb 0 Kb 2610 Lua.exe 7648 Kb 2572 Kb 0 Kb 2604 Lua.exe 6924 Kb 2572 Kb 0 Kb 3698 vds.exe 5868 Kb 2040 Kb 0 Kb 14f0 svchost.exe 5576 Kb 6356 Kb 0 Kb 13b8 VeeamDeploymentSvc.exe 5344 Kb 2048 Kb 0 Kb 5c38 conhost.exe 5012 Kb 4484 Kb 0 Kb 35d0 conhost.exe 5000 Kb 4484 Kb 0 Kb 304c rhs.exe 4916 Kb 6124 Kb 0 Kb 139c svchost.exe 4816 Kb 6128 Kb 0 Kb 1524 VeeamTransportSvc.exe 4764 Kb 2024 Kb 0 Kb 1e94 Veeam.Guest.Interaction. 4092 Kb 6140 Kb 0 Kb 14cc IPROSetMonitor.exe 3568 Kb 2052 Kb 0 Kb 14e8 dsm_sa_eventmgr64.exe 3416 Kb 2084 Kb 0 Kb a30 csrss.exe 3300 Kb 14616 Kb 0 Kb 300c MpCmdRun.exe 3288 Kb 6020 Kb 0 Kb 3e3c msdtc.exe 2888 Kb 1944 Kb 0 Kb 15dc svchost.exe 2840 Kb 352 Kb 0 Kb e54 svchost.exe 2668 Kb 1936 Kb 0 Kb 1498 svchost.exe 2636 Kb 6124 Kb 0 Kb 1488 iscsidcb.exe 2544 Kb 2024 Kb 0 Kb 30a8 svchost.exe 2376 Kb 1936 Kb 0 Kb 1628 IntelDCB.exe 2284 Kb 2028 Kb 0 Kb 3ab4 svchost.exe 2252 Kb 332 Kb 0 Kb 21a0 svchost.exe 1924 Kb 340 Kb 0 Kb 1568 VeeamNFSSvc.exe 1844 Kb 2036 Kb 0 Kb 1448 sqlwriter.exe 1788 Kb 6132 Kb 0 Kb 692c winlogon.exe 1620 Kb 9532 Kb 0 Kb 1b30 unsecapp.exe 1596 Kb 2028 Kb 0 Kb 781c conhost.exe 1384 Kb 4576 Kb 0 Kb 2614 conhost.exe 1348 Kb 4576 Kb 0 Kb 2654 conhost.exe 1340 Kb 4576 Kb 0 Kb 3ca0 csrss.exe 1336 Kb 6204 Kb 0 Kb 1f18 conhost.exe 1156 Kb 2004 Kb 0 Kb a84 wininit.exe 1048 Kb 4476 Kb 0 Kb 7840 crashpad_handler-x64.exe 980 Kb 2020 Kb 0 Kb 14ac pcns.exe 880 Kb 2016 Kb 0 Kb 900 smss.exe 408 Kb 232 Kb 0 Kb 4 System 128 Kb 512 Kb 0 Kb 78f4 VeeamAgent.exe 0 Kb 0 Kb 0 Kb 78cc VeeamAgent.exe 0 Kb 0 Kb 0 Kb 75ac VeeamAgent.exe 0 Kb 0 Kb 0 Kb 7448 VeeamAgent.exe 0 Kb 0 Kb 0 Kb 6e64 VeeamAgent.exe 0 Kb 0 Kb 0 Kb 6dec VeeamAgent.exe 0 Kb 0 Kb 0 Kb 6828 VeeamAgent.exe 0 Kb 0 Kb 0 Kb 5fd0 VeeamAgent.exe 0 Kb 0 Kb 0 Kb 5fcc VeeamAgent.exe 0 Kb 0 Kb 0 Kb 5a60 VeeamAgent.exe 0 Kb 0 Kb 0 Kb 587c VeeamAgent.exe 0 Kb 0 Kb 0 Kb 5780 VeeamAgent.exe 0 Kb 0 Kb 0 Kb 5760 VeeamPSDirectCtrl_X64.ex 0 Kb 0 Kb 0 Kb 3b68 VeeamAgent.exe 0 Kb 0 Kb 0 Kb 2d5c VeeamAgent.exe 0 Kb 0 Kb 0 Kb 2aa0 explorer.exe 0 Kb 0 Kb 0 Kb 25a4 logman.exe 0 Kb 0 Kb 0 Kb de0 VeeamAgent.exe 0 Kb 0 Kb 0 Kb b0c smss.exe 0 Kb 0 Kb 0 Kb 78c Secure System 0 Kb 0 Kb 0 Kb ```

Please let me know if I can provide any further data. Any help is greatly appreciated. Thanks in advance!

context.mem.zip

basil00 commented 1 year ago

Thanks for the report. After a quick look, it appears the error originates from the call to InsertTailList in windivert_reflect_open_event. This could be caused by memory corruption.

Do you have any PoC code to replicate the problem? If so, please send it to basil at reqrypt.org.

elmeyer commented 1 year ago

Thanks for your reply. I have no reliable PoC code to share at this time. We will advise the customer to check for memory errors; since I concur that memory corruption is a likely cause, I will close this issue and re-open only if the customer's memory diagnostics come back clean.

basil00 commented 1 year ago

I was thinking of memory corruption caused by a software bug, such as use-after-free. However, without PoC, it will be hard to track down. Any additional clues could also help, such as what handles were being open, how frequently the problem occurs, etc.?