KERNEL_MODE_HEAP_CORRUPTION after call to WfpNotifyFlowContextDelete (windivert_flow_delete_notify?)

Hi, we have a customer running SAP dispatcher/workers on a system on which we run code tracking flow events using WinDivert 2.2.0. The WinDivert handle is opened with no filter. The aforementioned crashes happened about once a day while our code was running. Here is the bugcheck output of one such crash:

*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

KERNEL_MODE_HEAP_CORRUPTION (13a)
The kernel mode heap manager has detected corruption in a heap.
Arguments:
Arg1: 0000000000000009, An invalid argument was specified for the current operation.
Arg2: ffffdf884b602000, Address of the heap that reported the corruption
Arg3: ffffdf88b2f7cd00, Address at which the corruption was detected
Arg4: 0000000000000000

Debugging Details:
------------------

KEY_VALUES_STRING: 1

    Key  : Analysis.CPU.mSec
    Value: 2655

    Key  : Analysis.DebugAnalysisManager
    Value: Create

    Key  : Analysis.Elapsed.mSec
    Value: 3090

    Key  : Analysis.Init.CPU.mSec
    Value: 2202

    Key  : Analysis.Init.Elapsed.mSec
    Value: 232440

    Key  : Analysis.Memory.CommitPeak.Mb
    Value: 107

    Key  : Bugcheck.Code.DumpHeader
    Value: 0x13a

    Key  : Bugcheck.Code.KiBugCheckData
    Value: 0x13a

    Key  : Bugcheck.Code.Register
    Value: 0x13a

    Key  : WER.OS.Branch
    Value: rs5_release

    Key  : WER.OS.Timestamp
    Value: 2018-09-14T14:34:00Z

    Key  : WER.OS.Version
    Value: 10.0.17763.1

FILE_IN_CAB:  MEMORY.DMP

VIRTUAL_MACHINE:  VMware

BUGCHECK_CODE:  13a

BUGCHECK_P1: 9

BUGCHECK_P2: ffffdf884b602000

BUGCHECK_P3: ffffdf88b2f7cd00

BUGCHECK_P4: 0

CORRUPTING_POOL_ADDRESS:  ffffdf88b2f7cd00 Nonpaged pool

BLACKBOXBSD: 1 (!blackboxbsd)

BLACKBOXPNP: 1 (!blackboxpnp)

PROCESS_NAME:  disp+work.exe

STACK_TEXT:  
ffffdc0e`23c15b48 fffff800`51b95249     : 00000000`0000013a 00000000`00000009 ffffdf88`4b602000 ffffdf88`b2f7cd00 : nt!KeBugCheckEx
ffffdc0e`23c15b50 fffff800`51b952a8     : 00000000`00000009 ffffdf88`4cd57c40 ffffdf88`4b602000 00000000`00000000 : nt!RtlpHeapHandleError+0x29
ffffdc0e`23c15b90 fffff800`51b94ed1     : ffffdf88`b2f7cd00 fffff800`5189e000 ffffdf88`4f057d58 ffffdc0e`23c166f0 : nt!RtlpHpHeapHandleError+0x58
ffffdc0e`23c15bc0 fffff800`51be65e0     : 17ae1dfe`0000176c fffff80b`01d8bf94 0100007f`dde30002 00000000`00000000 : nt!RtlpLogHeapFailure+0x45
ffffdc0e`23c15bf0 fffff800`54bd299d     : ffffdf88`4f622880 ffffdf88`4c7b08e0 ffffdf88`00000709 ffffdf88`76694457 : nt!ExFreePoolWithTag+0x980
ffffdc0e`23c15d20 fffff80b`4ab2f320     : ffffdf88`4f622880 ffffdf88`b2f7cd10 ffffdf88`4c7b08e0 ffffdf88`4c4b8110 : WinDivert64+0x299d
ffffdc0e`23c15df0 fffff80b`4ab2eeb0     : ffffdf88`4fe46d50 00000000`00000034 00000000`00000000 ffffdf88`4c4d5dc0 : NETIO!WfpNotifyFlowContextDelete+0x1c0
ffffdc0e`23c15e60 fffff80b`4ca339c5     : 00000000`0000ff02 ffffdf88`54c8d310 ffffdc0e`23c160d0 00000000`00000000 : NETIO!KfdAleNotifyFlowDeletion+0x170
ffffdc0e`23c15ed0 fffff80b`4ca0c7ad     : 00000000`00000000 00000000`00000000 ffffdf88`54c10cc0 00000046`eea977ff : tcpip!WfpAleFreeRemoteEndpoint+0x25
ffffdc0e`23c15f50 fffff80b`4cad9bf1     : ffffdf88`4fe46d88 00000000`00000000 ffffdc0e`23c160d0 fffff80b`4a9c5daa : tcpip!WfpAleDecrementWaitRef+0x75
ffffdc0e`23c15f80 fffff80b`4c9e4bf6     : 00000000`00000056 00000000`00000000 00000000`00000000 00000000`00000001 : tcpip!WfpAleSetUdpCachedRemoteEndpoint+0x5a95d
ffffdc0e`23c15fd0 fffff80b`4ca292cc     : 00000000`00000000 00000000`0000effa ffffdc0e`23c16710 ffffdc0e`23c16b04 : tcpip!WfpAleFastUdpInspection+0x1076
ffffdc0e`23c16390 fffff80b`4ca27e62     : ffffdf88`4c9cc040 00000000`00000001 00000000`00242359 00000000`00000000 : tcpip!UdpSendMessagesOnPath+0xf4c
ffffdc0e`23c16820 fffff80b`4ca27c55     : 00000000`00000003 ffffdc0e`23c17101 ffffdc0e`23c16cd0 ffffdc0e`23c16cd0 : tcpip!UdpSendMessages+0x1f2
ffffdc0e`23c16bc0 fffff800`518d3228     : ffffdc0e`23c16e20 ffffdf88`507f19d0 ffffdf88`ca992ac0 fffff80b`4ca13c18 : tcpip!UdpTlProviderSendMessagesCalloutRoutine+0x15
ffffdc0e`23c16bf0 fffff800`518d319d     : fffff80b`4ca27c40 ffffdc0e`23c16cd0 00000000`00000000 ffffc95b`40000000 : nt!KeExpandKernelStackAndCalloutInternal+0x78
ffffdc0e`23c16c60 fffff80b`4ca3631d     : 00000000`00000004 ffffdf88`54d022a0 00000000`00000000 fffff80b`4c0c28e0 : nt!KeExpandKernelStackAndCalloutEx+0x1d
ffffdc0e`23c16ca0 fffff80b`4c0dbc77     : 00000000`00000000 ffffdc0e`23c17540 ffffdf88`a2ef7ca0 00000000`00000740 : tcpip!UdpTlProviderSendMessages+0x6d
ffffdc0e`23c16d20 fffff80b`4c0c1603     : 00000000`00000000 00000000`00000000 ffffdf88`54d022a0 00000000`00000100 : afd!AfdFastDatagramSend+0x687
ffffdc0e`23c16f10 fffff800`51ea027b     : 006c006c`00000000 ffffdf88`57c3c690 ffffdc0e`23c17540 00000000`00000018 : afd!AfdFastIoDeviceControl+0x5b3
ffffdc0e`23c17290 fffff800`51ea0986     : 00000000`00000000 00000000`000002d8 00000000`00000000 00000000`00000000 : nt!IopXxxControlFile+0x81b
ffffdc0e`23c173e0 fffff800`51a67c05     : 00000000`00000000 00000000`00000001 00000000`00000000 00000000`6313c3ae : nt!NtDeviceIoControlFile+0x56
ffffdc0e`23c17450 00007ff9`9f000804     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x25
0000001a`b57fef18 00000000`00000000     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x00007ff9`9f000804

SYMBOL_NAME:  WinDivert64+299d

MODULE_NAME: WinDivert64

IMAGE_NAME:  WinDivert64.sys

STACK_COMMAND:  .cxr; .ecxr ; kb

BUCKET_ID_FUNC_OFFSET:  299d

FAILURE_BUCKET_ID:  0x13a_9_WinDivert64!unknown_function

OS_VERSION:  10.0.17763.1

BUILDLAB_STR:  rs5_release

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 10

FAILURE_ID_HASH:  {b146b3ac-ae60-5b81-f570-f8c59e01d1f8}

The call stack suggests that the crash happens when calling ExFreePoolWithTag from windivert_free in windivert_flow_delete_notify (https://github.com/basil00/Divert/blob/v2.2.0/sys/windivert.c#L4243).

Using WinDbg and the CodeMachine Kernel Debugger Extension DLL, I checked the parameters with which ExFreePoolWithTag gets called:

...
04 ffffdc0e23c15bf0 fffff80054bd299d nt!ExFreePoolWithTag+980 (perf)
    Parameter[0] = ffffdf88b2f7cd10
    Parameter[1] = 0000000076694457
    Parameter[2] = (unknown)       
    Parameter[3] = (unknown)
...

I have dumped the flow_s struct from the memory region pointed to by Parameter[0], as well as the context_s struct pointed to within it. The flow is as follows (apologies for the crude formatting):

{entry:{Flink:0xffffdf88b2f7d510 Blink:0xffffdf884cd57c68} context:0xffffdf884cd57c40 flow_id:46784 callout_id:300 layer_id:52 inserted:true deleted:true outbound:true loopback:true ipv6:false data:{EndpointId:6635 ParentEndpointId:0 ProcessId:5996 LocalAddr:127.0.0.1 RemoteAddr:127.0.0.1 LocalPort:53931 RemotePort:64239 Protocol:UDP}}

WINDIVERT_CONTEXT_SIZE reports as 792 bytes on my AMD64 system, so that is the size of the memory region I wrote to context.mem. I haven't further analyzed that memory. Please find both regions attached: flow_context.zip

I am inexperienced at analyzing memory dumps, so this is the extent of information I was able to obtain thus far. I will gladly perform further analyses on request and can provide additional partial memory extracts if necessary. A complete kernel-mode memory dump is available, but I cannot provide it publicly for obvious privacy reasons.

Since nothing about the flow_s struct seems off to me at first glance, I am not even sure if this is actually a bug in WinDivert or something that's rather caused by SAP's disp+work.exe? Any insight would be much appreciated. Thanks in advance!

basil00 / Divert

KERNEL_MODE_HEAP_CORRUPTION after call to WfpNotifyFlowContextDelete (windivert_flow_delete_notify?) #315