wtf: cabi/mem-cabi.cc:34: static bool BX_MEM_C::dbg_fetch_mem(BX_CPU_C*, bx_phy_address, unsigned int, Bit8u*): Assertion `false' failed.

[kiwids0220]

Hi, just giving it a shot at fuzzing with Windows. I have generated my corpus and states, but upon running them on windows it's throwing the assertion error. not sure what I am doing wrong here.

Command

### Server
./wtf master --name sspisrv  --max_len 1024 --runs=50000 --target /mnt/c/Users/Kiwi/Desktop/wtf/targets/

### Node
sudo ./wtf fuzz --name sspisrv --limit 500000 --target /mnt/c/Users/Kiwi/Desktop/wtf/targets

### On Linux Error
Setting @fptw to 0xff'ff.
The debugger instance is loaded with 5 items
Setting debug register status to zero.
Setting debug register status to zero.
Could not set a breakpoint at hal!HalpPerfInterrupt.
Failed to set breakpoint on HalpPerfInterrupt, but ignoring..
Dialing to tcp://localhost:31337/..
wtf: cabi/mem-cabi.cc:34: static bool BX_MEM_C::dbg_fetch_mem(BX_CPU_C*, bx_phy_address, unsigned int, Bit8u*): Assertion `false' failed.
Aborted
### On Windows Error
Setting @fptw to 0xff'ff.
Initializing the debugger instance.. (this takes a bit of time)
Setting debug register status to zero.
Setting debug register status to zero.
Could not set a breakpoint at hal!HalpPerfInterrupt.
Failed to set breakpoint on HalpPerfInterrupt, but ignoring..
Dialing to tcp://localhost:31337/..
Assertion failed: false, file cabi/mem-cabi.cc, line 34

Any pointer will be helpful!! Thanks!

[0vercl0k]

Hello 👋🏽

Pretty interesting one, I don't think I have ever seen this; it seems that you are hitting this assert:

bool BX_MEM_C::dbg_fetch_mem(BX_CPU_C *cpu, bx_phy_address addr, unsigned len, Bit8u *buf)
{
    assert(false);

    return false;
}

I have got no clue why though - I'll investigate what's going on; thank you for reporting.

Cheers

[kiwids0220]

Thank you for taking a look at this, I suspected it was issue with backend, so I tried with KVM and still getting a different error. Could it has something to do with how I wrote the harness fuzzer_*.cc?

[0vercl0k]

Are you able to share the error that you're getting?

I think a lot of things can happen but based on what I know, if it triggers some weird behavior on both the bx/kvm backends I wonder if it doesn't come from your snapshot and not your fuzzer module.

[kiwids0220]

For sure, Here is the screenshot for the errors I was getting! Another interesting behavior I noticed is that when running KVM backend, the fuzz fails with KVM backend initialization, and when running with bochscpu backend, the fuzzer will initialize successfully but fails over a dozen of cases (Shown in the Windows error screenshot below)

On WSL Ubuntu running kvm backend

On Windows 11 running bochscpu backend

[kiwids0220]

The state folder contains mem dump, reg.json, symbol-store.json (got it when running on Windows)

[0vercl0k]

Gotcha, thanks for the extra details.

My gut feeling is that the issues are not related to each other:

• The KVM issue you are running into is usually an issue when you are grabbing a dump on a specific hardware, and running it on a different one. This causes issues when, for example, the hardware you grabbed the snapshot on support feature X and has a bit in one of the system register. But the system that runs the snapshot doesn't support it, so KVM complains. I'd recommend to look through the closed issues, you might be hitting one of those, otherwise let's track this into a different issue
• For the bochscpu issue it seems that a testcase triggers a code-path that leads bochs into invoking this stubbed function and the author of bochscpu assumed this function could / would never be invoked. I need to follow up with him to see if he has any ideas of what could be going wrong.

[kiwids0220]

Gotcha, that makes much more sense! Thank you for the explaination! The testcase I put under Inputs folder contains reference pointers to previously allocated heap. Is that why bochscpu is calling fetch_mem function? And even it did make the call, it won't find the heap buffer because a new heap is allocated for each of the testcases and I only have one snapshot/memdmp from WinDbg. I wonder if I can solve it heap issue by set a breakpoint somewhere before Heap allocation and take a snapshot with WinDbg there?

[y0ny0ns0n]

Some callers of dbg_fetch_mem on Bochs:

• https://github.com/bochs-emu/Bochs/blob/master/bochs/gui/enh_dbg.cc#L2322
• https://github.com/bochs-emu/Bochs/blob/master/bochs/cpu/debugstuff.cc#L44
• https://github.com/bochs-emu/Bochs/blob/master/bochs/bx_debug/dbg_main.cc#L1377
• etc....

I think it could be an issue for memory dump. Previously allocated heap can not be a matter if it is still available on mem.dmp( you can just open the .dmp file and check it ).

I recommend setting a larger RAM size( 4gb or 8gb ), taking a new snapshot( using !bdump_full ), and testing again.

Also, allocating large heap memory( e.g. malloc(0x100000) ) during emulation will cause cr3. I also recommend to set limitation if there's any size variable on testcase.

[kiwids0220]

Some callers of dbg_fetch_mem on Bochs:

• https://github.com/bochs-emu/Bochs/blob/master/bochs/gui/enh_dbg.cc#L2322

• https://github.com/bochs-emu/Bochs/blob/master/bochs/cpu/debugstuff.cc#L44

• https://github.com/bochs-emu/Bochs/blob/master/bochs/bx_debug/dbg_main.cc#L1377

• etc....

I think it could be an issue for memory dump. Previously allocated heap can not be a matter if it is still available on mem.dmp( you can just open the .dmp file and check it ).

I recommend setting a larger RAM size( 4gb or 8gb ), taking a new snapshot( using !bdump_full ), and testing again.

Also, allocating large heap memory( e.g. malloc(0x100000) ) during emulation will cause cr3. I also recommend to set limitation if there's any size variable on testcase.

Gotcha, thanks for the comment and pointing out potential issues and their workarounds! I will try bdump_full and try running it again!

[yrp604]

My high level understanding of the dbg fetch memory paths is that bochs splits its memory access APIs based on their source. For example, if you asked the bochs debugger to read gva 0x41414141, you wouldnt want the debugger to perturb the system by raising a #PF if the gva doesn't translate.

If you're hitting this assert it means there is some guest triggerable behavior invoking the dbg path. Is it possible to get a stack trace of the assert being hit? A comprehensive fix will be difficult without understanding where/why this code path is being hit, and I really don't want to audit all the sites.

If you'd rather not provide this or just want to get something working, you can glue the dbg fetch path to the normal memory fetch path -- essentially just remove the assert and make the dbg fetch path call into the regular memory access path.

[kiwids0220]

I'd love to provide a stack trace. I should be able to put the fuzzer in a debugger and print the stack trace where the error occurs right?

[yrp604]

Yep. You’ll want to have debug info available, but aside from that should be straightforward.

[kiwids0220]

Sorry it's been a crazy week for me. But I managed to pull the fuzzer stack trace at the moment of crashing.

ModLoad: 00007ff9`5eae0000 00007ff9`5f359000   C:\Users\Kiwi\Desktop\wtf\src\build\RelWithDebInfo\dbgeng.dll
ModLoad: 00007ff9`7dde0000 00007ff9`7e00d000   C:\Users\Kiwi\Desktop\wtf\src\build\RelWithDebInfo\dbghelp.dll
ModLoad: 00007ff9`dfcb0000 00007ff9`dfce7000   C:\Windows\SYSTEM32\XmlLite.dll
ModLoad: 00007ff9`e7750000 00007ff9`e7827000   C:\Windows\System32\OLEAUT32.dll
ModLoad: 00007ff9`e6630000 00007ff9`e69ba000   C:\Windows\System32\combase.dll
ModLoad: 00007ff9`e5a10000 00007ff9`e5aaa000   C:\Windows\System32\msvcp_win.dll
ModLoad: 00007ff9`dab20000 00007ff9`dac06000   C:\Windows\SYSTEM32\dbgmodel.dll
ModLoad: 00007ff9`cb610000 00007ff9`cb61c000   C:\Windows\SYSTEM32\Secur32.dll
ModLoad: 00007ff9`e4b40000 00007ff9`e4b82000   C:\Windows\SYSTEM32\SSPICLI.DLL
ModLoad: 00007ff9`e63b0000 00007ff9`e640e000   C:\Windows\System32\shlwapi.dll
ModLoad: 00007ff9`decf0000 00007ff9`decfa000   C:\Windows\SYSTEM32\version.dll
ModLoad: 00007ff9`e6fa0000 00007ff9`e713c000   C:\Windows\System32\ole32.dll
ModLoad: 00007ff9`e75c0000 00007ff9`e75e9000   C:\Windows\System32\GDI32.dll
ModLoad: 00007ff9`e5d40000 00007ff9`e5d66000   C:\Windows\System32\win32u.dll
ModLoad: 00007ff9`e58f0000 00007ff9`e5a09000   C:\Windows\System32\gdi32full.dll
ModLoad: 00007ff9`e7830000 00007ff9`e79da000   C:\Windows\System32\USER32.dll
ModLoad: 00007ff9`e6410000 00007ff9`e6441000   C:\Windows\System32\IMM32.DLL
ModLoad: 00007ff9`e6a90000 00007ff9`e6b83000   C:\Windows\System32\shcore.dll
ModLoad: 00007ff9`874a0000 00007ff9`8752f000   C:\Users\Kiwi\Desktop\wtf\src\build\RelWithDebInfo\symsrv.dll
ModLoad: 00007ff9`e5bd0000 00007ff9`e5d37000   C:\Windows\System32\CRYPT32.dll
ModLoad: 00007ff9`b9480000 00007ff9`b9974000   C:\Windows\SYSTEM32\wininet.dll
ModLoad: 00007ff9`e4d60000 00007ff9`e4dc9000   C:\Windows\System32\mswsock.dll
ModLoad: 00007ff9`e4380000 00007ff9`e4473000   C:\Windows\SYSTEM32\DNSAPI.dll
ModLoad: 00007ff9`e4350000 00007ff9`e437d000   C:\Windows\SYSTEM32\IPHLPAPI.DLL
ModLoad: 00007ff9`e6570000 00007ff9`e6579000   C:\Windows\System32\NSI.dll
ModLoad: 00007ff9`ddc30000 00007ff9`ddcb4000   C:\Windows\System32\fwpuclnt.dll
ModLoad: 00007ff9`ddcc0000 00007ff9`ddcca000   C:\Windows\System32\rasadhlp.dll
(c248.77c4): Security check failure or stack buffer overrun - code c0000409 (!!! second chance !!!)
Subcode: 0x7 FAST_FAIL_FATAL_APP_EXIT 
ucrtbase!abort+0x4e:
00007ff9`e5b2f61e cd29            int     29h
0:000> k
 # Child-SP          RetAddr               Call Site
00 0000002a`bc0fc000 00007ff9`e5b3106a     ucrtbase!abort+0x4e
01 0000002a`bc0fc030 00007ff9`e5b30f5d     ucrtbase!common_assert_to_stderr_direct+0xb2
02 0000002a`bc0fc520 00007ff9`e5b311f1     ucrtbase!common_assert_to_stderr<wchar_t>+0x1d
03 0000002a`bc0fc560 00007ff7`d406c87e     ucrtbase!wassert+0x71
04 0000002a`bc0fc590 00007ff7`d404c94f     wtf!BX_MEM_C::dbg_fetch_mem+0x1e
05 0000002a`bc0fc5c0 00007ff7`d4036deb     wtf!BX_CPU_C::debug+0x8cf
06 0000002a`bc0fc8f0 00007ff7`d40365fd     wtf!BX_CPU_C::exception+0x11b
07 0000002a`bc0fc950 00007ff7`d403582a     wtf!BX_CPU_C::translate_linear_long_mode+0x24d
08 0000002a`bc0fca70 00007ff7`d4033dee     wtf!BX_CPU_C::translate_linear+0x11a
09 0000002a`bc0fcb00 00007ff7`d4047880     wtf!BX_CPU_C::access_read_linear+0xce
0a 0000002a`bc0fcb50 00007ff7`d4073ed4     wtf!BX_CPU_C::system_read_qword+0x90
0b 0000002a`bc0fcb90 00007ff7`d4037434     wtf!BX_CPU_C::get_RSP_from_TSS+0x74
0c 0000002a`bc0fcbc0 00007ff7`d4037033     wtf!BX_CPU_C::long_mode_int+0x344
0d 0000002a`bc0fcc80 00007ff7`d4036f2b     wtf!BX_CPU_C::interrupt+0xe3
0e 0000002a`bc0fccd0 00007ff7`d4036f01     wtf!BX_CPU_C::exception+0x25b
0f 0000002a`bc0fcd30 00007ff7`d40365fd     wtf!BX_CPU_C::exception+0x231
10 0000002a`bc0fcd90 00007ff7`d403582a     wtf!BX_CPU_C::translate_linear_long_mode+0x24d
11 0000002a`bc0fceb0 00007ff7`d4033dee     wtf!BX_CPU_C::translate_linear+0x11a
12 0000002a`bc0fcf40 00007ff7`d4047880     wtf!BX_CPU_C::access_read_linear+0xce
13 0000002a`bc0fcf90 00007ff7`d4073ed4     wtf!BX_CPU_C::system_read_qword+0x90
14 0000002a`bc0fcfd0 00007ff7`d4037434     wtf!BX_CPU_C::get_RSP_from_TSS+0x74
15 0000002a`bc0fd000 00007ff7`d4037033     wtf!BX_CPU_C::long_mode_int+0x344
16 0000002a`bc0fd0c0 00007ff7`d4036f2b     wtf!BX_CPU_C::interrupt+0xe3
17 0000002a`bc0fd110 00007ff7`d40365fd     wtf!BX_CPU_C::exception+0x25b
18 0000002a`bc0fd170 00007ff7`d403582a     wtf!BX_CPU_C::translate_linear_long_mode+0x24d
19 0000002a`bc0fd290 00007ff7`d40340cf     wtf!BX_CPU_C::translate_linear+0x11a
1a 0000002a`bc0fd320 00007ff7`d4070744     wtf!BX_CPU_C::access_write_linear+0xcf
1b 0000002a`bc0fd370 00007ff7`d417f152     wtf!BX_CPU_C::write_linear_dword+0x134
1c 0000002a`bc0fd3c0 00007ff7`d417f29b     wtf!BX_CPU_C::MOV64_EdGdM+0xa2
1d 0000002a`bc0fd3f0 00007ff7`d417ff79     wtf!BX_CPU_C::MOV64_GdEdM+0xeb
1e 0000002a`bc0fd420 00007ff7`d40327f7     wtf!BX_CPU_C::MOV_GqEqM+0xe9
1f 0000002a`bc0fd450 00007ff7`d3e76320     wtf!BX_CPU_C::cpu_loop+0x127
20 0000002a`bc0fd480 00007ff7`d3f4be26     wtf!ZN8bochscpu3cpu6CpuRun3run17h68b2c636903acc81E+0xd0
21 0000002a`bc0fd4d0 00007ff7`d3f62b7c     wtf!BochscpuBackend_t::Run+0x96 [C:\Users\Kiwi\Desktop\wtf\src\wtf\bochscpu_backend.cc @ 394] 
22 0000002a`bc0fd510 00007ff7`d3f63373     wtf!RunTestcaseAndRestore+0x7c [C:\Users\Kiwi\Desktop\wtf\src\wtf\client.cc @ 111] 
23 0000002a`bc0fd5b0 00007ff7`d3f91dec     wtf!Client_t::Run+0x163 [C:\Users\Kiwi\Desktop\wtf\src\wtf\client.cc @ 244] 
24 0000002a`bc0fd690 00007ff7`d3fb585e     wtf!FuzzSubcommand+0x2c [C:\Users\Kiwi\Desktop\wtf\src\wtf\subcommands.cc @ 96] 
25 0000002a`bc0fd700 00007ff7`d4193ee8     wtf!main+0x23de [C:\Users\Kiwi\Desktop\wtf\src\wtf\wtf.cc @ 472] 
26 (Inline Function) --------`--------     wtf!invoke_main+0x22 [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 78] 
27 0000002a`bc0ff7a0 00007ff9`e69d26ad     wtf!__scrt_common_main_seh+0x10c [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 288] 
28 0000002a`bc0ff7e0 00007ff9`e842aa68     KERNEL32!BaseThreadInitThunk+0x1d
29 0000002a`bc0ff810 00000000`00000000     ntdll!RtlUserThreadStart+0x28
0:000> g
WARNING: Continuing a non-continuable exception
(c248.77c4): Security check failure or stack buffer overrun - code c0000409 (!!! second chance !!!)
Subcode: 0x7 FAST_FAIL_FATAL_APP_EXIT 
ucrtbase!abort+0x4e:
00007ff9`e5b2f61e cd29            int     29h
0:000> k
 # Child-SP          RetAddr               Call Site
00 0000002a`bc0fc000 00007ff9`e5b3106a     ucrtbase!abort+0x4e
01 0000002a`bc0fc030 00007ff9`e5b30f5d     ucrtbase!common_assert_to_stderr_direct+0xb2
02 0000002a`bc0fc520 00007ff9`e5b311f1     ucrtbase!common_assert_to_stderr<wchar_t>+0x1d
03 0000002a`bc0fc560 00007ff7`d406c87e     ucrtbase!wassert+0x71
04 0000002a`bc0fc590 00007ff7`d404c94f     wtf!BX_MEM_C::dbg_fetch_mem+0x1e
05 0000002a`bc0fc5c0 00007ff7`d4036deb     wtf!BX_CPU_C::debug+0x8cf
06 0000002a`bc0fc8f0 00007ff7`d40365fd     wtf!BX_CPU_C::exception+0x11b
07 0000002a`bc0fc950 00007ff7`d403582a     wtf!BX_CPU_C::translate_linear_long_mode+0x24d
08 0000002a`bc0fca70 00007ff7`d4033dee     wtf!BX_CPU_C::translate_linear+0x11a
09 0000002a`bc0fcb00 00007ff7`d4047880     wtf!BX_CPU_C::access_read_linear+0xce
0a 0000002a`bc0fcb50 00007ff7`d4073ed4     wtf!BX_CPU_C::system_read_qword+0x90
0b 0000002a`bc0fcb90 00007ff7`d4037434     wtf!BX_CPU_C::get_RSP_from_TSS+0x74
0c 0000002a`bc0fcbc0 00007ff7`d4037033     wtf!BX_CPU_C::long_mode_int+0x344
0d 0000002a`bc0fcc80 00007ff7`d4036f2b     wtf!BX_CPU_C::interrupt+0xe3
0e 0000002a`bc0fccd0 00007ff7`d4036f01     wtf!BX_CPU_C::exception+0x25b
0f 0000002a`bc0fcd30 00007ff7`d40365fd     wtf!BX_CPU_C::exception+0x231
10 0000002a`bc0fcd90 00007ff7`d403582a     wtf!BX_CPU_C::translate_linear_long_mode+0x24d
11 0000002a`bc0fceb0 00007ff7`d4033dee     wtf!BX_CPU_C::translate_linear+0x11a
12 0000002a`bc0fcf40 00007ff7`d4047880     wtf!BX_CPU_C::access_read_linear+0xce
13 0000002a`bc0fcf90 00007ff7`d4073ed4     wtf!BX_CPU_C::system_read_qword+0x90
14 0000002a`bc0fcfd0 00007ff7`d4037434     wtf!BX_CPU_C::get_RSP_from_TSS+0x74
15 0000002a`bc0fd000 00007ff7`d4037033     wtf!BX_CPU_C::long_mode_int+0x344
16 0000002a`bc0fd0c0 00007ff7`d4036f2b     wtf!BX_CPU_C::interrupt+0xe3
17 0000002a`bc0fd110 00007ff7`d40365fd     wtf!BX_CPU_C::exception+0x25b
18 0000002a`bc0fd170 00007ff7`d403582a     wtf!BX_CPU_C::translate_linear_long_mode+0x24d
19 0000002a`bc0fd290 00007ff7`d40340cf     wtf!BX_CPU_C::translate_linear+0x11a
1a 0000002a`bc0fd320 00007ff7`d4070744     wtf!BX_CPU_C::access_write_linear+0xcf
1b 0000002a`bc0fd370 00007ff7`d417f152     wtf!BX_CPU_C::write_linear_dword+0x134
1c 0000002a`bc0fd3c0 00007ff7`d417f29b     wtf!BX_CPU_C::MOV64_EdGdM+0xa2
1d 0000002a`bc0fd3f0 00007ff7`d417ff79     wtf!BX_CPU_C::MOV64_GdEdM+0xeb
1e 0000002a`bc0fd420 00007ff7`d40327f7     wtf!BX_CPU_C::MOV_GqEqM+0xe9
1f 0000002a`bc0fd450 00007ff7`d3e76320     wtf!BX_CPU_C::cpu_loop+0x127
20 0000002a`bc0fd480 00007ff7`d3f4be26     wtf!ZN8bochscpu3cpu6CpuRun3run17h68b2c636903acc81E+0xd0
21 0000002a`bc0fd4d0 00007ff7`d3f62b7c     wtf!BochscpuBackend_t::Run+0x96 [C:\Users\Kiwi\Desktop\wtf\src\wtf\bochscpu_backend.cc @ 394] 
22 0000002a`bc0fd510 00007ff7`d3f63373     wtf!RunTestcaseAndRestore+0x7c [C:\Users\Kiwi\Desktop\wtf\src\wtf\client.cc @ 111] 
23 0000002a`bc0fd5b0 00007ff7`d3f91dec     wtf!Client_t::Run+0x163 [C:\Users\Kiwi\Desktop\wtf\src\wtf\client.cc @ 244] 
24 0000002a`bc0fd690 00007ff7`d3fb585e     wtf!FuzzSubcommand+0x2c [C:\Users\Kiwi\Desktop\wtf\src\wtf\subcommands.cc @ 96] 
25 0000002a`bc0fd700 00007ff7`d4193ee8     wtf!main+0x23de [C:\Users\Kiwi\Desktop\wtf\src\wtf\wtf.cc @ 472] 
26 (Inline Function) --------`--------     wtf!invoke_main+0x22 [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 78] 
27 0000002a`bc0ff7a0 00007ff9`e69d26ad     wtf!__scrt_common_main_seh+0x10c [D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 288] 
28 0000002a`bc0ff7e0 00007ff9`e842aa68     KERNEL32!BaseThreadInitThunk+0x1d
29 0000002a`bc0ff810 00000000`00000000     ntdll!RtlUserThreadStart+0x28

[kiwids0220]

It seemed like BX_CPU is running into issues with

18 000000c1`59dcd5e0 00007ff7`d403582a     wtf!BX_CPU_C::translate_linear_long_mode+0x24d

[yrp604]

Okay, I understand whats happening now. When trying to deref some memory it caused a page fault -- this happens all the time. However, when trying to service that page fault, there was a second page fault putting the cpu into a double faulted state. Inside the cpu exception handler, it kicks the debugger into action on a double fault. Before the debugger can break in, it attempts to fetch the contents of rip to disassemble it, which hits the assert.

So, to resolve the assert we should patch bx to not disassemble instructions in the double fault case -- this will resolve the immediate bug youre hitting. However, the actual problem youre running into here is that youre double faulting when trying to service a page fault. This indicates that something is probably wrong with the dump or cpu state that you have. You can turn on debug builds of bxcpu to see exactly the memory that is causing the problem, which will give you some clues to resolving it.

While a double fault is recoverable, in the vast majority of cases this indicates that things are permanently off the rails. Basically if we fix this assert, youre going to emulate another 100 odd instructions before hitting a triple fault which will also crash the emulator.

[kiwids0220]

It makes sense now, I will try to set some break points and observe what's causing it (maybe a mistake on how I inserted testcases). I am truly thankful for all the help that you guys offered! Will update on this later!

[0vercl0k]

Circling back on this; @kiwids0220 is there something else we can help you with or are we good to close?

Cheers

[kiwids0220]

Circling back on this; @kiwids0220 is there something else we can help you with or are we good to close?

Cheers

I moved onto something else and yes we can close this one now