missing frames in 64-bit Windows callstacks with VS2017 ucrt

[milianw]

Take this trivial example code:

int *foo()
{
    new char[100];
    return new int(42);
}

int main()
{
    static int *bar = foo();

    return *bar;
}

Compile it (I use qmake, but these are the effective compile flags:)

        cl -c -nologo -Zc:wchar_t -FS -Zc:rvalueCast -Zc:inline -Zc:strictStrings -Zc:throwingNew -Zc:referenceBinding -Zi -MDd -W3 -w34100 -w34189 -w44996 -w44456 -w44457 -w44458 -wd4577 -wd4467 -EHsc /Fddebug\ex_leak.vc.pdb -DUNICODE -D_UNICODE -DWIN32 -DWIN64 -DQT_GUI_LIB -DQT_CORE_LIB -I..\..\ex_leak -I. -IC:\Qt\5.9.2\msvc2017_64\include -IC:\Qt\5.9.2\msvc2017_64\include\QtGui -IC:\Qt\5.9.2\msvc2017_64\include\QtANGLE -IC:\Qt\5.9.2\msvc2017_64\include\QtCore -Idebug -IC:\Qt\5.9.2\msvc2017_64\mkspecs\win32-msvc -Fodebug\ @C:\Users\milian\AppData\Local\Temp\main.obj.3992.0.jom
main.cpp
        link /NOLOGO /DYNAMICBASE /NXCOMPAT /DEBUG /SUBSYSTEM:WINDOWS "/MANIFESTDEPENDENCY:type='win32' name='Microsoft.Windows.Common-Controls' version='6.0.0.0' publicKeyToken='6595b64144ccf1df' language='*' processorArchitecture='*'" /MANIFEST:embed /OUT:debug\ex_leak.exe @C:\Users\milian\AppData\Local\Temp\ex_leak.exe.3992.109.jom

Then run it through drmemory and look at the leak report:

Error #1: LEAK 100 direct bytes 0x000002a8b63b0290-0x000002a8b63b02f4 + 0 indirect bytes
# 0 replace_operator_new_array                [d:\drmemory_package\common\alloc_replace.c:2928]
# 1 ucrtbased.dll!__crt_fast_decode_pointer<> [minkernel\crts\ucrt\devdiv\vcruntime\inc\internal_shared.h:498]
# 2 ucrtbased.dll!_initterm_e                 [minkernel\crts\ucrt\src\appcrt\startup\initterm.cpp:40]
# 3 KERNEL32.dll!BaseThreadInitThunk

Note how it does not show any frame in the file I wrote at all. Both main and foo are nowhere to be found. In a debugger like from Visual Studio, I see the frames properly when I step through the execution.

[milianw]

After updating to the Fall Creators update and installing an updated syscalls_x64.txt, I see an improved backtrace:

~~Dr.M~~ Error #1: LEAK 100 direct bytes 0x0000018f2c4e0290-0x0000018f2c4e02f4 + 0 indirect bytes
~~Dr.M~~ # 0 replace_operator_new_array               [d:\drmemory_package\common\alloc_replace.c:2928]
~~Dr.M~~ # 1 ntdll.dll!LdrResolveDelayLoadedAPI      +0x15b    (0x00007ffbd6ef1dfc <ntdll.dll+0x21dfc>)
~~Dr.M~~ # 2 _Init_thread_unlock                      [f:\dd\vctools\crt\vcstartup\src\misc\thread_safe_statics.cpp:149]
~~Dr.M~~ # 3 foo                                      [c:\qt\src\training-material\addon\debugging\ex_leak\main.cpp:9]
~~Dr.M~~ # 4 _Init_thread_header                      [f:\dd\vctools\crt\vcstartup\src\misc\thread_safe_statics.cpp:224]
~~Dr.M~~ # 5 ucrtbased.dll!_initterm_e                [minkernel\crts\ucrt\src\appcrt\startup\initterm.cpp:40]
~~Dr.M~~ # 6 KERNEL32.dll!BaseThreadInitThunk        +0x13     (0x00007ffbd5ce1fe4 <KERNEL32.dll+0x11fe4>)

I.e. foo shows up now, but main is still missing. Is this maybe a strange behavior of statics in Windows?

[milianw]

Removing the static from the initialization of bar in main, I instead again get:

~~Dr.M~~ Error #1: LEAK 4 direct bytes 0x00000197a8be0340-0x00000197a8be0344 + 0 indirect bytes
~~Dr.M~~ # 0 replace_operator_new                   [d:\drmemory_package\common\alloc_replace.c:2899]
~~Dr.M~~ # 1 ntdll.dll!LdrResolveDelayLoadedAPI    +0x15b    (0x00007ffbd6ef1dfc <ntdll.dll+0x21dfc>)
~~Dr.M~~ # 2 ucrtbased.dll!_initterm_e              [minkernel\crts\ucrt\src\appcrt\startup\initterm.cpp:40]
~~Dr.M~~ # 3 KERNEL32.dll!BaseThreadInitThunk      +0x13     (0x00007ffbd5ce1fe4 <KERNEL32.dll+0x11fe4>)

Sadly, no foo, no main.

[derekbruening]

I suspect this is limited to VS2017 ucrt, though in general efficient callstacks on Windows are painful, and the current scheme has been tuned for 32-bit. Many functions in system or crt libs have FPO, which can result in skipping higher frames. To counter this Dr. Memory has a lot of scanning options to find retaddrs w/o frame pointers. Going to debug info (as debuggers do) is considered too expensive, as callstacks need to be gathered on every single malloc in case it's leaked later.

What we should do for 64-bit Windows is walk the SEH64 unwind info which is #1222 but that's not implemented (if someone would like to contribute some work toward that feature...).

You can see a number of options here: http://drmemory.org/docs/page_options.html -callstack_conservative or -callstack_bad_fp_list may help.