CRASH: using drsym_init causes DR assert+crash when running windows 32-bit program

yutaow97 commented 1 year ago

Describe the bug Using drsym_init causes application crash when running windows 32-bit program as shown below. The "a.exe" is a 32-bit program that prints out "hello world". The crash occurs only when drsym_init is used.

To Reproduce Steps to reproduce the behavior:

Pointer to a minimized application (ideally the source code for it and instructions on which toolchain it was built with).
Precise command line for running the application.
Exact output or incorrect behavior.

application source code

#include <stdio.h>

#include "dr_api.h"
#include "drmgr.h"
#include "drx.h"
#include "drreg.h"
#include "drsyms.h"

DR_EXPORT void dr_client_main(client_id_t id, int argc, const char *argv[])
{
    dr_set_client_name("test", "test");
    drsym_init(0);
}

CMakeLists.txt

cmake_minimum_required(VERSION 3.12.1)
project(CoverageMonitor)

find_package(DynamoRIO)
if (NOT DynamoRIO_FOUND)
  message(FATAL_ERROR "DynamoRIO package required to build")
endif(NOT DynamoRIO_FOUND)

set(srcs
    test.c
)

add_library(test SHARED ${srcs})

configure_DynamoRIO_client(test)
use_DynamoRIO_extension(test drmgr)
use_DynamoRIO_extension(test drx)
use_DynamoRIO_extension(test drcontainers)
use_DynamoRIO_extension(test drreg)
use_DynamoRIO_extension(test drwrap)
use_DynamoRIO_extension(test drsyms)

commands used to build

cmake  -A Win32 -DDynamoRIO_DIR=C:\Users\19730\projects\tools\DynamoRIO-Windows-9.92.19454\DynamoRIO-Windows-9.92.19454\cmake ..
cmake --build . --config Debug

run

C:\Users\19730\projects\tools\DynamoRIO-Windows-9.92.19454\DynamoRIO-Windows-9.92.19454\bin32\drrun.exe -c .\Debug\test.dll -- C:\Users\19730\projects\tools\dynamorio_issue_env\hello_word_32\a.exe

crash log

<Application C:\Users\19730\projects\tools\dynamorio_issue_env\hello_word_32\a.exe (13760).  test internal crash at PC 0x0144e4f2.  Please report this at test.  Program aborted.
0xc06d007f 0x00000000 0x0144e4f2 0x0144e4f2 0x00eff410 0x00000000
Base: 0x75110000
Registers: eax=0x00eff390 ebx=0x00000000 ecx=0x00000001 edx=0x00000000
        esi=0x01320000 edi=0x00000000 esp=0x00eff390 ebp=0x00eff3ec
        eflags=0x00000216
version 9.92.19454, custom build
-no_dynamic_options -client_lib 'C:\Users\19730\projects\tools\dynamorio_issue_env\build\Debug\test.dll;0;' -client_lib32 'C:\Users\19730\projects\tools\dynamorio_issue_env\build\Debug\test.dll;0;' -code_api -probe_api -stack_size 56K -max_elide_jmp 0 -max_elide_call 0 -no_inline_ignored_syscalls -native_exec_default_lis
0x00eff3ec 0x74ef23de
0x00eff444 0x74ef3b4c
0x00eff4a0 0x74e8c593
0x00eff4cc 0x74ed4d37
0x00eff500 0x74eacf0f
0x00eff518 0x74eae079
0x00eff794 0x74eae23e
0x00eff7b4 0x75392663
0x752465f8 0x753e0000
0x753e0000 0x00000003>

Please also answer these questions:

What happens when you run without any client? Running without any client is just fine.
What happens when you run with debug build ("-debug" flag to drrun/drconfig/drinject)?

Expected behavior The console prints out "hello world"

Versions

What version of DynamoRIO are you using? DynamoRIO-Windows-9.92.19454
Does the latest build from https://github.com/DynamoRIO/dynamorio/releases solve the problem? No

What operating system version are you running on? ("Windows 10" is not sufficient: give the release number.) Windows 10 19045.2728

lihasgupta commented 1 year ago

Hi. We need more information to help. I see that you encountered a crash. Could you provide us with a callstack of the crash?

Our debug page has instructions on how to get Crash Callstacks on Windows: https://dynamorio.org/page_debugging.html#autotoc_md156

yutaow97 commented 1 year ago

Hi. Here is the callstack.

dazhouzhou commented 1 year ago

Same problem, crash on dynamorio/core/win32/loader.c:771 ASSERT(!is_dynamo_address(dcontext->app_fls_data)).

derekbruening commented 1 year ago

@yutaow97: The request is for a symbolized callstack with function names of the crash; what you provided is instead an assert, not a crash, and with no function names. Probably the assert is debug build and the crash is release build; they may be two different issues; symbolized callstacks of both would be good.

What is labeled "application" above is really the client; which application(s) this happens on was never described.

yutaow97 commented 1 year ago

@yutaow97: The request is for a symbolized callstack with function names of the crash; what you provided is instead an assert, not a crash, and with no function names. Probably the assert is debug build and the crash is release build; they may be two different issues; symbolized callstacks of both would be good.

What is labeled "application" above is really the client; which application(s) this happens on was never described.

Here is the callstack of assert from the debug build: dynamorio/core/win32/loader.c:771 ASSERT(!is_dynamo_address(dcontext->app_fls_data))

1:004> kv
 # ChildEBP RetAddr      Args to Child              
00 181f1010 72c1a0c2     72d2e6f0 00000303 72c7108c dynamorio!d_r_internal_error+0x13 (FPO: [Non-Fpo]) (CONV: cdecl) [C:\Users\xzz\Desktop\tools\dynamorio\core\utils.c @ 145] 
01 181f107c 72c144e3     181b4080 00000001 ffffffff dynamorio!swap_peb_pointer_ex+0x702 (FPO: [Non-Fpo]) (CONV: cdecl) [C:\Users\xzz\Desktop\tools\dynamorio\core\win32\loader.c @ 771] 
02 181f1090 72bd4c4f     181b4080 00000001 00000288 dynamorio!swap_peb_pointer+0x13 (FPO: [Non-Fpo]) (CONV: cdecl) [C:\Users\xzz\Desktop\tools\dynamorio\core\win32\loader.c @ 793] 
03 181f1620 72bd4ff2     181b4080 005de1a8 005de158 dynamorio!client_exception_event+0x11f (FPO: [Non-Fpo]) (CONV: cdecl) [C:\Users\xzz\Desktop\tools\dynamorio\core\win32\callback.c @ 5097] 
04 181f167c 72bd6576     181b4080 005de1a8 005de158 dynamorio!check_internal_exception+0x112 (FPO: [Non-Fpo]) (CONV: cdecl) [C:\Users\xzz\Desktop\tools\dynamorio\core\win32\callback.c @ 5185] 
05 181f1d64 72d68016     181f1d70 00000000 72d680a0 dynamorio!intercept_exception+0x1136 (FPO: [Non-Fpo]) (CONV: cdecl) [C:\Users\xzz\Desktop\tools\dynamorio\core\win32\callback.c @ 5553] 
06 181f1d6c 00000000     72d680a0 00000000 00f90000 dynamorio!interception_code_array+0x1016

In the release build, the crash occurs in "dbghelp.dll" as shown below, which raises the exception (0x0C06D007Fh)

dbghelp!__delayLoadHelper2+0x25b:
72972910 8945f0          mov     dword ptr [ebp-10h],eax
72972913 e857fdffff      call    dbghelp!DloadReleaseSectionWriteAccess (7297266f)
72972918 8d45f0          lea     eax,[ebp-10h]
7297291b 50              push    eax
7297291c 6a01            push    1
7297291e 53              push    ebx
7297291f 687f006dc0      push    0C06D007Fh ; exception code
72972924 ff153860a972    call    dword ptr [dbghelp!_imp__RaiseException (72a96038)] ; raise exception

Here is the callstack of the crash:

dbghelp!__delayLoadHelper2+0x275
dbghelp!_tailMerge_api_ms_win_eventing_provider_l1_1_0_dll+0xd
dbghelp!TraceLoggingRegister_DbgToolsSafe_EventRegister_DbgToolsSafe_EventSetInformation+0x52
dbghelp!DbgToolsSafe_TraceLoggingRegister+0x5f
dbghelp!DbgToolsImplicitRegisterProviderCheck+0x35
dbghelp!SymInitializeW+0x9f
dbghelp!SymInitialize+0x1e
drsyms!drsym_init+0x75 [C:\Users\xzz\Desktop\tools\dynamorio\ext\drsyms\drsyms_windows.c @ 207]
test!dr_client_main+0x16

dynamorio commit id: 0e86f7948bd7a791c9f628e2a0932cc5c08de88b

derekbruening commented 1 year ago

The callstack of the assert in debug build shows it is a secondary effect happening after an exception already happened: what is the callstack of that exception? (The CONTEXT is a local var there; also good to run .exr on the exception record.). Maybe it is the same dbghelp crash as in release build.

dbghelp!SymInitialize crashing is not good.

Does this only happen with certain versions of dbghelp?

yutaow97 commented 1 year ago

The callstack of the assert in debug build shows it is a secondary effect happening after an exception already happened: what is the callstack of that exception? (The CONTEXT is a local var there; also good to run .exr on the exception record.). Maybe it is the same dbghelp crash as in release build.

dbghelp!SymInitialize crashing is not good.

Does this only happen with certain versions of dbghelp?

I'm using the latest Windows 10, and here is the version of dbghelp:

1:005> lmDva 0x7297292a
Browse full module list
start    end        module name
728e0000 72aaf000   dbghelp    (pdb symbols)          c:\symbols\dbghelp.pdb\411A17409C597925B4ED8DD6A965F3D71\dbghelp.pdb
    Loaded symbol image file: dbghelp.dll
    Image path: dbghelp.dll
    Image name: dbghelp.dll
    Browse all global symbols  functions  data
    Image was built with /Brepro flag.
    Timestamp:        60A29371 (This is a reproducible build file hash, not a timestamp)
    CheckSum:         001BCF33
    ImageSize:        001CF000
    File version:     10.0.22621.755
    Product version:  10.0.22621.755
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    Information from resource tables:
        CompanyName:      Microsoft Corporation
        ProductName:      Microsoft® Windows® Operating System
        InternalName:     DBGHELP.DLL
        OriginalFilename: DBGHELP.DLL
        ProductVersion:   10.0.22621.755
        FileVersion:      10.0.22621.755 (WinBuild.160101.0800)
        FileDescription:  Windows Image Helper
        LegalCopyright:   © Microsoft Corporation. All rights reserved.

yutaow97 commented 1 year ago

@derekbruening Do you have any solutions?

derekbruening commented 1 year ago

I would suggest either tracking down why dbghelp is crashing or if there is a different version of dbghelp that works (there must be as our regression tests on Github Actions Server 2019 machines pass) that would be a workaround.

yutaow97 commented 1 year ago

Tested on Windows Server 2019 build 17763.4377, still have the problem. @derekbruening

DynamoRIO / dynamorio

CRASH: using drsym_init causes DR assert+crash when running windows 32-bit program #5966