Debugger::OnDyldImageNotifier signal SIGABRT on MacOS BigSur

[gloxec]

• MacOS BigSur 11.4(Intel), SIP disabled, root
• The latest version of TinyInst

Whether I use Jackalope or call instrumentation->Run(argc, argv, 1000) by myself, the same abort signal error will be generated

...
Process 4104 stopped
* thread #2, stop reason = signal SIGABRT
    frame #0: 0x00007fff2053192e libsystem_kernel.dylib`__pthread_kill + 10
libsystem_kernel.dylib`__pthread_kill:
->  0x7fff2053192e <+10>: jae    0x7fff20531938            ; <+20>
    0x7fff20531930 <+12>: mov    rdi, rax
    0x7fff20531933 <+15>: jmp    0x7fff2052bad9            ; cerror_nocancel
    0x7fff20531938 <+20>: ret
Target 0: (fuzzer) stopped.
(lldb) bt
* thread #2, stop reason = signal SIGABRT
  * frame #0: 0x00007fff2053192e libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff205605bd libsystem_pthread.dylib`pthread_kill + 263
    frame #2: 0x00007fff204b54ab libsystem_c.dylib`__abort + 139
    frame #3: 0x00007fff2048f979 libsystem_c.dylib`__stack_chk_fail + 100
    frame #4: 0x000000010006904e fuzzer`Debugger::OnDyldImageNotifier(unsigned long, unsigned long, unsigned long long*) + 958

At the same time, it was found that the started process produced a crash report:

...
OS Version:            macOS 11.4 (20F71)
Report Version:        12
Bridge OS Version:     3.0 (14Y908)

Time Awake Since Boot: 3200 seconds

System Integrity Protection: disabled

Crashed Thread:        0

Exception Type:        EXC_BREAKPOINT (SIGTRAP)
Exception Codes:       0x0000000000000002, 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY

Termination Signal:    Trace/BPT trap: 5
Termination Reason:    Namespace SIGNAL, Code 0x5
Terminating Process:   exc handler [4106]

Application Specific Information:
dyld: launch, loading dependent libraries

Thread 0 Crashed:
0   dyld                            0x0000000100019242 _dyld_debugger_notification + 1
1   dyld                            0x00000001000183cb gdb_image_notifier(dyld_image_mode, unsigned int, dyld_image_info const*) + 203
2   dyld                            0x00000001000180bf notifyGDB(dyld_image_states, unsigned int, dyld_image_info const*) + 40
3   dyld                            0x000000010000af4b dyld::notifyBatchPartial(dyld_image_states, bool, char const* (*)(dyld_image_states, unsigned int, dyld_image_info const*), bool, bool) + 1493
4   dyld                            0x000000010001a2b9 ImageLoader::link(ImageLoader::LinkContext const&, bool, bool, bool, ImageLoader::RPathChain const&, char const*) + 101
5   dyld                            0x000000010000b513 dyld::link(ImageLoader*, bool, bool, ImageLoader::RPathChain const&, unsigned int) + 383
6   dyld                            0x000000010000dc48 dyld::_main(macho_header const*, unsigned long, int, char const**, char const**, char const**, unsigned long*) + 8084
7   dyld                            0x0000000100006224 dyldbootstrap::start(dyld3::MachOLoaded const*, int, char const**, dyld3::MachOLoaded const*, unsigned long*) + 450
8   dyld                            0x0000000100006025 _dyld_start + 37

Thread 0 crashed with X86 Thread State (64-bit):
  rax: 0x000000000000002b  rbx: 0x00007ffeefbfe0b8  rcx: 0x00007fff2a6fe000  rdx: 0x00007ffeefbfd910
  rdi: 0x0000000000000000  rsi: 0x000000000000002b  rbp: 0x00007ffeefbfdb10  rsp: 0x00007ffeefbfd908
   r8: 0x0000000000000000   r9: 0x0000000000000000  r10: 0x0000000000000000  r11: 0x0000000000000000
  r12: 0x00007ffeefbfda78  r13: 0x00007ffeefbfe0d0  r14: 0x0000000000000000  r15: 0x000000000000002b
  rip: 0x0000000100019242  rfl: 0x0000000000000293  cr2: 0x0000000100059905

Logical CPU:     4
Error Code:      0x00000000
Trap Number:     3

Thread 0 instruction stream:
  8d 45 10 48 89 42 08 48-b8 08 00 00 00 30 00 00  .E.H.B.H.....0..
  00 48 89 02 bf 02 00 00-00 4c 89 d6 e8 6d 1b 04  .H.......L...m..
  00 48 8d 05 f6 ff 08 00-48 8b 00 48 3b 45 f8 75  .H......H..H;E.u
  09 48 81 c4 d0 00 00 00-5d c3 e8 47 ff ff ff 55  .H......]..G...U
  48 89 e5 48 8d 3d a3 87-07 00 e8 41 0f ff ff 55  H..H.=.....A...U
  48 89 e5 48 8d 3d a6 87-07 00 e8 31 0f ff ff cc  H..H.=.....1....
 [c3]89 e5 5d c3 55 48 89-e5 48 8d 3d b5 87 07 00  ...].UH..H.=.... <==
  e8 1b 0f ff ff 83 fe 08-75 01 c3 55 48 89 e5 48  ........u..UH..H
  8d 3d b6 87 07 00 e8 05-0f ff ff 48 85 ff 74 2b  .=.........H..t+
  45 31 c0 48 39 f1 48 0f-47 ce b8 54 00 00 00 41  E1.H9.H.G..T...A
  0f 46 c0 48 85 c9 74 12-55 48 89 e5 31 f6 88 14  .F.H..t.UH..1...
  37 48 ff c6 48 39 f1 75-f5 5d c3 b8 16 00 00 00  7H..H9.u.]......

Thread 0 last branch register state not available.

Binary Images:
       0x100000000 -        0x100001fff +test (0) <6DBAD331-A5DF-3E65-B927-A807B64BD9F3> /Users/USER/*/test
       0x100005000 -        0x1000a0fff  dyld (852) <1AC76561-4F9A-34B1-BA7C-4516CACEAED7> /usr/lib/dyld
    0x7fff20298000 -     0x7fff20299fff  libsystem_blocks.dylib (79) <48AF56A9-6E42-3A5E-A213-E6AFD8F81044> /usr/lib/system/libsystem_blocks.dylib
...

[ifratric]

Hi, on my machine with BigSur 11.4 I don't see a crash like that. Could you fetch the latest TinyInst/Jackalope, rebuild and try the test Jackalope command from the Readme:

./fuzzer -in in -out out -t 1000 -delivery shmem -instrument_module test -target_module test -target_method __Z4fuzzPc -nargs 1 -iterations 10000 -persist -loop -cmp_coverage -- ./test -m @@

(you'll need to create the in directory with at least one file).

Does this still result in the same crash?

[gloxec]

Yes, TinyInst is the latest version from the repository, and ./fuzzer was able to run normally before.

But now neither the old version of the fuzzer compiled when running on MacOS Catalina (the old Jackalope and TinyInst without Arm64 debugger) nor the recompiled new version of Jackalope and TinyInst can not run.

It stands to reason that even if there is a problem with the code, the fuzzer program originally compiled in MacOS Catalina should still run normally after upgrading BigSur, but the real situation is that the above-mentioned error is reported.

At the same time, the same error was reported in the program I wrote locally. Combined with the lldb prompt, I guess whether BigSur updated the dyld and caused the code to fail to run normally?

TinyInst code used by the local program:
instrumentation->Run(argc, argv, 1000)

[ifratric]

Can you confirm that even the test binary crashes (command above)?

Could you run with -trace_debug_events and post the output?

[gloxec]

Yes, running the above command, both programs ./fuzzer & test will crash.

./fuzzer -in in -out out -t 1000 -delivery shmem -instrument_module test -target_module test -target_method __Z4fuzzPc -nargs 1 -iterations 10000 -persist -loop -cmp_coverage -trace_debug_events -- ./test -m @@
Fuzzer version 0.01
1 input files read
Running input sample in/seed-0
Debugger: Mach exception (5) @ address 0x113215000
Debugger: Process created or attached
Debugger: Loaded module /usr/lib/dyld at 0x113214000
Debugger: Loaded module test at 0x10ce2c000
Debugger: Loaded module libc++.1.dylib at 0x7fff204be000
Debugger: Loaded module libSystem.B.dylib at 0x7fff2a731000
Debugger: Loaded module libc++abi.dylib at 0x7fff20514000
Debugger: Loaded module libcache.dylib at 0x7fff2a72b000
Debugger: Loaded module libcommonCrypto.dylib at 0x7fff2a6e7000
Debugger: Loaded module libcompiler_rt.dylib at 0x7fff2a711000
Debugger: Loaded module libcopyfile.dylib at 0x7fff2a706000
Debugger: Loaded module libcorecrypto.dylib at 0x7fff202e8000
Debugger: Loaded module libdispatch.dylib at 0x7fff203b3000
Debugger: Loaded module libdyld.dylib at 0x7fff20566000
Debugger: Loaded module libkeymgr.dylib at 0x7fff2a722000
Debugger: Loaded module liblaunch.dylib at 0x7fff2db38000
Debugger: Loaded module libmacho.dylib at 0x7fff2a6c5000
Debugger: Loaded module libquarantine.dylib at 0x7fff29ebd000
Debugger: Loaded module libremovefile.dylib at 0x7fff2a71f000
Debugger: Loaded module libsystem_asl.dylib at 0x7fff24dc2000
Debugger: Loaded module libsystem_blocks.dylib at 0x7fff20298000
Debugger: Loaded module libsystem_c.dylib at 0x7fff20435000
Debugger: Loaded module libsystem_collections.dylib at 0x7fff2a719000
Debugger: Loaded module libsystem_configuration.dylib at 0x7fff291c3000
Debugger: Loaded module libsystem_containermanager.dylib at 0x7fff284ab000
Debugger: Loaded module libsystem_coreservices.dylib at 0x7fff2a468000
Debugger: Loaded module libsystem_darwin.dylib at 0x7fff229a1000
Debugger: Loaded module libsystem_dnssd.dylib at 0x7fff2a723000
Debugger: Loaded module libsystem_featureflags.dylib at 0x7fff20432000
Debugger: Loaded module libsystem_info.dylib at 0x7fff205ac000
Debugger: Loaded module libsystem_m.dylib at 0x7fff2a67c000
Debugger: Loaded module libsystem_malloc.dylib at 0x7fff20386000
Debugger: Loaded module libsystem_networkextension.dylib at 0x7fff24d55000
Debugger: Loaded module libsystem_notify.dylib at 0x7fff22dc0000
Debugger: Loaded module libsystem_product_info_filter.dylib at 0x7fff2ffc7000
Debugger: Loaded module libsystem_sandbox.dylib at 0x7fff291c7000
Debugger: Loaded module libsystem_secinit.dylib at 0x7fff2a71c000
Debugger: Loaded module libsystem_kernel.dylib at 0x7fff2052a000
Debugger: Loaded module libsystem_platform.dylib at 0x7fff205a2000
Debugger: Loaded module libsystem_pthread.dylib at 0x7fff2055a000
Debugger: Loaded module libsystem_symptoms.dylib at 0x7fff2649d000
Debugger: Loaded module libsystem_trace.dylib at 0x7fff202d0000
Debugger: Loaded module libunwind.dylib at 0x7fff2a6f3000
Debugger: Loaded module libxpc.dylib at 0x7fff2029a000
Debugger: Loaded module libobjc.A.dylib at 0x7fff203f8000
Debugger: Loaded module liboah.dylib at 0x7fff2a6fe000
[1]    6966 abort      ./fuzzer -in in -out out -t 1000 -delivery shmem -instrument_module test  tes

[ifratric]

Hmm, that's really strange. According to your original post, TinyInst crashes in Debugger::OnDyldImageNotifier, but that function appears to be working normally (at least in most of the cases), otherwise we wouldn't see all those "Loaded module" notifications.

Since I can't reproduce, could you add the following line at the start of Debugger::OnDyldImageNotifier:

printf("OnDyldImageNotifier %zd %lu %p\n", mode, infoCount, machHeaders);

re-run and post the output?

[gloxec]

OnDyldImageNotifier 0 43 0x7ffee4575970

./fuzzer -in in -out out -t 1000 -delivery shmem -instrument_module test -target_module test -target_method __Z4fuzzPc -nargs 1 -iterations 10000 -persist -loop -cmp_coverage -trace_debug_events -- ./test -m @@

Fuzzer version 0.01
1 input files read
Running input sample in/seed-0
Debugger: Mach exception (5) @ address 0x11366d000
Debugger: Process created or attached
Debugger: Loaded module /usr/lib/dyld at 0x11366c000
OnDyldImageNotifier 0 43 0x7ffee4575970
Debugger: Loaded module test at 0x10b688000
Debugger: Loaded module libc++.1.dylib at 0x7fff204be000
Debugger: Loaded module libSystem.B.dylib at 0x7fff2a731000
Debugger: Loaded module libc++abi.dylib at 0x7fff20514000
Debugger: Loaded module libcache.dylib at 0x7fff2a72b000
Debugger: Loaded module libcommonCrypto.dylib at 0x7fff2a6e7000
Debugger: Loaded module libcompiler_rt.dylib at 0x7fff2a711000
Debugger: Loaded module libcopyfile.dylib at 0x7fff2a706000
Debugger: Loaded module libcorecrypto.dylib at 0x7fff202e8000
Debugger: Loaded module libdispatch.dylib at 0x7fff203b3000
Debugger: Loaded module libdyld.dylib at 0x7fff20566000
Debugger: Loaded module libkeymgr.dylib at 0x7fff2a722000
Debugger: Loaded module liblaunch.dylib at 0x7fff2db38000
Debugger: Loaded module libmacho.dylib at 0x7fff2a6c5000
Debugger: Loaded module libquarantine.dylib at 0x7fff29ebd000
Debugger: Loaded module libremovefile.dylib at 0x7fff2a71f000
Debugger: Loaded module libsystem_asl.dylib at 0x7fff24dc2000
Debugger: Loaded module libsystem_blocks.dylib at 0x7fff20298000
Debugger: Loaded module libsystem_c.dylib at 0x7fff20435000
Debugger: Loaded module libsystem_collections.dylib at 0x7fff2a719000
Debugger: Loaded module libsystem_configuration.dylib at 0x7fff291c3000
Debugger: Loaded module libsystem_containermanager.dylib at 0x7fff284ab000
Debugger: Loaded module libsystem_coreservices.dylib at 0x7fff2a468000
Debugger: Loaded module libsystem_darwin.dylib at 0x7fff229a1000
Debugger: Loaded module libsystem_dnssd.dylib at 0x7fff2a723000
Debugger: Loaded module libsystem_featureflags.dylib at 0x7fff20432000
Debugger: Loaded module libsystem_info.dylib at 0x7fff205ac000
Debugger: Loaded module libsystem_m.dylib at 0x7fff2a67c000
Debugger: Loaded module libsystem_malloc.dylib at 0x7fff20386000
Debugger: Loaded module libsystem_networkextension.dylib at 0x7fff24d55000
Debugger: Loaded module libsystem_notify.dylib at 0x7fff22dc0000
Debugger: Loaded module libsystem_product_info_filter.dylib at 0x7fff2ffc7000
Debugger: Loaded module libsystem_sandbox.dylib at 0x7fff291c7000
Debugger: Loaded module libsystem_secinit.dylib at 0x7fff2a71c000
Debugger: Loaded module libsystem_kernel.dylib at 0x7fff2052a000
Debugger: Loaded module libsystem_platform.dylib at 0x7fff205a2000
Debugger: Loaded module libsystem_pthread.dylib at 0x7fff2055a000
Debugger: Loaded module libsystem_symptoms.dylib at 0x7fff2649d000
Debugger: Loaded module libsystem_trace.dylib at 0x7fff202d0000
Debugger: Loaded module libunwind.dylib at 0x7fff2a6f3000
Debugger: Loaded module libxpc.dylib at 0x7fff2029a000
Debugger: Loaded module libobjc.A.dylib at 0x7fff203f8000
Debugger: Loaded module liboah.dylib at 0x7fff2a6fe000
[1]    7814 abort      ./fuzzer -in in -out out -t 1000 -delivery shmem -instrument_module test  tes

[ifratric]

Yep, that's the same output that I'm getting, the arguments seem to be correct, the function appears to be working correctly all the way to the end (it processes all 43 modules), so no idea why it would crash.

Could you maybe create a Debug build of Jackalope/TinyInst and run it from Xcode? Perhaps it will be able to tell us something new?

[gloxec]

The result is very strange. Whether I run the fuzzer directly, or set its scheme to be rooted and run with those parameters, or set a breakpoint at main, I will enter the dyld error. Thread 1: EXC_BAD_ACCESS (code=2, address=0x7fff239c0879

[ifratric]

So it doesn't even reach main then? If so, this behavior is likely unrelated to the OnDyldImageNotifier issue (or, if they are related, it seems unrelated to the TinyInst code).

[ifratric]

What version of Xcode are you running?

[ifratric]

Did you manage to resolve the issues? My suspicion is that updating Xcode and rebuilding the project will resolve them.

[gloxec]

Yes!

What version of Xcode are you running?

Xcode 11.3.1

After updating Xcode(12.5), TinyInst started to work normally whether it was compiled with Xcode or command-line. Could the above problems be caused by BigSur's minor adjustments to MachO?

Therefore, the fuzzer compiled by Catalina cannot be used normally after upgrading to BigSur, and the fuzzer cannot be used after recompiling the fuzzer by Xcode in Catalina. Only after upgrading to BigSur's Xcode, the fuzzer compiled by clang can run normally.

[ifratric]

Awesome! Yes, looks like the declarations/size of some dyld-related structures changed between Catalina and Big Sur, so it requires a rebuild with up-to-date headers.