search

ivmai / bdwgc

The Boehm-Demers-Weiser conservative C/C++ Garbage Collector (bdwgc, also known as bdw-gc, boehm-gc, libgc)
https://www.hboehm.info/gc/
Other
2.98k stars 407 forks source link

gctest fail: KERN_PROTECTION_FAILURE while world is stopped (OSX) #178

Open ivmai opened 7 years ago

ivmai commented 7 years ago

Build: https://travis-ci.org/ivmai/bdwgc/jobs/269108337 Compiler: clang-703.0.31 OS (target): x86_64-apple-darwin15.6.0 Build command: ./configure --enable-cplusplus && make check Output: Switched to incremental mode Emulating dirty bits with mprotect/signals KERN_PROTECTION_FAILURE while world is stopped FAIL gctest (exit status: 138)

I failed to reproduce it on a local Mac.

PS. The code which prints this message was added in gc v6.2 (111a44f98)

ivmai commented 6 years ago

Build failed (recent master): https://travis-ci.org/ivmai/bdwgc/jobs/308863227

ivmai commented 3 years ago

No more reproduced (on Travis CI) since 2017

ivmai commented 2 years ago

Reproduced again: https://app.travis-ci.com/github/ivmai/bdwgc/jobs/552874298 Source: release-8_2 (commit 6bbc156) Config: CONF_OPTIONS="--enable-large-config --enable-cplusplus --disable-handle-fork"

ivmai commented 2 years ago

As reported (for v8.2.2) in https://github.com/NixOS/nixpkgs/pull/189020#issuecomment-1250131320 by @stephank:

The test consistently runs fine on a native x86-64_darwin machine, but fails sporadically on an M1 targetting Intel. (It does sometimes succeed.) $ ./gctest Switched to incremental mode Emulating dirty bits with mprotect/signals KERN_PROTECTION_FAILURE while world is stopped Bus error: 10

ivmai commented 2 years ago

Source: master (6dfa9a2) Build: https://app.travis-ci.com/github/ivmai/bdwgc/jobs/583732448 Host: Mac OS X 10.13.6 Compiler: Apple LLVM version 9.1.0 (clang-902.0.39.2) Target: x86_64-apple-darwin17.7.0 How to build: ./configure --enable-large-config --enable-cplusplus --disable-handle-fork && make -j check Output (gctest.log):

Switched to incremental mode
Emulating dirty bits with mprotect/signals
KERN_PROTECTION_FAILURE while world is stopped
ivmai commented 1 year ago

Source: master (0e63b39) Host: Mac OS X 10.13.6 Compiler: Apple LLVM version 9.1.0 (clang-902.0.39.2) Build: https://app.travis-ci.com/github/ivmai/bdwgc/jobs/596019045 How to build: ./configure --without-libatomic-ops --enable-munmap --enable-cplusplus --enable-static && make -j check CFLAGS_EXTRA="-march=native -D _FORTIFY_SOURCE=2 -D AO_DISABLE_GCC_ATOMICS"

ivmai commented 1 year ago

Source: master (b49d1ae) Build: https://app.travis-ci.com/github/ivmai/bdwgc/jobs/597606416 How to build: ./configure --enable-cplusplus && make -j check

ivmai commented 1 year ago

Source: release-8_2 (ee37e656c) Build: https://app.travis-ci.com/github/ivmai/bdwgc/jobs/601222935 Config: CFLAGS_EXTRA="-m32" CONF_OPTIONS="--enable-gc-assertions --enable-cplusplus"

ivmai commented 1 year ago

I failed to reproduce locally on: Mac-mini (Darwin Kernel Version 20.1.0) x86_64

ivmai commented 1 year ago

I failed to reproduce it locally (e.g. on release-8_2 (ee37e656c1) or master (2bcec83b6e)) on gcc104 host (Mac mini 2020, arm64, Apple M1, 8 cores)

ivmai commented 1 year ago

https://github.com/NixOS/nixpkgs/pull/189020#issuecomment-1250209121 @stephank, I failed to reproduce the failure locally (I tried several Apple PCs). If you still observe the failure (on release-8_2 or master branch), I could spend some time investigating it if you give ssh access to that host.

ivmai commented 1 year ago

Source: release-8_2 (0c12d15) Build: https://app.travis-ci.com/github/ivmai/bdwgc/jobs/602925318 gctest failed the the same error

ivmai commented 1 year ago

Source: master (43ef26f) Build: https://app.travis-ci.com/github/ivmai/bdwgc/jobs/611289510 Config: CMAKE_OPTIONS="-DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF -Denable_cplusplus=ON -Denable_gc_assertions=ON -Denable_large_config=ON"

ivmai commented 11 months ago

Source: master (88e6950e8) Build: https://github.com/ivmai/bdwgc/actions/runs/6801949243/job/18493924728 OS: macOS 12.6.9 Config: CMAKE_OPTIONS="-DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON"

ivmai commented 11 months ago

Source: master (ba6443b0a) Build: https://github.com/ivmai/bdwgc/actions/runs/6840396590/job/18599611865 Config: macos-latest clang thr:on rwlock::off redir:off Host: macOS 12.6.9

ivmai commented 11 months ago

Cannot reproduce it locally on Mac x64 Source: master (f2630a425)

ivmai commented 10 months ago

Source: master (f9a0ee08) Build: https://github.com/ivmai/bdwgc/actions/runs/7096934552/job/19316234315

ivmai commented 10 months ago

@coolbikerdad Maybe you could investigate and fix this issue too? If you could reproduce it on your host. Just in case. (It happens quite frequently on CI, but I can't catch in my OS X hosts.)

coolbikerdad commented 10 months ago

@coolbikerdad Maybe you could investigate and fix this issue too? If you could reproduce it on your host. Just in case. (It happens quite frequently on CI, but I can't catch in my OS X hosts.)

I will take a look. On my M1 Mac Studio I got 5 gctest failures in 11,000 runs, so I will now try to capture a core dump.

ivmai commented 10 months ago

Great!

coolbikerdad commented 10 months ago

@coolbikerdad Maybe you could investigate and fix this issue too? If you could reproduce it on your host. Just in case. (It happens quite frequently on CI, but I can't catch in my OS X hosts.)

I will take a look. On my M1 Mac Studio I got 5 gctest failures in 11,000 runs, so I will now try to capture a core dump.

It looks like I may getting different causes of Bus error. I have gctest creating a core dump and the backtraces looks like this:

➜ [](base) neil@BETH-STUDIO bdwgc lldb --core /cores/core.35260
(lldb) target create --core "/cores/core.35260"
warning: gctest was compiled with optimization - stepping may behave oddly; variables may not be available.
Core file '/cores/core.35260' (arm64) was loaded.
(lldb) bt
* thread #18, stop reason = ESR_EC_DABORT_EL0 (fault address: 0x102c204d0)
  * frame #0: 0x0000000102a6c438 gctest`mktree(n=15) at gctest.c:1075:22 [opt]
    frame #1: 0x0000000102a6c424 gctest`mktree(n=16) at gctest.c:1074:31 [opt]
    frame #2: 0x0000000102a6bec8 gctest`tree_test at gctest.c:1304:12 [opt]
    frame #3: 0x0000000102a6ba8c gctest`run_one_test at gctest.c:1895:5 [opt]
    frame #4: 0x0000000102a6b160 gctest`thr_run_one_test(arg=<unavailable>) at gctest.c:2588:5 [opt]
    frame #5: 0x0000000102fcb674 libgc.1.dylib`GC_pthread_start_inner(sb=<unavailable>, arg=<unavailable>) at pthread_start.c:55:12 [opt]
    frame #6: 0x0000000102fc1f44 libgc.1.dylib`GC_pthread_start [inlined] GC_call_with_stack_base(fn=(libgc.1.dylib`GC_pthread_start_inner at pthread_start.c:45), arg=<unavailable>) at misc.c:2237:14 [opt]
    frame #7: 0x0000000102fc1f34 libgc.1.dylib`GC_pthread_start(arg=<unavailable>) at pthread_support.c:2506:14 [opt]
    frame #8: 0x00000001854d5034 libsystem_pthread.dylib`_pthread_start + 136

➜ [](base) neil@BETH-STUDIO bdwgc lldb --core /cores/core.98733
(lldb) target create --core "/cores/core.98733"
warning: libobjc.A.dylib is being read from process memory. This indicates that LLDB could not read from the host's in-memory shared cache. This will likely reduce debugging performance.

Core file '/cores/core.98733' (arm64) was loaded.
(lldb) bt
* thread #1
  * frame #0: 0x000000018549d11c libsystem_kernel.dylib`__pthread_kill + 8
    frame #1: 0x00000001854d4cc0 libsystem_pthread.dylib`pthread_kill + 288
    frame #2: 0x00000001853e4a40 libsystem_c.dylib`abort + 180
    frame #3: 0x0000000104ed79f0 libgc.1.dylib`GC_printf.cold.1 at misc.c:1882:11 [opt]
    frame #4: 0x0000000104ebf5e0 libgc.1.dylib`GC_printf(format=<unavailable>) at misc.c:1882:11 [opt]
    frame #5: 0x0000000104976d08 gctest`main [inlined] check_heap_stats at gctest.c:2132:5 [opt]
    frame #6: 0x00000001049769c8 gctest`main at gctest.c:2742:5 [opt]
    frame #7: 0x00000001851590e0 dyld`start + 2360
(lldb)

➜ [](base) neil@BETH-STUDIO bdwgc lldb --core /cores/core.37570
(lldb) target create --core "/cores/core.37570"
warning: gctest was compiled with optimization - stepping may behave oddly; variables may not be available.
Core file '/cores/core.37570' (arm64) was loaded.
(lldb) bt
* thread #1, stop reason = ESR_EC_DABORT_EL0 (fault address: 0x102ff7000)
  * frame #0: 0x0000000102eec270 gctest`typed_test at gctest.c:1447:17 [opt]
    frame #1: 0x0000000102eeba44 gctest`run_one_test at gctest.c:1883:7 [opt]
    frame #2: 0x0000000102eea908 gctest`main at gctest.c:2728:9 [opt]
    frame #3: 0x000000018b9010e0 dyld`start + 2360
ivmai commented 10 months ago

It looks like I may getting different causes of Bus error.

There are not KERN_PROTECTION_FAILURE ones, right?

warning: gctest was compiled with optimization

If possible, please recompile w/o optimizations: ... && make check CFLAGS_EXTRA="-O0 -g"

coolbikerdad commented 10 months ago

I have caught several KERN_PROTECTION_FAILUREs on my M1 Mac Studio. I added a call to backtrace() in gctest.c and got this in all of them so far (addresses vary but the symbols/offset are the same in each):

KERN_PROTECTION_FAILURE while world is stopped, pid = 19134
0   libgc.1.dylib                       0x0000000102c744a4 catch_exception_raise + 456
1   libsystem_kernel.dylib              0x000000018bc5fc24 _Xexception_raise + 192
2   libsystem_kernel.dylib              0x000000018bc5fcc4 exc_server + 96
3   libgc.1.dylib                       0x0000000102c7edc4 GC_mprotect_thread + 444
4   libsystem_pthread.dylib             0x000000018bc82034 _pthread_start + 136
5   libsystem_pthread.dylib             0x000000018bc7ce3c thread_start + 8

I have several core dumps also:

➜ [](base) neil@BETH-STUDIO bdwgc lldb --core /cores/core.19134
(lldb) target create --core "/cores/core.19134"
warning: gctest was compiled with optimization - stepping may behave oddly; variables may not be available.
Core file '/cores/core.19134' (arm64) was loaded.
(lldb) bt
* thread #1, stop reason = ESR_EC_DABORT_EL0 (fault address: 0x102a6e000)
  * frame #0: 0x00000001027202fc gctest`typed_test at gctest.c:1447:17 [opt]
    frame #1: 0x000000010271fad0 gctest`run_one_test at gctest.c:1883:7 [opt]
    frame #2: 0x000000010271e994 gctest`main at gctest.c:2707:9 [opt]
    frame #3: 0x000000018b9010e0 dyld`start + 2360
(lldb)

➜ [](base) neil@Beth-Studio bdwgc lldb --core /cores/core.22031
(lldb) target create --core "/cores/core.22031"
Core file '/cores/core.22031' (arm64) was loaded.
warning: gctest was compiled with optimization - stepping may behave oddly; variables may not be available.
(lldb) bt
* thread #21, stop reason = ESR_EC_DABORT_EL0 (fault address: 0x103739000)
  * frame #0: 0x0000000102f8c2fc gctest`typed_test at gctest.c:1447:17 [opt]
    frame #1: 0x0000000102f8bad0 gctest`run_one_test at gctest.c:1883:7 [opt]
    frame #2: 0x0000000102f8b1ec gctest`thr_run_one_test(arg=<unavailable>) at gctest.c:2588:5 [opt]
    frame #3: 0x00000001034eb634 libgc.1.dylib`GC_pthread_start_inner(sb=<unavailable>, arg=<unavailable>) at pthread_start.c:55:12 [opt]
    frame #4: 0x00000001034e1f04 libgc.1.dylib`GC_pthread_start [inlined] GC_call_with_stack_base(fn=(libgc.1.dylib`GC_pthread_start_inner at pthread_start.c:45), arg=<unavailable>) at misc.c:2237:14 [opt]
    frame #5: 0x00000001034e1ef4 libgc.1.dylib`GC_pthread_start(arg=<unavailable>) at pthread_support.c:2506:14 [opt]
    frame #6: 0x000000018bc82034 libsystem_pthread.dylib`_pthread_start + 136
(lldb)

➜ [](base) neil@Beth-Studio bdwgc lldb --core /cores/core.75512
(lldb) target create --core "/cores/core.75512"
warning: libgc.1.dylib was compiled with optimization - stepping may behave oddly; variables may not be available.
Core file '/cores/core.75512' (arm64) was loaded.
(lldb) bt
* thread #22, stop reason = ESR_EC_DABORT_EL0 (fault address: 0x102f73838)
  * frame #0: 0x00000001033c8f68 libgc.1.dylib`GC_malloc_kind(bytes=1, kind=0) at thread_local_alloc.c:188:5 [opt]
    frame #1: 0x00000001033c5234 libgc.1.dylib`GC_malloc_atomic(lb=<unavailable>) at malloc.c:366:12 [opt] [artificial]
    frame #2: 0x0000000102e7b7fc gctest`run_one_test at gctest.c:1778:54 [opt]
    frame #3: 0x0000000102e7b1ec gctest`thr_run_one_test(arg=<unavailable>) at gctest.c:2588:5 [opt]
    frame #4: 0x00000001033db634 libgc.1.dylib`GC_pthread_start_inner(sb=<unavailable>, arg=<unavailable>) at pthread_start.c:55:12 [opt]
    frame #5: 0x00000001033d1f04 libgc.1.dylib`GC_pthread_start [inlined] GC_call_with_stack_base(fn=(libgc.1.dylib`GC_pthread_start_inner at pthread_start.c:45), arg=<unavailable>) at misc.c:2237:14 [opt]
    frame #6: 0x00000001033d1ef4 libgc.1.dylib`GC_pthread_start(arg=<unavailable>) at pthread_support.c:2506:14 [opt]
    frame #7: 0x000000018bc82034 libsystem_pthread.dylib`_pthread_start + 136
(lldb)
ivmai commented 10 months ago

I have several core dumps also: frame #0: 0x00000001027202fc gctest`typed_test at gctest.c:1447:17 [opt]

Is my understanding correct that "KERN_PROTECTION_FAILURE while world is stopped" happens during newP[0] = 0 (in typed_test)? Could you please list frames trace of other threads in core.19134 ?

coolbikerdad commented 10 months ago

I have several core dumps also: frame #0: 0x00000001027202fc gctest`typed_test at gctest.c:1447:17 [opt]

Is my understanding correct that "KERN_PROTECTION_FAILURE while world is stopped" happens during newP[0] = 0 (in typed_test)? Could you please list frames trace of other threads in core.19134 ?

Here's a later core, after I put in a check for GC_mprotect_state. Analysis of gctest, PID 4682

Log output:

GC test started, pid = 4682
Switched to incremental mode
Emulating dirty bits with mprotect/signals
KERN_PROTECTION_FAILURE while world is stopped, pid = 4682, GC_mprotect_state = 2
0   libgc.1.dylib                       0x0000000100f60494 catch_exception_raise + 460
1   libsystem_kernel.dylib              0x000000018bc5fc24 _Xexception_raise + 192
2   libsystem_kernel.dylib              0x000000018bc5fcc4 exc_server + 96
3   libgc.1.dylib                       0x0000000100f6adb4 GC_mprotect_thread + 444
4   libsystem_pthread.dylib             0x000000018bc82034 _pthread_start + 136
5   libsystem_pthread.dylib             0x000000018bc7ce3c thread_start + 8

Core dump analysis:

➜ [](base) neil@Beth-Studio bdwgc lldb --core /cores/core.4682
(lldb) target create --core "/cores/core.4682"
warning: gctest was compiled with optimization - stepping may behave oddly; variables may not be available.
Core file '/cores/core.4682' (arm64) was loaded.
(lldb) bt
* thread #1, stop reason = ESR_EC_DABORT_EL0 (fault address: 0x100dd01a0)
  * frame #0: 0x0000000100a0c2fc gctest`typed_test at gctest.c:1447:17 [opt]
    frame #1: 0x0000000100a0bad0 gctest`run_one_test at gctest.c:1883:7 [opt]
    frame #2: 0x0000000100a0a994 gctest`main at gctest.c:2707:9 [opt]
    frame #3: 0x000000018b9010e0 dyld`start + 2360
(lldb)
(lldb) bt all
warning: libgc.1.dylib was compiled with optimization - stepping may behave oddly; variables may not be available.
* thread #1, stop reason = ESR_EC_DABORT_EL0 (fault address: 0x100dd01a0)
  * frame #0: 0x0000000100a0c2fc gctest`typed_test at gctest.c:1447:17 [opt]
    frame #1: 0x0000000100a0bad0 gctest`run_one_test at gctest.c:1883:7 [opt]
    frame #2: 0x0000000100a0a994 gctest`main at gctest.c:2707:9 [opt]
    frame #3: 0x000000018b9010e0 dyld`start + 2360
  thread #2
    frame #0: 0x000000018bc41874 libsystem_kernel.dylib`mach_msg2_trap + 8
    frame #1: 0x000000018bc53cf0 libsystem_kernel.dylib`mach_msg2_internal + 80
    frame #2: 0x000000018bc4a4b0 libsystem_kernel.dylib`mach_msg_overwrite + 476
    frame #3: 0x000000018bc41bf8 libsystem_kernel.dylib`mach_msg + 24
    frame #4: 0x0000000100f6ad14 libgc.1.dylib`GC_mprotect_thread(arg=<unavailable>) at os_dep.c:4639:9 [opt]
    frame #5: 0x000000018bc82034 libsystem_pthread.dylib`_pthread_start + 136
  thread #3
    frame #0: 0x000000018bc4506c libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x000000018bc825fc libsystem_pthread.dylib`_pthread_cond_wait + 1228
    frame #2: 0x0000000100f6b3b4 libgc.1.dylib`GC_mark_thread [inlined] GC_wait_marker at pthread_support.c:2941:9 [opt]
    frame #3: 0x0000000100f6b3a8 libgc.1.dylib`GC_mark_thread [inlined] GC_help_marker(my_mark_no=59) at mark.c:1248:7 [opt]
    frame #4: 0x0000000100f6b3a8 libgc.1.dylib`GC_mark_thread(id=<unavailable>) at pthread_support.c:485:7 [opt]
    frame #5: 0x000000018bc82034 libsystem_pthread.dylib`_pthread_start + 136
  thread #4
    frame #0: 0x000000018bc4506c libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x000000018bc825fc libsystem_pthread.dylib`_pthread_cond_wait + 1228
    frame #2: 0x0000000100f6b3b4 libgc.1.dylib`GC_mark_thread [inlined] GC_wait_marker at pthread_support.c:2941:9 [opt]
    frame #3: 0x0000000100f6b3a8 libgc.1.dylib`GC_mark_thread [inlined] GC_help_marker(my_mark_no=59) at mark.c:1248:7 [opt]
    frame #4: 0x0000000100f6b3a8 libgc.1.dylib`GC_mark_thread(id=<unavailable>) at pthread_support.c:485:7 [opt]
    frame #5: 0x000000018bc82034 libsystem_pthread.dylib`_pthread_start + 136
  thread #5
    frame #0: 0x000000018bc4506c libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x000000018bc825fc libsystem_pthread.dylib`_pthread_cond_wait + 1228
    frame #2: 0x0000000100f6b3b4 libgc.1.dylib`GC_mark_thread [inlined] GC_wait_marker at pthread_support.c:2941:9 [opt]
    frame #3: 0x0000000100f6b3a8 libgc.1.dylib`GC_mark_thread [inlined] GC_help_marker(my_mark_no=59) at mark.c:1248:7 [opt]
    frame #4: 0x0000000100f6b3a8 libgc.1.dylib`GC_mark_thread(id=<unavailable>) at pthread_support.c:485:7 [opt]
    frame #5: 0x000000018bc82034 libsystem_pthread.dylib`_pthread_start + 136
  thread #6
    frame #0: 0x000000018bc4506c libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x000000018bc825fc libsystem_pthread.dylib`_pthread_cond_wait + 1228
    frame #2: 0x0000000100f6b3b4 libgc.1.dylib`GC_mark_thread [inlined] GC_wait_marker at pthread_support.c:2941:9 [opt]
    frame #3: 0x0000000100f6b3a8 libgc.1.dylib`GC_mark_thread [inlined] GC_help_marker(my_mark_no=59) at mark.c:1248:7 [opt]
    frame #4: 0x0000000100f6b3a8 libgc.1.dylib`GC_mark_thread(id=<unavailable>) at pthread_support.c:485:7 [opt]
    frame #5: 0x000000018bc82034 libsystem_pthread.dylib`_pthread_start + 136
  thread #7
    frame #0: 0x000000018bc4506c libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x000000018bc825fc libsystem_pthread.dylib`_pthread_cond_wait + 1228
    frame #2: 0x0000000100f6b3b4 libgc.1.dylib`GC_mark_thread [inlined] GC_wait_marker at pthread_support.c:2941:9 [opt]
    frame #3: 0x0000000100f6b3a8 libgc.1.dylib`GC_mark_thread [inlined] GC_help_marker(my_mark_no=59) at mark.c:1248:7 [opt]
    frame #4: 0x0000000100f6b3a8 libgc.1.dylib`GC_mark_thread(id=<unavailable>) at pthread_support.c:485:7 [opt]
    frame #5: 0x000000018bc82034 libsystem_pthread.dylib`_pthread_start + 136
  thread #8
    frame #0: 0x000000018bc4506c libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x000000018bc825fc libsystem_pthread.dylib`_pthread_cond_wait + 1228
    frame #2: 0x0000000100f6b3b4 libgc.1.dylib`GC_mark_thread [inlined] GC_wait_marker at pthread_support.c:2941:9 [opt]
    frame #3: 0x0000000100f6b3a8 libgc.1.dylib`GC_mark_thread [inlined] GC_help_marker(my_mark_no=59) at mark.c:1248:7 [opt]
    frame #4: 0x0000000100f6b3a8 libgc.1.dylib`GC_mark_thread(id=<unavailable>) at pthread_support.c:485:7 [opt]
    frame #5: 0x000000018bc82034 libsystem_pthread.dylib`_pthread_start + 136
  thread #9
    frame #0: 0x000000018bc4506c libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x000000018bc825fc libsystem_pthread.dylib`_pthread_cond_wait + 1228
    frame #2: 0x0000000100f6b3b4 libgc.1.dylib`GC_mark_thread [inlined] GC_wait_marker at pthread_support.c:2941:9 [opt]
    frame #3: 0x0000000100f6b3a8 libgc.1.dylib`GC_mark_thread [inlined] GC_help_marker(my_mark_no=59) at mark.c:1248:7 [opt]
    frame #4: 0x0000000100f6b3a8 libgc.1.dylib`GC_mark_thread(id=<unavailable>) at pthread_support.c:485:7 [opt]
    frame #5: 0x000000018bc82034 libsystem_pthread.dylib`_pthread_start + 136
  thread #10
    frame #0: 0x000000018bc4506c libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x000000018bc825fc libsystem_pthread.dylib`_pthread_cond_wait + 1228
    frame #2: 0x0000000100f6b3b4 libgc.1.dylib`GC_mark_thread [inlined] GC_wait_marker at pthread_support.c:2941:9 [opt]
    frame #3: 0x0000000100f6b3a8 libgc.1.dylib`GC_mark_thread [inlined] GC_help_marker(my_mark_no=59) at mark.c:1248:7 [opt]
    frame #4: 0x0000000100f6b3a8 libgc.1.dylib`GC_mark_thread(id=<unavailable>) at pthread_support.c:485:7 [opt]
    frame #5: 0x000000018bc82034 libsystem_pthread.dylib`_pthread_start + 136
  thread #11
    frame #0: 0x000000018bc4506c libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x000000018bc825fc libsystem_pthread.dylib`_pthread_cond_wait + 1228
    frame #2: 0x0000000100f6b3b4 libgc.1.dylib`GC_mark_thread [inlined] GC_wait_marker at pthread_support.c:2941:9 [opt]
    frame #3: 0x0000000100f6b3a8 libgc.1.dylib`GC_mark_thread [inlined] GC_help_marker(my_mark_no=59) at mark.c:1248:7 [opt]
    frame #4: 0x0000000100f6b3a8 libgc.1.dylib`GC_mark_thread(id=<unavailable>) at pthread_support.c:485:7 [opt]
    frame #5: 0x000000018bc82034 libsystem_pthread.dylib`_pthread_start + 136
  thread #12
    frame #0: 0x000000018bc4506c libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x000000018bc825fc libsystem_pthread.dylib`_pthread_cond_wait + 1228
    frame #2: 0x0000000100f6b3b4 libgc.1.dylib`GC_mark_thread [inlined] GC_wait_marker at pthread_support.c:2941:9 [opt]
    frame #3: 0x0000000100f6b3a8 libgc.1.dylib`GC_mark_thread [inlined] GC_help_marker(my_mark_no=59) at mark.c:1248:7 [opt]
    frame #4: 0x0000000100f6b3a8 libgc.1.dylib`GC_mark_thread(id=<unavailable>) at pthread_support.c:485:7 [opt]
    frame #5: 0x000000018bc82034 libsystem_pthread.dylib`_pthread_start + 136
  thread #13
    frame #0: 0x000000018bc4506c libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x000000018bc825fc libsystem_pthread.dylib`_pthread_cond_wait + 1228
    frame #2: 0x0000000100f6b3b4 libgc.1.dylib`GC_mark_thread [inlined] GC_wait_marker at pthread_support.c:2941:9 [opt]
    frame #3: 0x0000000100f6b3a8 libgc.1.dylib`GC_mark_thread [inlined] GC_help_marker(my_mark_no=59) at mark.c:1248:7 [opt]
    frame #4: 0x0000000100f6b3a8 libgc.1.dylib`GC_mark_thread(id=<unavailable>) at pthread_support.c:485:7 [opt]
    frame #5: 0x000000018bc82034 libsystem_pthread.dylib`_pthread_start + 136
  thread #14
    frame #0: 0x000000018bc4506c libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x000000018bc825fc libsystem_pthread.dylib`_pthread_cond_wait + 1228
    frame #2: 0x0000000100f6b3b4 libgc.1.dylib`GC_mark_thread [inlined] GC_wait_marker at pthread_support.c:2941:9 [opt]
    frame #3: 0x0000000100f6b3a8 libgc.1.dylib`GC_mark_thread [inlined] GC_help_marker(my_mark_no=59) at mark.c:1248:7 [opt]
    frame #4: 0x0000000100f6b3a8 libgc.1.dylib`GC_mark_thread(id=<unavailable>) at pthread_support.c:485:7 [opt]
    frame #5: 0x000000018bc82034 libsystem_pthread.dylib`_pthread_start + 136
  thread #15
    frame #0: 0x000000018bc4506c libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x000000018bc825fc libsystem_pthread.dylib`_pthread_cond_wait + 1228
    frame #2: 0x0000000100f6b3b4 libgc.1.dylib`GC_mark_thread [inlined] GC_wait_marker at pthread_support.c:2941:9 [opt]
    frame #3: 0x0000000100f6b3a8 libgc.1.dylib`GC_mark_thread [inlined] GC_help_marker(my_mark_no=59) at mark.c:1248:7 [opt]
    frame #4: 0x0000000100f6b3a8 libgc.1.dylib`GC_mark_thread(id=<unavailable>) at pthread_support.c:485:7 [opt]
    frame #5: 0x000000018bc82034 libsystem_pthread.dylib`_pthread_start + 136
  thread #16
    frame #0: 0x000000018bc4506c libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x000000018bc825fc libsystem_pthread.dylib`_pthread_cond_wait + 1228
    frame #2: 0x0000000100f6b3b4 libgc.1.dylib`GC_mark_thread [inlined] GC_wait_marker at pthread_support.c:2941:9 [opt]
    frame #3: 0x0000000100f6b3a8 libgc.1.dylib`GC_mark_thread [inlined] GC_help_marker(my_mark_no=59) at mark.c:1248:7 [opt]
    frame #4: 0x0000000100f6b3a8 libgc.1.dylib`GC_mark_thread(id=<unavailable>) at pthread_support.c:485:7 [opt]
    frame #5: 0x000000018bc82034 libsystem_pthread.dylib`_pthread_start + 136
  thread #17
    frame #0: 0x000000018bc4506c libsystem_kernel.dylib`__psynch_cvwait + 8
    frame #1: 0x000000018bc825fc libsystem_pthread.dylib`_pthread_cond_wait + 1228
    frame #2: 0x0000000100f6b3b4 libgc.1.dylib`GC_mark_thread [inlined] GC_wait_marker at pthread_support.c:2941:9 [opt]
    frame #3: 0x0000000100f6b3a8 libgc.1.dylib`GC_mark_thread [inlined] GC_help_marker(my_mark_no=59) at mark.c:1248:7 [opt]
    frame #4: 0x0000000100f6b3a8 libgc.1.dylib`GC_mark_thread(id=<unavailable>) at pthread_support.c:485:7 [opt]
    frame #5: 0x000000018bc82034 libsystem_pthread.dylib`_pthread_start + 136
  thread #18, stop reason = ESR_EC_DABORT_EL0 (fault address: 0x1014ef000)
    frame #0: 0x0000000100a0c2fc gctest`typed_test at gctest.c:1447:17 [opt]
    frame #1: 0x0000000100a0bad0 gctest`run_one_test at gctest.c:1883:7 [opt]
    frame #2: 0x0000000100a0b1ec gctest`thr_run_one_test(arg=<unavailable>) at gctest.c:2588:5 [opt]
    frame #3: 0x0000000100f6b624 libgc.1.dylib`GC_pthread_start_inner(sb=<unavailable>, arg=<unavailable>) at pthread_start.c:55:12 [opt]
    frame #4: 0x0000000100f61ef4 libgc.1.dylib`GC_pthread_start [inlined] GC_call_with_stack_base(fn=(libgc.1.dylib`GC_pthread_start_inner at pthread_start.c:45), arg=<unavailable>) at misc.c:2237:14 [opt]
    frame #5: 0x0000000100f61ee4 libgc.1.dylib`GC_pthread_start(arg=<unavailable>) at pthread_support.c:2506:14 [opt]
    frame #6: 0x000000018bc82034 libsystem_pthread.dylib`_pthread_start + 136
  thread #19, stop reason = ESR_EC_DABORT_EL0 (fault address: 0x1016b5000)
    frame #0: 0x0000000100a0c2fc gctest`typed_test at gctest.c:1447:17 [opt]
    frame #1: 0x0000000100a0bad0 gctest`run_one_test at gctest.c:1883:7 [opt]
    frame #2: 0x0000000100a0b1ec gctest`thr_run_one_test(arg=<unavailable>) at gctest.c:2588:5 [opt]
    frame #3: 0x0000000100f6b624 libgc.1.dylib`GC_pthread_start_inner(sb=<unavailable>, arg=<unavailable>) at pthread_start.c:55:12 [opt]
    frame #4: 0x0000000100f61ef4 libgc.1.dylib`GC_pthread_start [inlined] GC_call_with_stack_base(fn=(libgc.1.dylib`GC_pthread_start_inner at pthread_start.c:45), arg=<unavailable>) at misc.c:2237:14 [opt]
    frame #5: 0x0000000100f61ee4 libgc.1.dylib`GC_pthread_start(arg=<unavailable>) at pthread_support.c:2506:14 [opt]
    frame #6: 0x000000018bc82034 libsystem_pthread.dylib`_pthread_start + 136
  thread #20
    frame #0: 0x0000000100f62400 libgc.1.dylib`GC_mark_some [inlined] GC_push_next_marked_uncollectable(h=0x00000001016d9000) at headers.c:0:9 [opt]
    frame #1: 0x0000000100f623c0 libgc.1.dylib`GC_mark_some(cold_gc_frame=" \xaa\xfco\U00000001") at mark.c:419:31 [opt]
    frame #2: 0x0000000100f631cc libgc.1.dylib`GC_stopped_mark(stop_func=(libgc.1.dylib`GC_never_stop_func at alloc.c:175:3)) at alloc.c:891:15 [opt]
    frame #3: 0x0000000100f5e4ac libgc.1.dylib`GC_try_to_collect_inner(stop_func=(libgc.1.dylib`GC_never_stop_func at alloc.c:175:3)) at alloc.c:606:10 [opt]
    frame #4: 0x0000000100f543e0 libgc.1.dylib`GC_try_to_collect_general(stop_func=0x0000000000000000, force_unmap=<unavailable>) at alloc.c:1315:14 [opt]
    frame #5: 0x0000000100f54460 libgc.1.dylib`GC_gcollect at alloc.c:1340:11 [opt]
    frame #6: 0x0000000100a0c314 gctest`typed_test at gctest.c:1451:5 [opt]
    frame #7: 0x0000000100a0bad0 gctest`run_one_test at gctest.c:1883:7 [opt]
    frame #8: 0x0000000100a0b1ec gctest`thr_run_one_test(arg=<unavailable>) at gctest.c:2588:5 [opt]
    frame #9: 0x0000000100f6b624 libgc.1.dylib`GC_pthread_start_inner(sb=<unavailable>, arg=<unavailable>) at pthread_start.c:55:12 [opt]
    frame #10: 0x0000000100f61ef4 libgc.1.dylib`GC_pthread_start [inlined] GC_call_with_stack_base(fn=(libgc.1.dylib`GC_pthread_start_inner at pthread_start.c:45), arg=<unavailable>) at misc.c:2237:14 [opt]
    frame #11: 0x0000000100f61ee4 libgc.1.dylib`GC_pthread_start(arg=<unavailable>) at pthread_support.c:2506:14 [opt]
    frame #12: 0x000000018bc82034 libsystem_pthread.dylib`_pthread_start + 136
  thread #21
    frame #0: 0x000000018bc44524 libsystem_kernel.dylib`__psynch_mutexwait + 8
    frame #1: 0x000000018bc7f168 libsystem_pthread.dylib`_pthread_mutex_firstfit_lock_wait + 84
    frame #2: 0x000000018bc7caf8 libsystem_pthread.dylib`_pthread_mutex_firstfit_lock_slow + 248
    frame #3: 0x0000000100f59270 libgc.1.dylib`GC_malloc_kind_aligned_global(lb=1436, k=1, align_m1=0) at malloc.c:325:9 [opt]
    frame #4: 0x0000000100f58fe0 libgc.1.dylib`GC_malloc_kind(bytes=<unavailable>, kind=<unavailable>) at thread_local_alloc.c:0 [opt] [artificial]
    frame #5: 0x0000000100f54d1c libgc.1.dylib`GC_malloc(lb=<unavailable>) at malloc.c:372:12 [opt] [artificial]
    frame #6: 0x0000000100a0cac4 gctest`reverse [inlined] cons(x=0x0000000100c259c0, y=0x0000000000000000) at gctest.c:337:25 [opt]
    frame #7: 0x0000000100a0ca9c gctest`reverse [inlined] reverse1(x=<unavailable>, y=0x0000000000000000) at gctest.c:464:33 [opt]
    frame #8: 0x0000000100a0ca54 gctest`reverse(x=0x0000000101b88800) at gctest.c:473:12 [opt]
    frame #9: 0x0000000100a0cf38 gctest`reverse_test_inner(data=<unavailable>) at gctest.c:942:21 [opt]
    frame #10: 0x0000000100f61264 libgc.1.dylib`GC_call_with_gc_active(fn=(gctest`reverse_test_inner at gctest.c:850), client_data=0x0000000000000001) at pthread_support.c:2103:19 [opt]
    frame #11: 0x0000000100f5f7f8 libgc.1.dylib`GC_do_blocking_inner(data="\xb0̠", context=<unavailable>) at pthread_support.c:1907:24 [opt]
    frame #12: 0x0000000100f5f718 libgc.1.dylib`GC_do_blocking [inlined] GC_with_callee_saves_pushed(fn=(libgc.1.dylib`GC_do_blocking_inner at pthread_support.c:1896), arg="\xb0̠") at mach_dep.c:337:3 [opt]
    frame #13: 0x0000000100f5f704 libgc.1.dylib`GC_do_blocking(fn=<unavailable>, client_data=0x0000000000000000) at misc.c:2370:5 [opt]
    frame #14: 0x0000000100a0ba8c gctest`run_one_test [inlined] reverse_test at gctest.c:996:11 [opt]
    frame #15: 0x0000000100a0ba7c gctest`run_one_test at gctest.c:1873:9 [opt]
    frame #16: 0x0000000100a0b1ec gctest`thr_run_one_test(arg=<unavailable>) at gctest.c:2588:5 [opt]
    frame #17: 0x0000000100f6b624 libgc.1.dylib`GC_pthread_start_inner(sb=<unavailable>, arg=<unavailable>) at pthread_start.c:55:12 [opt]
    frame #18: 0x0000000100f61ef4 libgc.1.dylib`GC_pthread_start [inlined] GC_call_with_stack_base(fn=(libgc.1.dylib`GC_pthread_start_inner at pthread_start.c:45), arg=<unavailable>) at misc.c:2237:14 [opt]
    frame #19: 0x0000000100f61ee4 libgc.1.dylib`GC_pthread_start(arg=<unavailable>) at pthread_support.c:2506:14 [opt]
    frame #20: 0x000000018bc82034 libsystem_pthread.dylib`_pthread_start + 136
  thread #22
    frame #0: 0x000000018bc48edc libsystem_kernel.dylib`__wait4 + 8
    frame #1: 0x0000000100a0ba44 gctest`run_one_test at gctest.c:1821:15 [opt]
    frame #2: 0x0000000100a0b1ec gctest`thr_run_one_test(arg=<unavailable>) at gctest.c:2588:5 [opt]
    frame #3: 0x0000000100f6b624 libgc.1.dylib`GC_pthread_start_inner(sb=<unavailable>, arg=<unavailable>) at pthread_start.c:55:12 [opt]
    frame #4: 0x0000000100f61ef4 libgc.1.dylib`GC_pthread_start [inlined] GC_call_with_stack_base(fn=(libgc.1.dylib`GC_pthread_start_inner at pthread_start.c:45), arg=<unavailable>) at misc.c:2237:14 [opt]
    frame #5: 0x0000000100f61ee4 libgc.1.dylib`GC_pthread_start(arg=<unavailable>) at pthread_support.c:2506:14 [opt]
    frame #6: 0x000000018bc82034 libsystem_pthread.dylib`_pthread_start + 136
(lldb)