Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests segfault in Mono

danmoseley commented 1 year ago

https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-81202-merge-135c923415104d6c85/Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests/1/console.6c3784b3.log?helixlogtype=result

net8.0-linux-Debug-arm64-Mono_release-(Ubuntu.1804.Arm64.Open)Ubuntu.1804.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8

Possible stack trace for the crash

Thread 9 (Thread 0xff7909f741c0 (LWP 34)):
#0  0x0000ff791595bd5c in __waitpid (pid=<optimized out>, stat_loc=0xff7909f6e9f0, options=<optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:30
#1  0x0000ff79153519e8 in dump_native_stacktrace (signal=<optimized out>, mctx=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:843
#2  mono_dump_native_crash_info (signal=<optimized out>, mctx=0xff7909f6f450, info=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:870
#3  0x0000ff79153105c0 in mono_handle_native_crash (signal=0xff791510e1d2 "SIGSEGV", mctx=0xff7909f6f450, info=0xff7909f6f7b0) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:3005
#4  0x0000ff7915279b90 in mono_sigsegv_signal_handler_debug (_dummy=11, _info=0xff7909f6f7b0, context=0xff7909f6f830, debug_fault_addr=0x0) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:3749
#5  <signal handler called>
#6  0x0000000000000000 in ?? ()
#7  0x0000ff7908de1d18 in ?? ()
#8  0x0000ff7914ad9848 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

(original issue description note: this is not the problematic stack trace)

Not sure how it indicates which thread failed, but I'm guessing it's this one?

Thread 10 (Thread 0xff7909d431c0 (LWP 35)):
#0  monoeg_g_hash_table_insert_replace (hash=0xab9b66ab1fb0, key=0xff78e945dca0, value=0xff78e945dca0, replace=0) at /__w/1/s/src/mono/mono/eglib/ghashtable.c:246
#1  0x0000ff79151467b4 in mono_class_inflate_generic_method_full_checked (method=<optimized out>, klass_hint=<optimized out>, context=0xff78e8e928f8, error=<optimized out>) at /__w/1/s/src/mono/mono/metadata/class.c:1312
#2  0x0000ff79151588f0 in mono_class_setup_vtable_ginst (klass=0xff78e8e92920, in_setup=<optimized out>) at /__w/1/s/src/mono/mono/metadata/class-setup-vtable.c:1363
#3  mono_class_setup_vtable_general (klass=0xff78e8e92920, overrides=<optimized out>, onum=<optimized out>, in_setup=<optimized out>) at /__w/1/s/src/mono/mono/metadata/class-setup-vtable.c:1723
#4  0x0000ff7915158484 in mono_class_setup_vtable_full (klass=0xff78e8e92920, in_setup=0xff78e945a130) at /__w/1/s/src/mono/mono/metadata/class-setup-vtable.c:960
#5  0x0000ff791529ca68 in mono_method_to_ir (cfg=<optimized out>, method=<optimized out>, start_bblock=0xff78e9443078, end_bblock=0xff78e94431b8, return_var=0x0, inline_args=<optimized out>, inline_offset=<optimized out>, is_virtual_call=<optimized out>) at /__w/1/s/src/mono/mono/mini/method-to-ir.c:7924
#6  0x0000ff7915271728 in mini_method_compile (method=<optimized out>, opts=374417919, flags=JIT_FLAG_RUN_CCTORS, parts=0, aot_method_index=-1) at /__w/1/s/src/mono/mono/mini/mini.c:3495
#7  0x0000ff791527412c in mono_jit_compile_method_inner (method=0xff78e8e89940, opt=-381297504, error=0xff7909d3c970) at /__w/1/s/src/mono/mono/mini/mini.c:4129
#8  0x0000ff7915278cb0 in mono_jit_compile_method_with_opt (method=0xff78e8e89940, opt=374417919, jit_only=0, error=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2709
#9  jit_compile_method_with_opt_cb (arg=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2764
#10 jit_compile_method_with_opt (params=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2780
#11 0x0000ff79152780dc in mono_jit_compile_method (method=<optimized out>, error=0xff7909d3c970) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2799
#12 0x0000ff7915312a2c in common_call_trampoline (regs=0xff7909d3ca00, code=0xff79082dc1ec "", m=0xff78e8e89940, vt=0x0, vtable_slot=<optimized out>, error=0xff7909d3c970) at /__w/1/s/src/mono/mono/mini/mini-trampolines.c:618
#13 0x0000ff791531254c in mono_magic_trampoline (regs=0xff7909d3ca00, code=0xff79082dc1ec "", arg=0xff78e93e72a0, tramp=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-trampolines.c:759
#14 0x0000ff7914ee8304 in ?? ()
#15 0x0000ff7914bdf620 in ?? ()
#16 0x0000ff7914b219c0 in ?? ()
#17 0x0064002e00740073 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

there is a core dump https://dev.azure.com/dnceng-public/public/_build/results?buildId=148909&view=ms.vss-test-web.build-test-results-tab&runId=3170018&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab&resultId=200309

stephentoub commented 1 year ago

There's another issue representing a crash in source generators on mono that's been plaguing us in CI: https://github.com/dotnet/runtime/issues/81123 I don't know if it's the same thing.

danmoseley commented 1 year ago

@janvorli would you or someone on the VM team be interested in the dump here before it vanishes? at least #81123 seems fairly common and this might be related.

janvorli commented 1 year ago

@danmoseley this is mono, so it would be better to point some mono folks to this. The coreclr runtime team doesn't have experience debugging mono.

danmoseley commented 1 year ago

@janvorli - doh - of course, brainstorm. Thanks.

@SamMonoRT there's a dump here if you want it, not sure how long before they time out.

SamMonoRT commented 1 year ago

@lambdageek - can you take an initial look

lambdageek commented 1 year ago

Thread 10 looks ok, actually. it's just doing some work. Thread 9 looks like it crashed:

Thread 9 (Thread 0xff7909f741c0 (LWP 34)):
#0  0x0000ff791595bd5c in __waitpid (pid=<optimized out>, stat_loc=0xff7909f6e9f0, options=<optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:30
#1  0x0000ff79153519e8 in dump_native_stacktrace (signal=<optimized out>, mctx=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:843
#2  mono_dump_native_crash_info (signal=<optimized out>, mctx=0xff7909f6f450, info=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:870
#3  0x0000ff79153105c0 in mono_handle_native_crash (signal=0xff791510e1d2 "SIGSEGV", mctx=0xff7909f6f450, info=0xff7909f6f7b0) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:3005
#4  0x0000ff7915279b90 in mono_sigsegv_signal_handler_debug (_dummy=11, _info=0xff7909f6f7b0, context=0xff7909f6f830, debug_fault_addr=0x0) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:3749
#5  <signal handler called>
#6  0x0000000000000000 in ?? ()
#7  0x0000ff7908de1d18 in ?? ()
#8  0x0000ff7914ad9848 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

and some output from mono_dump_native_crash_info

=====================================================
instruction pointer is NULL, skip dumping
=====================================================

Frames 7 and 8 look like JITed code. we'll need to try repro locally.

~~I'll try to grab the coredump once I'm on my work computer~~ Core dump expired.

steveisok commented 1 year ago

@lambdageek the links to builds in https://github.com/dotnet/runtime/issues/81123 might contain coredumps.

lewing commented 1 year ago

@lambdageek as another data point the crashes only seem to happen on arm64

lambdageek commented 1 year ago

Got it to crash in a VM after a couple of hours of running the test in a loop. Not the same crash as this issue, unfortunately. An assertion failure during class setup due to - I'm guessing - a data race:

    g_assert (klass == klass->supertypes [klass->idepth - 1]);

I see two threads in mono_class_setup_vtable_full for the same class. It does seem like it's possible for klass->supertypes and klass->idepth not to be set atomically.

I can't convince myself that it would cause the same kinds of failures we see in the CI jobs.

lambdageek commented 1 year ago

Probably same as https://github.com/dotnet/runtime/issues/81123 - which is a very similar stack trace in the Roslyn3.11 test suite. Going to try running that other testsuite in a loop, too

carlossanlop commented 1 year ago

Seen again in an unrelated PR in main: https://github.com/dotnet/runtime/pull/83356

Queue: Libraries Test Run release mono linux arm64 Debug
Job: https://dev.azure.com/dnceng-public/public/_build/results?buildId=206231&view=logs&j=0aa53445-e474-5f87-17f4-cf1800f4293c&t=f8159c71-541f-59ca-1f6b-cfff92851d99
Log: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-83356-merge-426b96be0b3a42819c/Microsoft.Extensions.Logging.Generators.Roslyn3.11.Tests/1/console.1a24fb92.log?helixlogtype=result

Callstacks (expand)

``` ================================================================= External Debugger Dump: ================================================================= [New LWP 23] [New LWP 24] [New LWP 25] [New LWP 26] [New LWP 29] [New LWP 30] [New LWP 31] [New LWP 32] [New LWP 33] [New LWP 34] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1". futex_wait_cancelable (private=0, expected=0, futex_word=0xab1a6c0e25ac) at ../sysdeps/nptl/futex-internal.h:186 186 ../sysdeps/nptl/futex-internal.h: No such file or directory. Id Target Id Frame * 1 Thread 0xff0694dec010 (LWP 22) "dotnet" futex_wait_cancelable (private=0, expected=0, futex_word=0xab1a6c0e25ac) at ../sysdeps/nptl/futex-internal.h:186 2 Thread 0xff0693bff1b0 (LWP 23) "SGen worker" futex_wait_cancelable (private=0, expected=0, futex_word=0xff069480fe80 ) at ../sysdeps/nptl/futex-internal.h:186 3 Thread 0xff0691e611b0 (LWP 24) ".NET EventPipe" 0x0000ff0694a30ef4 in __GI___poll (fds=0xff06840034f0, nfds=1, timeout=) at ../sysdeps/unix/sysv/linux/poll.c:41 4 Thread 0xff0691c601b0 (LWP 25) "Finalizer" futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, futex_word=0xff0694801950 ) at ../sysdeps/nptl/futex-internal.h:323 5 Thread 0xff06898ff1b0 (LWP 26) ".NET SigHandler" __libc_read (nbytes=1, buf=0xff06898fe977, fd=) at ../sysdeps/unix/sysv/linux/read.c:26 6 Thread 0xff06897d81b0 (LWP 29) ".NET Long Runni" futex_wait_cancelable (private=0, expected=0, futex_word=0xff06780057e8) at ../sysdeps/nptl/futex-internal.h:186 7 Thread 0xff06895c71b0 (LWP 30) ".NET TP Worker" futex_abstimed_wait_cancelable (private=0, abstime=0xff06895c60e8, clockid=, expected=0, futex_word=0xff067c005348) at ../sysdeps/nptl/futex-internal.h:323 8 Thread 0xff06890cb1b0 (LWP 31) ".NET TP Gate" futex_abstimed_wait_cancelable (private=0, abstime=0xff06890ca2a8, clockid=, expected=0, futex_word=0xff0670005488) at ../sysdeps/nptl/futex-internal.h:323 9 Thread 0xff068906a1b0 (LWP 32) ".NET TP Worker" futex_abstimed_wait_cancelable (private=0, abstime=0xff0689069320, clockid=, expected=0, futex_word=0xff0689069398) at ../sysdeps/nptl/futex-internal.h:323 10 Thread 0xff0688e491b0 (LWP 33) ".NET Long Runni" mono_metadata_generic_inst_hash (data=0xff0688e419e0) at /__w/1/s/src/mono/mono/metadata/metadata.c:1878 11 Thread 0xff0688c481b0 (LWP 34) ".NET Long Runni" 0x0000ff0694a0c77c in __GI___wait4 (pid=, stat_loc=0xff0688c41350, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:27 Thread 11 (Thread 0xff0688c481b0 (LWP 34) ".NET Long Runni"): #0 0x0000ff0694a0c77c in __GI___wait4 (pid=, stat_loc=0xff0688c41350, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:27 #1 0x0000ff069460eb34 in dump_native_stacktrace (signal=, mctx=) at /__w/1/s/src/mono/mono/mini/mini-posix.c:843 #2 mono_dump_native_crash_info (signal=, mctx=0xff0688c41db0, info=) at /__w/1/s/src/mono/mono/mini/mini-posix.c:870 #3 0x0000ff06945c5f54 in mono_handle_native_crash (signal=0xff06944df2e5 "SIGSEGV", mctx=0xff0688c41db0, info=0xff0688c42110) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:2979 #4 0x0000ff069452da60 in mono_sigsegv_signal_handler_debug (_dummy=11, _info=0xff0688c42110, context=0xff0688c42190, debug_fault_addr=0x0) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:3758 #5 #6 0x0000000000000000 in ?? () #7 0x0000ff068804d478 in ?? () #8 0x0000ff0693dc6ae0 in ?? () Backtrace stopped: previous frame inner to this frame (corrupt stack?) Thread 10 (Thread 0xff0688e491b0 (LWP 33) ".NET Long Runni"): #0 mono_metadata_generic_inst_hash (data=0xff0688e419e0) at /__w/1/s/src/mono/mono/metadata/metadata.c:1878 #1 0x0000ff06946a28dc in monoeg_g_hash_table_lookup_extended (hash=0xab1a6b0dd880, key=0xff0688e419e0, orig_key=, value=) at /__w/1/s/src/mono/mono/eglib/ghashtable.c:338 #2 monoeg_g_hash_table_lookup (hash=0xab1a6b0dd880, key=0xff0688e419e0) at /__w/1/s/src/mono/mono/eglib/ghashtable.c:321 #3 0x0000ff0694724d8c in mono_metadata_get_canonical_generic_inst (candidate=0xff0688e419e0) at /__w/1/s/src/mono/mono/metadata/metadata.c:3414 #4 0x0000ff0694724cc4 in mono_metadata_get_generic_inst (type_argc=, type_argv=0xff066895f730) at /__w/1/s/src/mono/mono/metadata/metadata.c:3381 #5 0x0000ff0694725758 in mono_metadata_inflate_generic_inst (ginst=0xab1a6b1bb588, context=0xab1a6b1a7198, error=0xff066892f968) at /__w/1/s/src/mono/mono/metadata/metadata.c:3571 #6 0x0000ff069470ea74 in method_from_methodspec (image=0xab1a6b090b80, context=0xab1a6b1a7198, idx=, error=0xff066892f968) at /__w/1/s/src/mono/mono/metadata/loader.c:980 #7 mono_get_method_from_token (image=0xab1a6b090b80, token=721422599, klass=, context=0xab1a6b1a7198, used_context=0xff0688e41a9c, error=0xff066892f968) at /__w/1/s/src/mono/mono/metadata/loader.c:1044 #8 mono_get_method_checked (image=0xab1a6b090b80, token=721422599, klass=, context=0xab1a6b1a7198, error=0xff066892f968) at /__w/1/s/src/mono/mono/metadata/loader.c:1199 #9 0x0000ff069454e780 in mini_get_method_allow_open (m=0xab1a6b2eac80, token=, klass=0x0, context=0xab1a6b1a7198, error=0xff066892f968) at /__w/1/s/src/mono/mono/mini/method-to-ir.c:5093 #10 mono_method_to_ir (cfg=, method=, start_bblock=0xff0668979620, end_bblock=0xff0668979760, return_var=, inline_args=, inline_offset=, is_virtual_call=) at /__w/1/s/src/mono/mono/mini/method-to-ir.c:7684 #11 0x0000ff0694534b5c in inline_method (cfg=0xff066892f460, cmethod=, fsig=, sp=0xff0668978570, ip=0xff06920fc8fb "(\f\t", real_offset=, inline_always=, is_empty=) at /__w/1/s/src/mono/mono/mini/method-to-ir.c:4834 #12 0x0000ff069454e730 in mono_method_to_ir (cfg=, method=, start_bblock=0xff0668977750, end_bblock=0xff0668977890, return_var=, inline_args=, inline_offset=, is_virtual_call=) at /__w/1/s/src/mono/mono/mini/method-to-ir.c:7903 #13 0x0000ff0694534b5c in inline_method (cfg=0xff066892f460, cmethod=, fsig=, sp=0xff0668963e48, ip=0xff06920f65fb "(\004\021", real_offset=, inline_always=, is_empty=) at /__w/1/s/src/mono/mono/mini/method-to-ir.c:4834 #14 0x0000ff069454e730 in mono_method_to_ir (cfg=, method=, start_bblock=0xff0668963380, end_bblock=0xff06689634c0, return_var=, inline_args=, inline_offset=, is_virtual_call=) at /__w/1/s/src/mono/mono/mini/method-to-ir.c:7903 #15 0x0000ff0694534b5c in inline_method (cfg=0xff066892f460, cmethod=, fsig=, sp=0xff06689514a8, ip=0xff06920498e3 "(V\003", real_offset=, inline_always=, is_empty=) at /__w/1/s/src/mono/mono/mini/method-to-ir.c:4834 #16 0x0000ff069454e730 in mono_method_to_ir (cfg=, method=, start_bblock=0xff0668950158, end_bblock=0xff0668950298, return_var=, inline_args=, inline_offset=, is_virtual_call=) at /__w/1/s/src/mono/mono/mini/method-to-ir.c:7903 #17 0x0000ff0694534b5c in inline_method (cfg=0xff066892f460, cmethod=, fsig=, sp=0xff066894ff70, ip=0xff0692049c53 "(\032\021", real_offset=, inline_always=, is_empty=) at /__w/1/s/src/mono/mono/mini/method-to-ir.c:4834 #18 0x0000ff069454e730 in mono_method_to_ir (cfg=, method=, start_bblock=0xff066894f8a0, end_bblock=0xff066894f9e0, return_var=, inline_args=, inline_offset=, is_virtual_call=) at /__w/1/s/src/mono/mono/mini/method-to-ir.c:7903 #19 0x0000ff0694534b5c in inline_method (cfg=0xff066892f460, cmethod=, fsig=, sp=0xff0668929578, ip=0xff0689c115e0 "(\204\006", real_offset=, inline_always=, is_empty=) at /__w/1/s/src/mono/mono/mini/method-to-ir.c:4834 #20 0x0000ff069454e730 in mono_method_to_ir (cfg=, method=, start_bblock=0xff0668021e68, end_bblock=0xff0668021fa8, return_var=, inline_args=, inline_offset=, is_virtual_call=) at /__w/1/s/src/mono/mono/mini/method-to-ir.c:7903 #21 0x0000ff0694525458 in mini_method_compile (method=, opts=374417919, flags=JIT_FLAG_RUN_CCTORS, parts=0, aot_method_index=-1) at /__w/1/s/src/mono/mono/mini/mini.c:3492 #22 0x0000ff0694527e5c in mono_jit_compile_method_inner (method=0xff06682b8e28, opt=-1998317088, error=0xff0688e44110) at /__w/1/s/src/mono/mono/mini/mini.c:4126 #23 0x0000ff069452cb80 in mono_jit_compile_method_with_opt (method=0xff06682b8e28, opt=374417919, jit_only=0, error=) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2718 #24 jit_compile_method_with_opt_cb (arg=) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2773 #25 jit_compile_method_with_opt (params=) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2789 #26 0x0000ff069452bfac in mono_jit_compile_method (method=, error=0xff0688e44110) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:2808 #27 0x0000ff06945c83c0 in common_call_trampoline (regs=0xff0688e441a0, code=0xff0667decfd4 "\240", , m=0xff06682b8e28, vt=0x0, vtable_slot=, error=0xff0688e44110) at /__w/1/s/src/mono/mono/mini/mini-trampolines.c:618 #28 0x0000ff06945c7ee0 in mono_magic_trampoline (regs=0xff0688e441a0, code=0xff0667decfd4 "\240", , arg=0xff06682b8e28, tramp=) at /__w/1/s/src/mono/mono/mini/mini-trampolines.c:759 #29 0x0000ff069416b304 in ?? () #30 0x0000ff0688e44450 in ?? () #31 0x0000ff0693c8d6a8 in ?? () Backtrace stopped: not enough registers or memory available to unwind further Thread 9 (Thread 0xff068906a1b0 (LWP 32) ".NET TP Worker"): #0 futex_abstimed_wait_cancelable (private=0, abstime=0xff0689069320, clockid=, expected=0, futex_word=0xff0689069398) at ../sysdeps/nptl/futex-internal.h:323 #1 __pthread_cond_wait_common (abstime=0xff0689069320, clockid=, mutex=0xff0678036f40, cond=0xff0689069370) at pthread_cond_wait.c:520 #2 __pthread_cond_timedwait (cond=0xff0689069370, mutex=0xff0678036f40, abstime=0xff0689069320) at pthread_cond_wait.c:656 #3 0x0000ff06946b7a9c in mono_os_cond_timedwait (cond=0xff0689069370, mutex=0xff0678036f40, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/mono-os-mutex.c:75 #4 0x0000ff06946bcbbc in mono_coop_cond_timedwait (cond=0xff0689069370, mutex=, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/../../mono/utils/mono-coop-mutex.h:103 #5 mono_lifo_semaphore_timed_wait (semaphore=0xff0678036f40, timeout_ms=20000) at /__w/1/s/src/mono/mono/utils/lifo-semaphore.c:48 #6 0x0000ff0689079c38 in ?? () #7 0x0000ff0693d9ae80 in ?? () Backtrace stopped: previous frame inner to this frame (corrupt stack?) Thread 8 (Thread 0xff06890cb1b0 (LWP 31) ".NET TP Gate"): #0 futex_abstimed_wait_cancelable (private=0, abstime=0xff06890ca2a8, clockid=, expected=0, futex_word=0xff0670005488) at ../sysdeps/nptl/futex-internal.h:323 #1 __pthread_cond_wait_common (abstime=0xff06890ca2a8, clockid=, mutex=0xff0670005430, cond=0xff0670005460) at pthread_cond_wait.c:520 #2 __pthread_cond_timedwait (cond=0xff0670005460, mutex=0xff0670005430, abstime=0xff06890ca2a8) at pthread_cond_wait.c:656 #3 0x0000ff069404dc88 in SystemNative_LowLevelMonitor_TimedWait (monitor=0xff0670005430, timeoutMilliseconds=500) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:195 #4 0x0000ff0689080744 in ?? () #5 0x0000ff06895cef08 in ?? () #6 0x0000ff06890ca8e0 in ?? () Backtrace stopped: previous frame inner to this frame (corrupt stack?) Thread 7 (Thread 0xff06895c71b0 (LWP 30) ".NET TP Worker"): #0 futex_abstimed_wait_cancelable (private=0, abstime=0xff06895c60e8, clockid=, expected=0, futex_word=0xff067c005348) at ../sysdeps/nptl/futex-internal.h:323 #1 __pthread_cond_wait_common (abstime=0xff06895c60e8, clockid=, mutex=0xff067c0052f0, cond=0xff067c005320) at pthread_cond_wait.c:520 #2 __pthread_cond_timedwait (cond=0xff067c005320, mutex=0xff067c0052f0, abstime=0xff06895c60e8) at pthread_cond_wait.c:656 #3 0x0000ff069404dc88 in SystemNative_LowLevelMonitor_TimedWait (monitor=0xff067c0052f0, timeoutMilliseconds=12000) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:195 #4 0x0000ff0689080744 in ?? () #5 0x0000ff06895cef08 in ?? () #6 0x0000ff06895c68e0 in ?? () Backtrace stopped: previous frame inner to this frame (corrupt stack?) Thread 6 (Thread 0xff06897d81b0 (LWP 29) ".NET Long Runni"): #0 futex_wait_cancelable (private=0, expected=0, futex_word=0xff06780057e8) at ../sysdeps/nptl/futex-internal.h:186 #1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0xff0678005790, cond=0xff06780057c0) at pthread_cond_wait.c:508 #2 __pthread_cond_wait (cond=0xff06780057c0, mutex=0xff0678005790) at pthread_cond_wait.c:638 #3 0x0000ff069404db24 in SystemNative_LowLevelMonitor_Wait (monitor=0xff0678005790) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:155 #4 0x0000ff06895cf4cc in ?? () #5 0x0000ff0693d0b850 in ?? () #6 0x0000ff0693d0c8f0 in ?? () Backtrace stopped: previous frame inner to this frame (corrupt stack?) Thread 5 (Thread 0xff06898ff1b0 (LWP 26) ".NET SigHandler"): #0 __libc_read (nbytes=1, buf=0xff06898fe977, fd=) at ../sysdeps/unix/sysv/linux/read.c:26 #1 __libc_read (fd=, buf=0xff06898fe977, nbytes=1) at ../sysdeps/unix/sysv/linux/read.c:24 #2 0x0000ff069404d1d8 in SignalHandlerLoop (arg=0xab1a6b6b20c0) at /__w/1/s/src/native/libs/System.Native/pal_signal.c:331 #3 0x0000ff0694d9f648 in start_thread (arg=0xff06898feab0) at pthread_create.c:477 #4 0x0000ff0694a3ac1c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78 Thread 4 (Thread 0xff0691c601b0 (LWP 25) "Finalizer"): #0 futex_abstimed_wait_cancelable (private=0, abstime=0x0, clockid=0, expected=0, futex_word=0xff0694801950 ) at ../sysdeps/nptl/futex-internal.h:323 #1 do_futex_wait (sem=sem@entry=0xff0694801950 , abstime=0x0, clockid=0) at sem_waitcommon.c:112 #2 0x0000ff0694da933c in __new_sem_wait_slow (sem=0xff0694801950 , abstime=0x0, clockid=0) at sem_waitcommon.c:184 #3 0x0000ff0694774514 in mono_os_sem_wait (sem=, flags=MONO_SEM_FLAGS_ALERTABLE) at /__w/1/s/src/mono/mono/metadata/../utils/mono-os-semaphore.h:204 #4 mono_coop_sem_wait (sem=, flags=MONO_SEM_FLAGS_ALERTABLE) at /__w/1/s/src/mono/mono/metadata/../../mono/utils/mono-coop-semaphore.h:41 #5 finalizer_thread (unused=) at /__w/1/s/src/mono/mono/metadata/gc.c:891 #6 0x0000ff069474e744 in start_wrapper_internal (start_info=0x0, stack_ptr=) at /__w/1/s/src/mono/mono/metadata/threads.c:1202 #7 0x0000ff069474e5f0 in start_wrapper (data=0xab1a6b14a3f0) at /__w/1/s/src/mono/mono/metadata/threads.c:1264 #8 0x0000ff0694d9f648 in start_thread (arg=0xff0691c5fab0) at pthread_create.c:477 #9 0x0000ff0694a3ac1c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78 Thread 3 (Thread 0xff0691e611b0 (LWP 24) ".NET EventPipe"): #0 0x0000ff0694a30ef4 in __GI___poll (fds=0xff06840034f0, nfds=1, timeout=) at ../sysdeps/unix/sysv/linux/poll.c:41 #1 0x0000ff0694699474 in ipc_poll_fds (fds=, nfds=1, timeout=4294967295) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:470 #2 ds_ipc_poll (poll_handles_data=0xff06840032e0, poll_handles_data_len=1, timeout_ms=4294967295, callback=0xff0694698850 ) at /__w/1/s/src/native/eventpipe/ds-ipc-pal-socket.c:1098 #3 0x0000ff0694696968 in ds_ipc_stream_factory_get_next_available_stream (callback=0xff0694698850 ) at /__w/1/s/src/native/eventpipe/ds-ipc.c:385 #4 0x0000ff0694695004 in server_thread (data=) at /__w/1/s/src/native/eventpipe/ds-server.c:129 #5 0x0000ff0694698830 in ep_rt_thread_mono_start_func (data=0xab1a6b125300) at /__w/1/s/src/mono/mono/mini/../eventpipe/ep-rt-mono.h:880 #6 0x0000ff0694d9f648 in start_thread (arg=0xff0691e60ab0) at pthread_create.c:477 #7 0x0000ff0694a3ac1c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78 Thread 2 (Thread 0xff0693bff1b0 (LWP 23) "SGen worker"): #0 futex_wait_cancelable (private=0, expected=0, futex_word=0xff069480fe80 ) at ../sysdeps/nptl/futex-internal.h:186 #1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0xff069480fe28 , cond=0xff069480fe58 ) at pthread_cond_wait.c:508 #2 __pthread_cond_wait (cond=0xff069480fe58 , mutex=0xff069480fe28 ) at pthread_cond_wait.c:638 #3 0x0000ff06947d1e20 in mono_os_cond_wait (cond=0xff069480fe80 , mutex=) at /__w/1/s/src/mono/mono/sgen/../../mono/utils/mono-os-mutex.h:219 #4 get_work (worker_index=, work_context=, do_idle=, job=) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:167 #5 thread_func (data=0x0) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:198 #6 0x0000ff0694d9f648 in start_thread (arg=0xff0693bfeab0) at pthread_create.c:477 #7 0x0000ff0694a3ac1c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78 Thread 1 (Thread 0xff0694dec010 (LWP 22) "dotnet"): #0 futex_wait_cancelable (private=0, expected=0, futex_word=0xab1a6c0e25ac) at ../sysdeps/nptl/futex-internal.h:186 #1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0xab1a6c0e2550, cond=0xab1a6c0e2580) at pthread_cond_wait.c:508 #2 __pthread_cond_wait (cond=0xab1a6c0e2580, mutex=0xab1a6c0e2550) at pthread_cond_wait.c:638 #3 0x0000ff069404db24 in SystemNative_LowLevelMonitor_Wait (monitor=0xab1a6c0e2550) at /__w/1/s/src/native/libs/System.Native/pal_threading.c:155 #4 0x0000ff06895cf4cc in ?? () #5 0x0000ff0693d071b0 in ?? () Backtrace stopped: previous frame inner to this frame (corrupt stack?) [Inferior 1 (process 22) detached] ================================================================= Basic Fault Address Reporting ================================================================= instruction pointer is NULL, skip dumping ================================================================= Managed Stacktrace: ================================================================= at <0xffffffff> at System.Reflection.RuntimeMethodInfo:InternalInvoke <0x00007> at System.Reflection.MethodInvoker:InterpretedInvoke <0x00073> at System.Reflection.MethodInvoker:Invoke <0x00103> at System.Reflection.RuntimeMethodInfo:Invoke <0x001a7> at System.Reflection.MethodBase:Invoke <0x00053> at Xunit.Sdk.TestInvoker`1:CallTestMethod <0x00047> at <b__1>d:MoveNext <0x00377> at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0009f> at System.Runtime.CompilerServices.AsyncTaskMethodBuilder:Start <0x0002f> at <>c__DisplayClass48_0:b__1 <0x000cb> at d__4:MoveNext <0x000c3> at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0009b> at Xunit.Sdk.ExecutionTimer:AggregateAsync <0x000eb> at <>c__DisplayClass48_0:b__0 <0x0017b> at d__9:MoveNext <0x00067> at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x00093> at Xunit.Sdk.ExceptionAggregator:RunAsync <0x000e7> at d__48:MoveNext <0x00287> at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000a7> at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002f> at Xunit.Sdk.TestInvoker`1:InvokeTestMethodAsync <0x00113> at Xunit.Sdk.XunitTestInvoker:InvokeTestMethodAsync <0x0011b> at <b__47_0>d:MoveNext <0x00673> at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000af> at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002f> at Xunit.Sdk.TestInvoker`1:b__47_0 <0x000d3> at d__10`1:MoveNext <0x00073> at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x00093> at Xunit.Sdk.ExceptionAggregator:RunAsync <0x000e7> at Xunit.Sdk.TestInvoker`1:RunAsync <0x0010b> at Xunit.Sdk.XunitTestRunner:InvokeTestMethodAsync <0x000cb> at d__4:MoveNext <0x001c3> at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000a3> at Xunit.Sdk.XunitTestRunner:InvokeTestAsync <0x000ef> at <>c__DisplayClass43_0:b__0 <0x00043> at d__10`1:MoveNext <0x00083> at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0009f> at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002f> at Xunit.Sdk.ExceptionAggregator:RunAsync <0x00113> at d__43:MoveNext <0x0048f> at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000af> at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002f> at Xunit.Sdk.TestRunner`1:RunAsync <0x000d3> at Xunit.Sdk.XunitTestCaseRunner:RunTestAsync <0x000df> at d__19:MoveNext <0x0031f> at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000a7> at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002f> at Xunit.Sdk.TestCaseRunner`1:RunAsync <0x000cf> at Xunit.Sdk.XunitTestCase:RunAsync <0x000b3> at Xunit.Sdk.XunitTestMethodRunner:RunTestCaseAsync <0x0009f> at d__32:MoveNext <0x001c3> at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000af> at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002f> at Xunit.Sdk.TestMethodRunner`1:RunTestCasesAsync <0x000d3> at d__31:MoveNext <0x0019f> at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x0009f> at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002f> at Xunit.Sdk.TestMethodRunner`1:RunAsync <0x000cb> at Xunit.Sdk.XunitTestClassRunner:RunTestMethodAsync <0x000df> at d__38:MoveNext <0x007b3> at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000b7> at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002f> at Xunit.Sdk.TestClassRunner`1:RunTestMethodsAsync <0x000d7> at d__37:MoveNext <0x0033f> at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000a7> at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002f> at Xunit.Sdk.TestClassRunner`1:RunAsync <0x000cf> at Xunit.Sdk.XunitTestCollectionRunner:RunTestClassAsync <0x000e3> at d__28:MoveNext <0x003cf> at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000af> at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002f> at Xunit.Sdk.TestCollectionRunner`1:RunTestClassesAsync <0x000d3> at d__27:MoveNext <0x0033f> at System.Runtime.CompilerServices.AsyncMethodBuilderCore:Start <0x000a7> at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1:Start <0x0002f> at Xunit.Sdk.TestCollectionRunner`1:RunAsync <0x000cf> at Xunit.Sdk.XunitTestAssemblyRunner:RunTestCollectionAsync <0x000af> at <>c__DisplayClass14_2:b__2 <0x0006f> at System.Threading.Tasks.Task`1:InnerInvoke <0x0006f> at <>c:<.cctor>b__273_0 <0x0003b> at System.Threading.ExecutionContext:RunInternal <0x000bf> at System.Threading.Tasks.Task:ExecuteWithThreadLocal <0x00257> at System.Threading.Tasks.Task:ExecuteEntry <0x000bb> at <>c:<.cctor>b__8_0 <0x00067> at Xunit.Sdk.MaxConcurrencySyncContext:RunOnSyncContext <0x0004f> at <>c__DisplayClass11_0:b__0 <0x00053> at System.Threading.ExecutionContext:RunInternal <0x000bf> at System.Threading.ExecutionContext:Run <0x00047> at System.Object:lambda_method2 <0x0008b> at Xunit.Sdk.ExecutionContextHelper:Run <0x00063> at Xunit.Sdk.MaxConcurrencySyncContext:WorkerThreadProc <0x0022f> at <>c:b__5_0 <0x00077> at System.Threading.Tasks.Task:InnerInvoke <0x000b7> at <>c:<.cctor>b__273_0 <0x0003b> at System.Threading.ExecutionContext:RunInternal <0x000bf> at System.Threading.Tasks.Task:ExecuteWithThreadLocal <0x00257> at System.Threading.Tasks.Task:ExecuteEntryUnsafe <0x000af> at <>c:<.cctor>b__10_0 <0x0006b> at System.Threading.Thread:StartCallback <0x0012b> at System.Object:runtime_invoke_void__this__ <0x00087> ================================================================= ```

steveisok commented 1 year ago

This is a dupe of https://github.com/dotnet/runtime/issues/81123

dotnet / runtime

Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests segfault in Mono #81249