search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.45k stars 4.76k forks source link

[mono][llvm] b143840.dll fails with LLVM AOT + JIT #48914

Open fanyang-mono opened 3 years ago

fanyang-mono commented 3 years ago

The following runtime test failed on CI lane Mono llvmaot Pri0 Runtime Tests Run Linux arm64 release of PR #48397:

JIT/Regression/CLR-x86-JIT/V1.1-M1-Beta1/b143840/b143840/b143840.sh [FAIL]

      =================================================================
      External Debugger Dump:
      =================================================================
      [New LWP 2178]
      [New LWP 2179]
      [New LWP 2181]
      [New LWP 2196]
      [New LWP 2197]
      [New LWP 2198]
      [New LWP 2199]
      [New LWP 2200]
      [New LWP 2201]
      [New LWP 2202]
      [New LWP 2203]
      [New LWP 2204]
      [New LWP 2205]
      [Thread debugging using libthread_db enabled]
      Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
      0x0000007fa79b322c in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fc8719af0) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      88../sysdeps/unix/sysv/linux/futex-internal.h: No such file or directory.
        Id   Target Id         Frame 
      * 1    Thread 0x7fa7a06fc0 (LWP 2177) "corerun" 0x0000007fa79b322c in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fc8719af0) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
        2    Thread 0x7fa53ff1d0 (LWP 2178) "SGen worker" 0x0000007fa79b322c in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa75a4f10 <work_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
        3    Thread 0x7fa38fe1d0 (LWP 2179) "corerun" 0x0000007fa76a3048 in __GI___poll (fds=0x7f9c003b60, nfds=548265110680, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
        4    Thread 0x7fa36fd1d0 (LWP 2181) "Finalizer" 0x0000007fa79b59c8 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7fa7595e60 <finalizer_sem>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
        5    Thread 0x7fa32ff1d0 (LWP 2196) "corerun" 0x0000007fa79b59c8 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7f98000b80) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
        6    Thread 0x7fa30fe1d0 (LWP 2197) "corerun" 0x0000007fa79b59c8 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7f8c000b80) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
        7    Thread 0x7fa2efd1d0 (LWP 2198) "corerun" 0x0000007fa79b59c8 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7fa759afe0 <suspend_semaphore>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
        8    Thread 0x7fa2cfc1d0 (LWP 2199) "corerun" 0x0000007fa79b59c8 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7f84000b80) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
        9    Thread 0x7fa2afb1d0 (LWP 2200) "corerun" 0x0000007fa79b65f0 in __lll_lock_wait (futex=futex@entry=0x7f90003690, private=0) at lowlevellock.c:43
        10   Thread 0x7fa28fa1d0 (LWP 2201) "corerun" 0x0000007fa79b59c8 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7f7c000b80) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
        11   Thread 0x7fa26f91d0 (LWP 2202) "corerun" 0x0000007fa79b59c8 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7f80000b80) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
        12   Thread 0x7fa24f81d0 (LWP 2203) "corerun" 0x0000007fa79b59c8 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7f74000b80) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
        13   Thread 0x7fa22f71d0 (LWP 2204) "corerun" 0x0000007fa79b59c8 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7f78000b80) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
        14   Thread 0x7fa20f61d0 (LWP 2205) "corerun" 0x0000007fa79b7ce4 in __waitpid (pid=<optimized out>, stat_loc=0x7fa20f0bf4, options=<optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:30

      Thread 14 (Thread 0x7fa20f61d0 (LWP 2205)):
      #0  0x0000007fa79b7ce4 in __waitpid (pid=<optimized out>, stat_loc=0x7fa20f0bf4, options=<optimized out>) at ../sysdeps/unix/sysv/linux/waitpid.c:30
      #1  0x0000007fa62a5c50 in dump_native_stacktrace (signal=<optimized out>, mctx=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:1052
      #2  mono_dump_native_crash_info (signal=<optimized out>, mctx=<optimized out>, info=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-posix.c:1096
      #3  0x0000007fa62635d0 in mono_handle_native_crash (signal=0x7fa7254041 "SIGSEGV", mctx=0x7fa20f1780, info=0x7fa20f1ae0) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:3389
      #4  0x0000007fa61d2080 in mono_sigsegv_signal_handler_debug (_dummy=11, _info=0x7fa20f1ae0, context=0x7fa20f1b60, debug_fault_addr=0x55ccd564) at /__w/1/s/src/mono/mono/mini/mini-runtime.c:3575
      #5  <signal handler called>
      #6  mono_class_is_assignable_from_general (klass=0x55ccb5a0f8, oklass=0x55ccd544, signature_assignment=0, result=0x7fa20f2e3c, error=0x7fa20f2de8) at /__w/1/s/src/mono/mono/metadata/class.c:3876
      #7  0x0000007fa6081454 in mono_class_is_assignable_from_checked (klass=0x55ccb5a0f8, oklass=0x55ccd544, result=0x7fa20f2e3c, error=0x7fa20f2de8) at /__w/1/s/src/mono/mono/metadata/class.c:3852
      #8  mono_class_is_assignable_from_internal (klass=0x55ccb5a0f8, oklass=0x55ccd544) at /__w/1/s/src/mono/mono/metadata/class.c:3789
      #9  0x0000007fa60eac00 in mono_object_handle_isinst (obj=..., klass=0x55ccb5a0f8, error=0x7fa20f32b8) at /__w/1/s/src/mono/mono/metadata/object.c:6578
      #10 0x0000007fa60eacd4 in mono_object_isinst_checked (obj_raw=0x7fa5610880, klass=0x55ccb5a0f8, error=0x7fa20f32b8) at /__w/1/s/src/mono/mono/metadata/object.c:6553
      #11 0x0000007fa62618dc in mono_handle_exception_internal (ctx=0x7fa20f3b00, obj=0x7fa5610880, resume=1, out_ji=0x0) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:2554
      #12 0x0000007fa62639a8 in mono_resume_unwind (ctx=0x7fa20f3e40) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:3489
      #13 0x0000007fa62a1c18 in mono_arm_resume_unwind (arg=<optimized out>, pc=2, int_regs=0x0, fp_regs=0x7fa20f2e3c, corlib=<optimized out>, rethrow=<optimized out>) at /__w/1/s/src/mono/mono/mini/exceptions-arm64.c:422
      #14 0x0000007fa5ae8fa8 in ?? ()
      #15 0x0000007fa20f6370 in ?? ()
      Backtrace stopped: previous frame inner to this frame (corrupt stack?)

      Thread 13 (Thread 0x7fa22f71d0 (LWP 2204)):
      #0  0x0000007fa79b59c8 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7f78000b80) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  do_futex_wait (sem=sem@entry=0x7f78000b80, abstime=0x0) at sem_waitcommon.c:111
      #2  0x0000007fa79b5ae8 in __new_sem_wait_slow (sem=0x7f78000b80, abstime=0x0) at sem_waitcommon.c:181
      #3  0x0000007fa615ecac in mono_os_sem_wait (sem=0x7f78000b80, flags=MONO_SEM_FLAGS_NONE) at /__w/1/s/src/mono/mono/utils/mono-os-semaphore.h:204
      #4  mono_thread_info_wait_for_resume (info=<optimized out>) at /__w/1/s/src/mono/mono/utils/mono-threads.c:236
      #5  0x0000007fa6164e94 in mono_threads_state_poll_with_info (info=0x7f78000b20) at /__w/1/s/src/mono/mono/utils/mono-threads-coop.c:149
      #6  0x0000007fa5929ad4 in ?? ()
      Backtrace stopped: previous frame identical to this frame (corrupt stack?)

      Thread 12 (Thread 0x7fa24f81d0 (LWP 2203)):
      #0  0x0000007fa79b59c8 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7f74000b80) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  do_futex_wait (sem=sem@entry=0x7f74000b80, abstime=0x0) at sem_waitcommon.c:111
      #2  0x0000007fa79b5ae8 in __new_sem_wait_slow (sem=0x7f74000b80, abstime=0x0) at sem_waitcommon.c:181
      #3  0x0000007fa615ecac in mono_os_sem_wait (sem=0x7f74000b80, flags=MONO_SEM_FLAGS_NONE) at /__w/1/s/src/mono/mono/utils/mono-os-semaphore.h:204
      #4  mono_thread_info_wait_for_resume (info=<optimized out>) at /__w/1/s/src/mono/mono/utils/mono-threads.c:236
      #5  0x0000007fa6164e94 in mono_threads_state_poll_with_info (info=0x7f74000b20) at /__w/1/s/src/mono/mono/utils/mono-threads-coop.c:149
      #6  0x0000007fa61650c8 in mono_threads_enter_gc_safe_region_unbalanced_with_info (info=0x7f74000b20, stackdata=<optimized out>) at /__w/1/s/src/mono/mono/utils/mono-threads-coop.c:329
      #7  0x0000007fa6174ea0 in mono_coop_mutex_lock (mutex=<optimized out>) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-mutex.h:55
      #8  sgen_gc_lock () at /__w/1/s/src/mono/mono/sgen/sgen-gc.c:3914
      #9  0x0000007fa6133db8 in mono_gc_alloc_vector (vtable=0x7f8c5ad9c8, size=2528, max_length=312) at /__w/1/s/src/mono/mono/metadata/sgen-mono.c:1338
      #10 0x0000007fa60e9bb8 in mono_array_new_specific_internal (vtable=0x7f8c5ad9c8, n=312, pinned=0, error=0x7fa24f50b0) at /__w/1/s/src/mono/mono/metadata/object.c:5799
      #11 mono_array_new_specific_checked (vtable=0x7f8c5ad9c8, n=312, error=0x7fa24f50b0) at /__w/1/s/src/mono/mono/metadata/object.c:5814
      #12 0x0000007fa60ec598 in mono_array_new_checked (eclass=0x55ccb54cd8, n=393, error=0x7fa24f50b0) at /__w/1/s/src/mono/mono/metadata/object.c:5759
      #13 mono_glist_to_array (list=0x7f74008150, eclass=0x55ccb54cd8, error=0x7fa24f50b0) at /__w/1/s/src/mono/mono/metadata/object.c:7688
      #14 0x0000007fa6264e04 in setup_stack_trace (mono_ex=0x7fa5698930, dynamic_methods=0x7fa24f5cd0, trace_ips=<optimized out>, remove_wrappers=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:2125
      #15 0x0000007fa6262ce0 in handle_exception_first_pass (ctx=0x7fa24f5240, obj=<optimized out>, out_ji=<optimized out>, non_exception=<optimized out>, catch_frame=0x7fa24f51d0, out_filter_idx=<optimized out>, out_prev_ji=<optimized out>, last_mono_wrapper_runtime_invoke=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:2432
      #16 mono_handle_exception_internal (ctx=0x7fa24f5db0, obj=<optimized out>, resume=0, out_ji=0x0) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:2667
      #17 0x0000007fa62617ec in mono_handle_exception (ctx=0x7f74000b80, void_obj=0x189) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:3058
      #18 0x0000007fa62a1ac8 in mono_arm_throw_exception (arg=<optimized out>, pc=<optimized out>, int_regs=0x7fa24f6170, fp_regs=0x7fa24f6270, corlib=<optimized out>, rethrow=0, preserve_ips=0) at /__w/1/s/src/mono/mono/mini/exceptions-arm64.c:401
      #19 0x0000007fa5ae8aac in ?? ()
      #20 0x0000007fa24f8370 in ?? ()
      Backtrace stopped: previous frame identical to this frame (corrupt stack?)

      Thread 11 (Thread 0x7fa26f91d0 (LWP 2202)):
      #0  0x0000007fa79b59c8 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7f80000b80) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  do_futex_wait (sem=sem@entry=0x7f80000b80, abstime=0x0) at sem_waitcommon.c:111
      #2  0x0000007fa79b5ae8 in __new_sem_wait_slow (sem=0x7f80000b80, abstime=0x0) at sem_waitcommon.c:181
      #3  0x0000007fa615ecac in mono_os_sem_wait (sem=0x7f80000b80, flags=MONO_SEM_FLAGS_NONE) at /__w/1/s/src/mono/mono/utils/mono-os-semaphore.h:204
      #4  mono_thread_info_wait_for_resume (info=<optimized out>) at /__w/1/s/src/mono/mono/utils/mono-threads.c:236
      #5  0x0000007fa6164e94 in mono_threads_state_poll_with_info (info=0x7f80000b20) at /__w/1/s/src/mono/mono/utils/mono-threads-coop.c:149
      #6  0x0000007fa61650c8 in mono_threads_enter_gc_safe_region_unbalanced_with_info (info=0x7f80000b20, stackdata=<optimized out>) at /__w/1/s/src/mono/mono/utils/mono-threads-coop.c:329
      #7  0x0000007fa6174ea0 in mono_coop_mutex_lock (mutex=<optimized out>) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-mutex.h:55
      #8  sgen_gc_lock () at /__w/1/s/src/mono/mono/sgen/sgen-gc.c:3914
      #9  0x0000007fa6133db8 in mono_gc_alloc_vector (vtable=0x7f8c5ad9c8, size=2528, max_length=312) at /__w/1/s/src/mono/mono/metadata/sgen-mono.c:1338
      #10 0x0000007fa60e9bb8 in mono_array_new_specific_internal (vtable=0x7f8c5ad9c8, n=312, pinned=0, error=0x7fa26f60b0) at /__w/1/s/src/mono/mono/metadata/object.c:5799
      #11 mono_array_new_specific_checked (vtable=0x7f8c5ad9c8, n=312, error=0x7fa26f60b0) at /__w/1/s/src/mono/mono/metadata/object.c:5814
      #12 0x0000007fa60ec598 in mono_array_new_checked (eclass=0x55ccb54cd8, n=393, error=0x7fa26f60b0) at /__w/1/s/src/mono/mono/metadata/object.c:5759
      #13 mono_glist_to_array (list=0x7f8001ffd0, eclass=0x55ccb54cd8, error=0x7fa26f60b0) at /__w/1/s/src/mono/mono/metadata/object.c:7688
      #14 0x0000007fa6264e04 in setup_stack_trace (mono_ex=0x7fa57f0f70, dynamic_methods=0x7fa26f6cd0, trace_ips=<optimized out>, remove_wrappers=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:2125
      #15 0x0000007fa6262ce0 in handle_exception_first_pass (ctx=0x7fa26f6240, obj=<optimized out>, out_ji=<optimized out>, non_exception=<optimized out>, catch_frame=0x7fa26f61d0, out_filter_idx=<optimized out>, out_prev_ji=<optimized out>, last_mono_wrapper_runtime_invoke=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:2432
      #16 mono_handle_exception_internal (ctx=0x7fa26f6db0, obj=<optimized out>, resume=0, out_ji=0x0) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:2667
      #17 0x0000007fa62617ec in mono_handle_exception (ctx=0x7f80000b80, void_obj=0x189) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:3058
      #18 0x0000007fa62a1ac8 in mono_arm_throw_exception (arg=<optimized out>, pc=<optimized out>, int_regs=0x7fa26f7170, fp_regs=0x7fa26f7270, corlib=<optimized out>, rethrow=0, preserve_ips=0) at /__w/1/s/src/mono/mono/mini/exceptions-arm64.c:401
      #19 0x0000007fa5ae8aac in ?? ()
      #20 0x0000007fa26f9370 in ?? ()
      Backtrace stopped: previous frame identical to this frame (corrupt stack?)

      Thread 10 (Thread 0x7fa28fa1d0 (LWP 2201)):
      #0  0x0000007fa79b59c8 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7f7c000b80) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  do_futex_wait (sem=sem@entry=0x7f7c000b80, abstime=0x0) at sem_waitcommon.c:111
      #2  0x0000007fa79b5ae8 in __new_sem_wait_slow (sem=0x7f7c000b80, abstime=0x0) at sem_waitcommon.c:181
      #3  0x0000007fa615ecac in mono_os_sem_wait (sem=0x7f7c000b80, flags=MONO_SEM_FLAGS_NONE) at /__w/1/s/src/mono/mono/utils/mono-os-semaphore.h:204
      #4  mono_thread_info_wait_for_resume (info=<optimized out>) at /__w/1/s/src/mono/mono/utils/mono-threads.c:236
      #5  0x0000007fa6165250 in mono_threads_exit_gc_safe_region_unbalanced_internal (cookie=0x7f7c000b20, stackdata=<optimized out>) at /__w/1/s/src/mono/mono/utils/mono-threads-coop.c:392
      #6  0x0000007fa6126784 in mono_coop_cond_wait (cond=0x7f900036d0, mutex=<optimized out>) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-mutex.h:93
      #7  mono_monitor_try_enter_inflated (obj=0x7fa540a0f8, ms=4294967295, allow_interruption=0, id=<optimized out>) at /__w/1/s/src/mono/mono/metadata/monitor.c:930
      #8  0x0000007fa612518c in mono_monitor_try_enter_loop_if_interrupted (obj=0x7fa540a0f8, ms=4294967295, allow_interruption=<optimized out>, lockTaken=0x7fa28f8ca8 "", error=0x0) at /__w/1/s/src/mono/mono/metadata/monitor.c:1194
      #9  0x0000007fa588e7e4 in ?? ()
      #10 0x0000007f7c0033e0 in ?? ()
      Backtrace stopped: previous frame identical to this frame (corrupt stack?)

      Thread 9 (Thread 0x7fa2afb1d0 (LWP 2200)):
      #0  0x0000007fa79b65f0 in __lll_lock_wait (futex=futex@entry=0x7f90003690, private=0) at lowlevellock.c:43
      #1  0x0000007fa79b158c in __pthread_mutex_cond_lock (mutex=mutex@entry=0x7f90003690) at ../nptl/pthread_mutex_lock.c:78
      #2  0x0000007fa79b30fc in __pthread_cond_wait_common (abstime=0x0, mutex=0x7f90003690, cond=0x7f900036d0) at pthread_cond_wait.c:645
      #3  __pthread_cond_wait (cond=0x7f900036d0, mutex=0x7f90003690) at pthread_cond_wait.c:655
      #4  0x0000007fa6126770 in mono_os_cond_wait (cond=0x7f900036d0, mutex=0x7f90003690) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-os-mutex.h:219
      #5  mono_coop_cond_wait (cond=0x7f900036d0, mutex=0x7f90003690) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-mutex.h:91
      #6  mono_monitor_try_enter_inflated (obj=0x7fa540a0f8, ms=4294967295, allow_interruption=0, id=<optimized out>) at /__w/1/s/src/mono/mono/metadata/monitor.c:930
      #7  0x0000007fa612518c in mono_monitor_try_enter_loop_if_interrupted (obj=0x7fa540a0f8, ms=4294967295, allow_interruption=<optimized out>, lockTaken=0x7fa2af9ca8 "", error=0x0) at /__w/1/s/src/mono/mono/metadata/monitor.c:1194
      #8  0x0000007fa588e7e4 in ?? ()
      #9  0x0000007f88003180 in ?? ()
      Backtrace stopped: previous frame identical to this frame (corrupt stack?)

      Thread 8 (Thread 0x7fa2cfc1d0 (LWP 2199)):
      #0  0x0000007fa79b59c8 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7f84000b80) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  do_futex_wait (sem=sem@entry=0x7f84000b80, abstime=0x0) at sem_waitcommon.c:111
      #2  0x0000007fa79b5ae8 in __new_sem_wait_slow (sem=0x7f84000b80, abstime=0x0) at sem_waitcommon.c:181
      #3  0x0000007fa615ecac in mono_os_sem_wait (sem=0x7f84000b80, flags=MONO_SEM_FLAGS_NONE) at /__w/1/s/src/mono/mono/utils/mono-os-semaphore.h:204
      #4  mono_thread_info_wait_for_resume (info=<optimized out>) at /__w/1/s/src/mono/mono/utils/mono-threads.c:236
      #5  0x0000007fa6164e94 in mono_threads_state_poll_with_info (info=0x7f84000b20) at /__w/1/s/src/mono/mono/utils/mono-threads-coop.c:149
      #6  0x0000007fa61650c8 in mono_threads_enter_gc_safe_region_unbalanced_with_info (info=0x7f84000b20, stackdata=<optimized out>) at /__w/1/s/src/mono/mono/utils/mono-threads-coop.c:329
      #7  0x0000007fa6174ea0 in mono_coop_mutex_lock (mutex=<optimized out>) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-mutex.h:55
      #8  sgen_gc_lock () at /__w/1/s/src/mono/mono/sgen/sgen-gc.c:3914
      #9  0x0000007fa6133db8 in mono_gc_alloc_vector (vtable=0x7f8c5ad9c8, size=2528, max_length=312) at /__w/1/s/src/mono/mono/metadata/sgen-mono.c:1338
      #10 0x0000007fa60e9bb8 in mono_array_new_specific_internal (vtable=0x7f8c5ad9c8, n=312, pinned=0, error=0x7fa2cf90b0) at /__w/1/s/src/mono/mono/metadata/object.c:5799
      #11 mono_array_new_specific_checked (vtable=0x7f8c5ad9c8, n=312, error=0x7fa2cf90b0) at /__w/1/s/src/mono/mono/metadata/object.c:5814
      #12 0x0000007fa60ec598 in mono_array_new_checked (eclass=0x55ccb54cd8, n=393, error=0x7fa2cf90b0) at /__w/1/s/src/mono/mono/metadata/object.c:5759
      #13 mono_glist_to_array (list=0x7f840081a0, eclass=0x55ccb54cd8, error=0x7fa2cf90b0) at /__w/1/s/src/mono/mono/metadata/object.c:7688
      #14 0x0000007fa6264e04 in setup_stack_trace (mono_ex=0x7fa5508590, dynamic_methods=0x7fa2cf9cd0, trace_ips=<optimized out>, remove_wrappers=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:2125
      #15 0x0000007fa6262ce0 in handle_exception_first_pass (ctx=0x7fa2cf9240, obj=<optimized out>, out_ji=<optimized out>, non_exception=<optimized out>, catch_frame=0x7fa2cf91d0, out_filter_idx=<optimized out>, out_prev_ji=<optimized out>, last_mono_wrapper_runtime_invoke=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:2432
      #16 mono_handle_exception_internal (ctx=0x7fa2cf9db0, obj=<optimized out>, resume=0, out_ji=0x0) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:2667
      #17 0x0000007fa62617ec in mono_handle_exception (ctx=0x7f84000b80, void_obj=0x189) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:3058
      #18 0x0000007fa62a1ac8 in mono_arm_throw_exception (arg=<optimized out>, pc=<optimized out>, int_regs=0x7fa2cfa170, fp_regs=0x7fa2cfa270, corlib=<optimized out>, rethrow=0, preserve_ips=0) at /__w/1/s/src/mono/mono/mini/exceptions-arm64.c:401
      #19 0x0000007fa5ae8aac in ?? ()
      #20 0x0000007fa2cfc370 in ?? ()
      Backtrace stopped: previous frame identical to this frame (corrupt stack?)

      Thread 7 (Thread 0x7fa2efd1d0 (LWP 2198)):
      #0  0x0000007fa79b59c8 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7fa759afe0 <suspend_semaphore>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  do_futex_wait (sem=sem@entry=0x7fa759afe0 <suspend_semaphore>, abstime=0x0) at sem_waitcommon.c:111
      #2  0x0000007fa79b5ae8 in __new_sem_wait_slow (sem=0x7fa759afe0 <suspend_semaphore>, abstime=0x0) at sem_waitcommon.c:181
      #3  0x0000007fa615f164 in mono_os_sem_wait (sem=<optimized out>, flags=<optimized out>) at /__w/1/s/src/mono/mono/utils/mono-os-semaphore.h:204
      #4  mono_os_sem_timedwait (sem=0x7fa759afe0 <suspend_semaphore>, timeout_ms=4294967295, flags=MONO_SEM_FLAGS_NONE) at /__w/1/s/src/mono/mono/utils/mono-os-semaphore.h:237
      #5  0x0000007fa615ee88 in mono_threads_wait_pending_operations () at /__w/1/s/src/mono/mono/utils/mono-threads.c:322
      #6  0x0000007fa6131524 in unified_suspend_stop_world (flags=MONO_THREAD_INFO_FLAGS_NO_GC, thread_stopped_callback=0x7fa6131950 <sgen_client_stop_world_thread_stopped_callback>) at /__w/1/s/src/mono/mono/metadata/sgen-stw.c:343
      #7  0x0000007fa613120c in sgen_client_stop_world (generation=0, serial_collection=0) at /__w/1/s/src/mono/mono/metadata/sgen-stw.c:153
      #8  0x0000007fa61779b4 in sgen_stop_world (generation=0, serial_collection=0) at /__w/1/s/src/mono/mono/sgen/sgen-gc.c:3972
      #9  0x0000007fa61744f0 in sgen_perform_collection_inner (requested_size=<optimized out>, generation_to_collect=<optimized out>, reason=<optimized out>, forced_serial=<optimized out>, stw=<optimized out>) at /__w/1/s/src/mono/mono/sgen/sgen-gc.c:2640
      #10 sgen_perform_collection (requested_size=2528, generation_to_collect=0, reason=0x7fa7255570 "Nursery full", forced_serial=0, stw=1) at /__w/1/s/src/mono/mono/sgen/sgen-gc.c:2753
      #11 0x0000007fa617442c in sgen_ensure_free_space (size=2528, generation=<optimized out>) at /__w/1/s/src/mono/mono/sgen/sgen-gc.c:2619
      #12 0x0000007fa6169284 in sgen_alloc_obj_nolock (vtable=0x7f8c5ad9c8, size=2528) at /__w/1/s/src/mono/mono/sgen/sgen-alloc.c:258
      #13 0x0000007fa6133dc4 in mono_gc_alloc_vector (vtable=0x7f8c5ad9c8, size=2528, max_length=312) at /__w/1/s/src/mono/mono/metadata/sgen-mono.c:1340
      #14 0x0000007fa60e9bb8 in mono_array_new_specific_internal (vtable=0x7f8c5ad9c8, n=312, pinned=0, error=0x7fa2efa0b0) at /__w/1/s/src/mono/mono/metadata/object.c:5799
      #15 mono_array_new_specific_checked (vtable=0x7f8c5ad9c8, n=312, error=0x7fa2efa0b0) at /__w/1/s/src/mono/mono/metadata/object.c:5814
      #16 0x0000007fa60ec598 in mono_array_new_checked (eclass=0x55ccb54cd8, n=393, error=0x7fa2efa0b0) at /__w/1/s/src/mono/mono/metadata/object.c:5759
      #17 mono_glist_to_array (list=0x7f9008ad10, eclass=0x55ccb54cd8, error=0x7fa2efa0b0) at /__w/1/s/src/mono/mono/metadata/object.c:7688
      #18 0x0000007fa6264e04 in setup_stack_trace (mono_ex=0x7fa57e9d10, dynamic_methods=0x7fa2efacd0, trace_ips=<optimized out>, remove_wrappers=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:2125
      #19 0x0000007fa6262ce0 in handle_exception_first_pass (ctx=0x7fa2efa240, obj=<optimized out>, out_ji=<optimized out>, non_exception=<optimized out>, catch_frame=0x7fa2efa1d0, out_filter_idx=<optimized out>, out_prev_ji=<optimized out>, last_mono_wrapper_runtime_invoke=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:2432
      #20 mono_handle_exception_internal (ctx=0x7fa2efadb0, obj=<optimized out>, resume=0, out_ji=0x0) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:2667
      #21 0x0000007fa62617ec in mono_handle_exception (ctx=0x7fa759afe0 <suspend_semaphore>, void_obj=0x189) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:3058
      #22 0x0000007fa62a1ac8 in mono_arm_throw_exception (arg=<optimized out>, pc=<optimized out>, int_regs=0x7fa2efb170, fp_regs=0x7fa2efb270, corlib=<optimized out>, rethrow=0, preserve_ips=0) at /__w/1/s/src/mono/mono/mini/exceptions-arm64.c:401
      #23 0x0000007fa5ae8aac in ?? ()
      #24 0x0000007fa2efd370 in ?? ()
      Backtrace stopped: previous frame identical to this frame (corrupt stack?)

      Thread 6 (Thread 0x7fa30fe1d0 (LWP 2197)):
      #0  0x0000007fa79b59c8 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7f8c000b80) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  do_futex_wait (sem=sem@entry=0x7f8c000b80, abstime=0x0) at sem_waitcommon.c:111
      #2  0x0000007fa79b5ae8 in __new_sem_wait_slow (sem=0x7f8c000b80, abstime=0x0) at sem_waitcommon.c:181
      #3  0x0000007fa615ecac in mono_os_sem_wait (sem=0x7f8c000b80, flags=MONO_SEM_FLAGS_NONE) at /__w/1/s/src/mono/mono/utils/mono-os-semaphore.h:204
      #4  mono_thread_info_wait_for_resume (info=<optimized out>) at /__w/1/s/src/mono/mono/utils/mono-threads.c:236
      #5  0x0000007fa6164e94 in mono_threads_state_poll_with_info (info=0x7f8c000b20) at /__w/1/s/src/mono/mono/utils/mono-threads-coop.c:149
      #6  0x0000007fa5929ad4 in ?? ()
      #7  0x0000000400000004 in ?? ()
      Backtrace stopped: previous frame identical to this frame (corrupt stack?)

      Thread 5 (Thread 0x7fa32ff1d0 (LWP 2196)):
      #0  0x0000007fa79b59c8 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7f98000b80) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  do_futex_wait (sem=sem@entry=0x7f98000b80, abstime=0x0) at sem_waitcommon.c:111
      #2  0x0000007fa79b5ae8 in __new_sem_wait_slow (sem=0x7f98000b80, abstime=0x0) at sem_waitcommon.c:181
      #3  0x0000007fa615ecac in mono_os_sem_wait (sem=0x7f98000b80, flags=MONO_SEM_FLAGS_NONE) at /__w/1/s/src/mono/mono/utils/mono-os-semaphore.h:204
      #4  mono_thread_info_wait_for_resume (info=<optimized out>) at /__w/1/s/src/mono/mono/utils/mono-threads.c:236
      #5  0x0000007fa6164e94 in mono_threads_state_poll_with_info (info=0x7f98000b20) at /__w/1/s/src/mono/mono/utils/mono-threads-coop.c:149
      #6  0x0000007fa61650c8 in mono_threads_enter_gc_safe_region_unbalanced_with_info (info=0x7f98000b20, stackdata=<optimized out>) at /__w/1/s/src/mono/mono/utils/mono-threads-coop.c:329
      #7  0x0000007fa6174ea0 in mono_coop_mutex_lock (mutex=<optimized out>) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-mutex.h:55
      #8  sgen_gc_lock () at /__w/1/s/src/mono/mono/sgen/sgen-gc.c:3914
      #9  0x0000007fa6133db8 in mono_gc_alloc_vector (vtable=0x7f8c5ad9c8, size=2528, max_length=312) at /__w/1/s/src/mono/mono/metadata/sgen-mono.c:1338
      #10 0x0000007fa60e9bb8 in mono_array_new_specific_internal (vtable=0x7f8c5ad9c8, n=312, pinned=0, error=0x7fa32fc0b0) at /__w/1/s/src/mono/mono/metadata/object.c:5799
      #11 mono_array_new_specific_checked (vtable=0x7f8c5ad9c8, n=312, error=0x7fa32fc0b0) at /__w/1/s/src/mono/mono/metadata/object.c:5814
      #12 0x0000007fa60ec598 in mono_array_new_checked (eclass=0x55ccb54cd8, n=393, error=0x7fa32fc0b0) at /__w/1/s/src/mono/mono/metadata/object.c:5759
      #13 mono_glist_to_array (list=0x7f9800f200, eclass=0x55ccb54cd8, error=0x7fa32fc0b0) at /__w/1/s/src/mono/mono/metadata/object.c:7688
      #14 0x0000007fa6264e04 in setup_stack_trace (mono_ex=0x7fa55cf0f0, dynamic_methods=0x7fa32fccd0, trace_ips=<optimized out>, remove_wrappers=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:2125
      #15 0x0000007fa6262ce0 in handle_exception_first_pass (ctx=0x7fa32fc240, obj=<optimized out>, out_ji=<optimized out>, non_exception=<optimized out>, catch_frame=0x7fa32fc1d0, out_filter_idx=<optimized out>, out_prev_ji=<optimized out>, last_mono_wrapper_runtime_invoke=<optimized out>) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:2432
      #16 mono_handle_exception_internal (ctx=0x7fa32fcdb0, obj=<optimized out>, resume=0, out_ji=0x0) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:2667
      #17 0x0000007fa62617ec in mono_handle_exception (ctx=0x7f98000b80, void_obj=0x189) at /__w/1/s/src/mono/mono/mini/mini-exceptions.c:3058
      #18 0x0000007fa62a1ac8 in mono_arm_throw_exception (arg=<optimized out>, pc=<optimized out>, int_regs=0x7fa32fd170, fp_regs=0x7fa32fd270, corlib=<optimized out>, rethrow=0, preserve_ips=0) at /__w/1/s/src/mono/mono/mini/exceptions-arm64.c:401
      #19 0x0000007fa5ae8aac in ?? ()
      #20 0x0000007fa32ff370 in ?? ()
      Backtrace stopped: previous frame identical to this frame (corrupt stack?)

      Thread 4 (Thread 0x7fa36fd1d0 (LWP 2181)):
      #0  0x0000007fa79b59c8 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x7fa7595e60 <finalizer_sem>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
      #1  do_futex_wait (sem=sem@entry=0x7fa7595e60 <finalizer_sem>, abstime=0x0) at sem_waitcommon.c:111
      #2  0x0000007fa79b5ae8 in __new_sem_wait_slow (sem=0x7fa7595e60 <finalizer_sem>, abstime=0x0) at sem_waitcommon.c:181
      #3  0x0000007fa6124318 in mono_os_sem_wait (sem=<optimized out>, flags=MONO_SEM_FLAGS_ALERTABLE) at /__w/1/s/src/mono/mono/mini/../utils/mono-os-semaphore.h:204
      #4  mono_coop_sem_wait (sem=<optimized out>, flags=MONO_SEM_FLAGS_ALERTABLE) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-coop-semaphore.h:41
      #5  finalizer_thread (unused=<optimized out>) at /__w/1/s/src/mono/mono/metadata/gc.c:862
      #6  0x0000007fa60fdaf0 in start_wrapper_internal (start_info=0x0, stack_ptr=<optimized out>) at /__w/1/s/src/mono/mono/metadata/threads.c:1142
      #7  0x0000007fa60fd998 in start_wrapper (data=0x55ccbaddb0) at /__w/1/s/src/mono/mono/metadata/threads.c:1204
      #8  0x0000007fa79ad088 in start_thread (arg=0x7fc8719d3f) at pthread_create.c:463
      #9  0x0000007fa76ac4ec in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

      Thread 3 (Thread 0x7fa38fe1d0 (LWP 2179)):
      #0  0x0000007fa76a3048 in __GI___poll (fds=0x7f9c003b60, nfds=548265110680, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
      #1  0x0000007fa61c4500 in ds_ipc_poll (poll_handles_data=0x7f9c0032d0, poll_handles_data_len=1, timeout_ms=4294967295, callback=0x7fa61c3c80 <server_warning_callback>) at /__w/1/s/src/native/eventpipe/ds-ipc-posix.c:425
      #2  0x0000007fa61c2658 in ds_ipc_stream_factory_get_next_available_stream (callback=0x7fa61c3c80 <server_warning_callback>) at /__w/1/s/src/native/eventpipe/ds-ipc.c:369
      #3  0x0000007fa61c1160 in server_thread (data=<optimized out>) at /__w/1/s/src/native/eventpipe/ds-server.c:124
      #4  0x0000007fa61c3c5c in ep_rt_thread_mono_start_func (data=0x55ccb6d0c0) at /__w/1/s/src/mono/mono/mini/../eventpipe/ep-rt-mono.h:1402
      #5  0x0000007fa79ad088 in start_thread (arg=0x7fc8719e2f) at pthread_create.c:463
      #6  0x0000007fa76ac4ec in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

      Thread 2 (Thread 0x7fa53ff1d0 (LWP 2178)):
      #0  0x0000007fa79b322c in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fa75a4f10 <work_cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fa75a4eb8 <lock>, cond=0x7fa75a4ee8 <work_cond>) at pthread_cond_wait.c:502
      #2  __pthread_cond_wait (cond=0x7fa75a4ee8 <work_cond>, mutex=0x7fa75a4eb8 <lock>) at pthread_cond_wait.c:655
      #3  0x0000007fa61a95ec in mono_os_cond_wait (cond=0x7fa75a4f10 <work_cond+40>, mutex=<optimized out>) at /__w/1/s/src/mono/mono/mini/../../mono/utils/mono-os-mutex.h:219
      #4  get_work (worker_index=<optimized out>, work_context=<optimized out>, do_idle=<optimized out>, job=<optimized out>) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:167
      #5  thread_func (data=0x0) at /__w/1/s/src/mono/mono/sgen/sgen-thread-pool.c:198
      #6  0x0000007fa79ad088 in start_thread (arg=0x7fc8719e3f) at pthread_create.c:463
      #7  0x0000007fa76ac4ec in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

      Thread 1 (Thread 0x7fa7a06fc0 (LWP 2177)):
      #0  0x0000007fa79b322c in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7fc8719af0) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
      #1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fa759ab78 <signal_mutex>, cond=0x7fc8719ac8) at pthread_cond_wait.c:502
      #2  __pthread_cond_wait (cond=0x7fc8719ac8, mutex=0x7fa759ab78 <signal_mutex>) at pthread_cond_wait.c:655
      #3  0x0000007fa61481c4 in mono_os_cond_wait (cond=0x7fc8719ac8, mutex=0x7fa759ab78 <signal_mutex>) at /__w/1/s/src/mono/mono/utils/mono-os-mutex.h:219
      #4  mono_os_event_wait_multiple (events=0x7fc8719b60, nevents=11, waitall=0, timeout=4294967295, alertable=1) at /__w/1/s/src/mono/mono/utils/os-event-unix.c:190
      #5  0x0000007fa6162494 in mono_thread_info_wait_multiple_handle (thread_handles=<optimized out>, nhandles=11, background_change_event=<optimized out>, waitall=<optimized out>, timeout=<optimized out>, alertable=-1482658128) at /__w/1/s/src/mono/mono/utils/mono-threads.c:2121
      #6  0x0000007fa60fa6ec in wait_for_tids (wait=0x7fc8719d88, timeout=4294967295, check_state_change=1) at /__w/1/s/src/mono/mono/metadata/threads.c:3191
      #7  mono_thread_manage_internal () at /__w/1/s/src/mono/mono/metadata/threads.c:3362
      #8  0x0000007fa6228c0c in mono_main (argc=<optimized out>, argv=<optimized out>) at /__w/1/s/src/mono/mono/mini/driver.c:2688
      #9  0x0000007fa6292864 in monovm_execute_assembly (argc=0, argv=0x0, managedAssemblyPath=<optimized out>, exitCode=0x7fc871a46c) at /__w/1/s/src/mono/mono/mini/monovm.c:259
      #10 0x00000055950187d0 in ExecuteManagedAssembly (currentExeAbsolutePath=<optimized out>, clrFilesAbsolutePath=<optimized out>, managedAssemblyAbsolutePath=<optimized out>, managedAssemblyArgc=0, managedAssemblyArgv=<optimized out>) at /__w/1/s/src/coreclr/hosts/unixcorerun/corerun.cpp:385
      #11 0x0000005595018f70 in main (argc=<optimized out>, argv=<optimized out>) at /__w/1/s/src/coreclr/hosts/unixcorerun/corerun.cpp:573
      /root/helix/work/workitem/JIT/Regression/CLR-x86-JIT/V1.1-M1-Beta1/b143840/b143840/b143840.sh: line 404:  2177 Aborted                 (core dumped) $LAUNCHER $ExePath "${CLRTestExecutionArguments[@]}"

cc: @imhameed

imhameed commented 3 years ago

This fails intermittently (but frequently enough) on my amd64 PC, with stacks that look pretty similar: https://gist.github.com/imhameed/9faa48b70be7c540b00f4b3279b9a0c9 https://gist.github.com/imhameed/e397265d683f357db28b4a158e38429a

LLVM AOT compiling the test assembly (with `MONO_ENV_OPTIONS="--aot=llvm,llvm-path=path/prefix/to/opt/and/llc" and then running the assembly with LLVM JIT enabled is sufficient to reproduce this (after a few tries).

imhameed commented 3 years ago

Also fails on arm64 FullAOT. A CI log can be found here. The log is also duplicated here: https://gist.github.com/imhameed/4d2a3d559314e9a7ac112408f616737f

FullAOT tests are run using MONO_ENV_OPTIONS=--full-aot.

FullAOT compilation uses these flags on x64:

MONO_ENV_OPTIONS="--aot=full,llvm,llvm-path=$MONO_PFX,mattr=crc,mattr=crypto"
fanyang-mono commented 2 years ago

Didn't have bandwidth to work on this in .NET7. Moving to milestone 8.0.0.

fanyang-mono commented 1 year ago

Moving to 9.0.0