search

mono / mono

Mono open source ECMA CLI, C# and .NET implementation.
https://www.mono-project.com
Other
11.13k stars 3.83k forks source link

Build break/test failures on FullAOT+LLVM configuration on ARM #9229

Open akoeplinger opened 6 years ago

akoeplinger commented 6 years ago

Steps to Reproduce

  1. Run FullAOT+LLVM config
  2. Issue 1: Observe test failures on arm64/armhf
  3. Issue 2: Observe build failures on armel Fixed by https://github.com/mono/mono/pull/9283

Current Behavior

Issue 1:

Test failures on arm64 (https://jenkins.mono-project.com/job/test-mono-mainline-linux-fullaot+llvm/label=debian-9-arm64/588/) and armhf (https://jenkins.mono-project.com/job/test-mono-mainline-linux-fullaot+llvm/label=debian-9-armhf/588/):

Issue 2: Build on armel is broken (https://jenkins.mono-project.com/job/test-mono-mainline-linux-fullaot+llvm/label=debian-9-armel/588/): Fixed by https://github.com/mono/mono/pull/9283

MONO_PATH=".//class/lib/testing_aot_full" /home/builder/jenkins/workspace/test-mono-mainline-linux-fullaot+llvm/label/debian-9-armel/runtime/mono-wrapper  --runtime=mobile -O=gsharedvt --llvm --aot=full,nimt-trampolines=2000,ntrampolines=10000,nrgctx-fetch-trampolines=256,ngsharedvt-trampolines=4000,temp-path=class/lib/testing_aot_full/I18N.Rare.dll_bitcode_tmp --verbose class/lib/testing_aot_full/I18N.Rare.dll > class/lib/testing_aot_full/I18N.Rare.dll.so.aot-log
llc: Unknown command line argument '-soft-float'.  Try: 'llc -help'
llc: Did you mean '-load'?
build/rules.make:194: recipe for target 'class/lib/testing_aot_full/I18N.MidEast.dll.so' failed
AOT of image class/lib/testing_aot_full/I18N.MidEast.dll failed.
make[6]: *** [class/lib/testing_aot_full/I18N.MidEast.dll.so] Error 1
make[6]: *** Waiting for unfinished jobs....
llc: Unknown command line argument '-soft-float'.  Try: 'llc -help'
llc: Did you mean '-load'?
AOT of image class/lib/testing_aot_full/I18N.Other.dll failed.
build/rules.make:194: recipe for target 'class/lib/testing_aot_full/I18N.Other.dll.so' failed
make[6]: *** [class/lib/testing_aot_full/I18N.Other.dll.so] Error 1
llc: Unknown command line argument '-soft-float'.  Try: 'llc -help'
llc: Did you mean '-load'?
AOT of image class/lib/testing_aot_full/I18N.CJK.dll failed.
make[6]: *** [class/lib/testing_aot_full/I18N.CJK.dll.so] Error 1
build/rules.make:194: recipe for target 'class/lib/testing_aot_full/I18N.CJK.dll.so' failed
llc: Unknown command line argument '-soft-float'.  Try: 'llc -help'
llc: Did you mean '-load'?
AOT of image class/lib/testing_aot_full/I18N.Rare.dll failed.
make[6]: *** [class/lib/testing_aot_full/I18N.Rare.dll.so] Error 1
make[5]: *** [do-all-aot] Error 2
make[4]: *** [profile-do--testing_aot_full--all] Error 2
make[3]: *** [profiles-do--all] Error 2
make[2]: *** [all-local] Error 2
build/rules.make:194: recipe for target 'class/lib/testing_aot_full/I18N.Rare.dll.so' failed
make[6]: Leaving directory '/home/builder/jenkins/workspace/test-mono-mainline-linux-fullaot+llvm/label/debian-9-armel/mcs'
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2

Expected Behavior

No build/test failures

On which platforms did you notice this

[ ] macOS [ X ] Linux [ ] Windows

Version Used: master

Continuation from https://github.com/mono/mono/issues/8321

vargaz commented 6 years ago

6.0 is MONO_API_VERSION>=600

vargaz commented 6 years ago

Its already done:

ifdef LLVM_API_VERSION > 100

    g_string_append (acfg->llc_args, " -mattr=+soft-float");

else

    g_string_append (acfg->llc_args, " -soft-float");

endif

alexanderkyte commented 6 years ago

After checking with @akoeplinger , it looks like the x86 failures are fixed. I'm going to investigate reproducing the armhf failures locally before I close this.

alexanderkyte commented 6 years ago

Issue encountered/fixed: https://github.com/mono/mono/pull/11074

alexanderkyte commented 6 years ago

I'm not seeing the failures linked but I'm definitely seeing some failures: 

Executing llc: "llc"  -mattr=+vfp2,-neon,+d16 -float-abi=hard -asm-verbose=false -disable-gnu-eh-frame -enable-mono-eh-frame -mono-eh-frame-symbol=mono_aot_I18N_CJK_eh_frame -disable-tail-calls -relocation-model=pic -filetype=obj -o "mono_aot_BTOu5V/temp-llvm.o" "mono_aot_BTOu5V/temp.opt.bc"
Stack dump:
0.      Program arguments: llc -mattr=+vfp2,-neon,+d16 -float-abi=hard -asm-verbose=false -disable-gnu-eh-frame -enable-mono-eh-frame -mono-eh-frame-symbol=mono_aot_I18N_CJK_eh_frame -disable-tail-calls -relocation-model=pic -filetype=obj -o mono_aot_BTOu5V/temp-llvm.o mono_aot_BTOu5V/temp.opt.bc 
1.      Running pass 'Function Pass Manager' on module 'mono_aot_BTOu5V/temp.opt.bc'.
2.      Running pass 'ARM Instruction Selection' on function '@I18N_CJK_CP51932__ctor'
Segmentation fault (core dumped)
AOT of image ../../class/lib/testing_aot_full/I18N.CJK.dll failed.
alexanderkyte commented 6 years ago

https://github.com/mono/mono/pull/11172 fixes ^

alexanderkyte commented 6 years ago

Currently seeing: 

Thread 1 (Thread 0xf7bdcc80 (LWP 10962)):
#0  0xf79fd344 in __libc_do_syscall () from /lib/arm-linux-gnueabihf/libpthread.so.0
#1  0xf79fc0e6 in waitpid () from /lib/arm-linux-gnueabihf/libpthread.so.0
#2  0x002953fc in dump_native_stacktrace (signal=0x8a09ac "SIGSEGV", ctx=0xff9ea3e8) at mini-posix.c:1084
#3  0x00294fa8 in mono_dump_native_crash_info (signal=0x8a09ac "SIGSEGV", ctx=0xff9ea3e8, info=0xff9ea368) at mini-posix.c:1107
#4  0x001f10cc in mono_handle_native_crash (signal=0x8a09ac "SIGSEGV", ctx=0xff9ea3e8, info=0xff9ea368) at mini-exceptions.c:3038
#5  0x000f1800 in mono_sigsegv_signal_handler_debug (_dummy=11, _info=0xff9ea368, context=0xff9ea3e8, debug_fault_addr=0x0) at mini-runtime.c:3504
#6  0x000f1600 in mono_sigsegv_signal_handler (_dummy=11, _info=0xff9ea368, context=0xff9ea3e8) at mini-runtime.c:3533
#7  <signal handler called>
#8  0x00000000 in ?? ()
#9  0xf539bbb8 in System_Collections_Generic_ArraySortHelper_1_T_REF_SwapIfGreater_T_REF___System_Comparison_1_T_REF_int_int () from /home/builder/mono/mcs/class/lib/testing_aot_full/mscorlib.dll.so

now

timmi-on-rails commented 5 years ago

Can you please check if there is a relation with the LLVM bug in issue #11095? There seems to go something wrong with passing parameters as @lewurm pointed out...

lewurm commented 5 years ago

@akoeplinger do you still see those issues on CI?

akoeplinger commented 5 years ago

@lewurm yes, armel/armhf are still broken:

lewurm commented 5 years ago

all the crashes I see are hitting this issue:

Thread 1 (Thread 0xf7a4bbb0 (LWP 24250)):
#0  0xf77de8ec in waitpid () from /lib/arm-linux-gnueabi/libpthread.so.0
#1  0x00d4136c in dump_native_stacktrace (ctx=0xffe17340, ctx@entry=0x0, signal=0x139e83c "SIGSEGV") at mini-posix.c:1083
#2  0x00d41680 in mono_dump_native_crash_info (signal=signal@entry=0x139e83c "SIGSEGV", ctx=0x0, ctx@entry=0xffe17340, info=info@entry=0xffe172c0) at mini-posix.c:1112
#3  0x00ce2d1c in mono_handle_native_crash (signal=0x139e83c "SIGSEGV", ctx=ctx@entry=0xffe17340, info=info@entry=0xffe172c0) at mini-exceptions.c:3196
#4  0x00c58500 in mono_sigsegv_signal_handler_debug (_dummy=11, _info=0xffe172c0, context=0xffe17340, debug_fault_addr=0x0) at mini-runtime.c:3551
#5  <signal handler called>
#6  0x00000000 in ?? ()
#7  0xf537ce9c in System_Collections_Generic_ArraySortHelper_1_T_REF_SwapIfGreater_T_REF___System_Comparison_1_T_REF_int_int () from /home/builder/jenkins/workspace/test-mono-mainline-linux-fullaot+llvm/label/debian-9-armel/mcs/class/lib/testing_aot_full/mscorlib.dll.so
#8  0xf537d270 in System_Collections_Generic_ArraySortHelper_1_T_REF_PickPivotAndPartition_T_REF___int_int_System_Comparison_1_T_REF () from /home/builder/jenkins/workspace/test-mono-mainline-linux-fullaot+llvm/label/debian-9-armel/mcs/class/lib/testing_aot_full/mscorlib.dll.so
#9  0xf537d0d8 in System_Collections_Generic_ArraySortHelper_1_T_REF_IntroSort_T_REF___int_int_int_System_Comparison_1_T_REF () from /home/builder/jenkins/workspace/test-mono-mainline-linux-fullaot+llvm/label/debian-9-armel/mcs/class/lib/testing_aot_full/mscorlib.dll.so
#10 0xf537d068 in System_Collections_Generic_ArraySortHelper_1_T_REF_IntrospectiveSort_T_REF___int_int_System_Comparison_1_T_REF () from /home/builder/jenkins/workspace/test-mono-mainline-linux-fullaot+llvm/label/debian-9-armel/mcs/class/lib/testing_aot_full/mscorlib.dll.so
#11 0xf567017c in System_Collections_Generic_ArraySortHelper_1_T_REF_Sort_T_REF___int_int_System_Collections_Generic_IComparer_1_T_REF () from /home/builder/jenkins/workspace/test-mono-mainline-linux-fullaot+llvm/label/debian-9-armel/mcs/class/lib/testing_aot_full/mscorlib.dll.so
#12 0xf5065b94 in System_Array_Sort_T_REF_T_REF___int_int_System_Collections_Generic_IComparer_1_T_REF () from