search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.43k stars 4.76k forks source link

Re-enable arm64 and x64 Mono fullAOT llvm and mini jobs #90427

Open SamMonoRT opened 1 year ago

SamMonoRT commented 1 year ago

Description

Currently, the Mono fullAOT LLVM jobs are encountering failures and don't provide any test coverage. Specifically, the linux-arm64 job must build the cross-compiler using an image which incorporates both x64 and arm64 rootfs. Additionally, the aot-compiler.c utilizes GNU as which is not available since we don't have GNU binutils on the Mariner.

To resolve this issue, we have to fix the mono-aot-cross build for linux-arm64 to use target rootfs when building the runtime and host rootfs when building the compiler. After that, the aot-compiler.c has to be updated to use llvm-as or clang instead of GNU as.

Additionally, the AOT compiler with llvm 16 toolchain hits OOM issue with CBL_Mariner images.

Tasks

Initial discussion can be found at https://github.com/dotnet/runtime/issues/86324#issuecomment-1549896743.

Disabled tests

Upon the initial setup of the jobs, the following tests require triage.

FullAOT mini

AOT compilation failure:

src/tests/JIT/Methodical/Boxing/boxunbox/BoxPatternMatchAndSideEffects.csproj
src/tests/JIT/Methodical/Methodical_others.csproj
src/tests/JIT/Regression/JitBlue/GitHub_26491/GitHub_26491.ilproj
src/tests/JIT/Regression/Regression_3.csproj
src/tests/Interop/StringMarshalling/AnsiBSTR/AnsiBStrTest.csproj
src/tests/Interop/StringMarshalling/BSTR/BSTRTest.csproj
src/tests/Interop/StringMarshalling/LPTSTR/LPTSTRTest.csproj
src/tests/Interop/StringMarshalling/VBByRefStr/VBByRefStrTest.csproj
src/tests/Interop/Interop.csproj
src/tests/readytorun/coreroot_determinism/readytorun_coreroot_determinism.csproj
src/tests/readytorun/readytorun.csproj

Execution failure:

JIT/Methodical/Invoke/25params/25paramMixed_d/**
JIT/Methodical/Invoke/SEH/catchfault_jmp_d/**
JIT/Methodical/Invoke/SEH/catchfinally_jmpind_d/**
JIT/Methodical/Invoke/SEH/catchfinally_jmp_d/**
JIT/Methodical/tailcall/deep_array_d/**
JIT/Methodical/tailcall/deep_array_nz_d/**
JIT/Methodical/tailcall/deep_gc_d/**
JIT/Methodical/tailcall/deep_inst_d/**
JIT/Methodical/tailcall/deep_value_d/**
JIT/Methodical/tailcall/deep_virt_d/**
JIT/Methodical/VT/callconv/jumps2_d/**
JIT/Directed/zeroinit/tail_zeroinit/**
JIT/Directed/callconv/ThisCall/ThisCallTest/**
JIT/Directed/pinvoke/tail_pinvoke/**
JIT/Methodical/Invoke/25params/25paramMixed_r/**
JIT/Methodical/Invoke/SEH/catchfault_jmp_r/**
JIT/Methodical/Invoke/SEH/catchfinally_jmpind_r/**
JIT/Methodical/Invoke/SEH/catchfinally_jmp_r/**
JIT/Methodical/tailcall/deep_array_nz_r/**
JIT/Methodical/tailcall/deep_array_r/**
JIT/Methodical/tailcall/deep_gc_r/**
JIT/Methodical/tailcall/deep_inst_r/**
JIT/Methodical/tailcall/deep_value_r/**
JIT/Methodical/tailcall/deep_virt_r/**
JIT/Methodical/VT/callconv/jumper4_r/**
JIT/Methodical/VT/callconv/jumper5_r/**
JIT/Methodical/VT/callconv/jumps2_r/**
Regressions/coreclr/16354/notimplemented/**
JIT/Directed/perffix/primitivevt/callconv3_d/**
JIT/Directed/perffix/primitivevt/callconv3_r/**
JIT/SIMD/Vector3Interop_r/Vector3Interop_r.**
JIT/SIMD/Vector3Interop_ro/Vector3Interop_ro.**
JIT/Regression/CLR-x86-JIT/V1.2-M01/b13452/b13452/**
JIT/Regression/CLR-x86-JIT/V2.0-Beta2/b353858/b353858/**
JIT/Methodical/Boxing/misc/tailjump_r/**
JIT/Methodical/Boxing/misc/tailjump_d/**
baseservices/compilerservices/UnsafeAccessors/UnsafeAccessorsTests/**
baseservices/TieredCompilation/TieredVtableMethodTests/**
baseservices/varargs/varargsupport/**
JIT/opt/Devirtualization/Comparer_get_Default/Comparer_get_Default.**
JIT/Regression/JitBlue/Runtime_80731/Runtime_80731/**
Loader/classloader/DictionaryExpansion/DictionaryExpansion/**
Loader/classloader/explicitlayout/NestedStructs/case03/**
Loader/classloader/explicitlayout/NestedStructs/case04/**
Loader/classloader/explicitlayout/NestedStructs/case05/**
Loader/classloader/explicitlayout/Regressions/ASURT/ASURT150271/test13/**
Loader/classloader/MethodImpl/generics_override1/**
Loader/classloader/Statics/Misc/LiteralStatic/**
Loader/CustomAttributes/DynamicObjects/**
JIT/jit64/localloc/call/call05_dynamic/**
JIT/jit64/verif/sniff/fg/ver_fg_13/**
JIT/Directed/coverage/importer/badendfinally/**
JIT/Directed/coverage/importer/Desktop/badendfinally_d/**
JIT/Directed/coverage/importer/Desktop/badendfinally_r/**
JIT/Directed/coverage/importer/ceeillegal/**
JIT/Directed/coverage/importer/Desktop/ceeillegal_d/**
JIT/Directed/coverage/importer/Desktop/ceeillegal_r/**
JIT/SIMD/Vector3Interop_r/**
JIT/SIMD/Vector3Interop_ro/**
JIT/opt/Devirtualization/Comparer_get_Default/**
baseservices/varargs/varargsupport_r/**
Loader/classloader/MethodImpl/CovariantReturns/ReturnTypeValidation/**
Loader/classloader/MethodImpl/CovariantReturns/UnitTest/UnitTest_GVM/**
Interop/StringMarshalling/AnsiBSTR/AnsiBStrTest/**
Interop/StringMarshalling/BSTR/BSTRTest/**
Interop/StringMarshalling/LPTSTR/LPTSTRTest/**
Interop/StringMarshalling/VBByRefStr/VBByRefStrTest/**
JIT/Methodical/Boxing/boxunbox/BoxPatternMatchAndSideEffects/**
JIT/Intrinsics/TypeIntrinsics_r/**
JIT/Intrinsics/TypeIntrinsics_ro/**
ilasm/System/Runtime/CompilerServices/MethodImplOptionsTests/**
ilasm/PortablePdb/IlasmPortablePdbTests/**

FullAOT LLVM

AOT compilation failure:

src/tests/JIT/Methodical/Boxing/boxunbox/BoxPatternMatchAndSideEffects.csproj
src/tests/JIT/Methodical/Methodical_others.csproj
src/tests/JIT/Regression/JitBlue/GitHub_26491/GitHub_26491.ilproj
src/tests/JIT/Regression/Regression_3.csproj
src/tests/Interop/StringMarshalling/AnsiBSTR/AnsiBStrTest.csproj
src/tests/Interop/StringMarshalling/BSTR/BSTRTest.csproj
src/tests/Interop/StringMarshalling/LPTSTR/LPTSTRTest.csproj
src/tests/Interop/StringMarshalling/VBByRefStr/VBByRefStrTest.csproj
src/tests/Interop/Interop.csproj
src/tests/readytorun/coreroot_determinism/readytorun_coreroot_determinism.csproj
src/tests/readytorun/readytorun.csproj

Execution failure:

JIT/Methodical/Invoke/25params/25paramMixed_r
JIT/Methodical/Invoke/SEH/catchfault_jmp_r
JIT/Methodical/Invoke/SEH/catchfinally_jmpind_r
JIT/Methodical/Invoke/SEH/catchfinally_jmp_r
JIT/Methodical/tailcall/deep_array_nz_r
JIT/Methodical/tailcall/deep_array_r
JIT/Methodical/tailcall/deep_gc_r
JIT/Methodical/tailcall/deep_inst_r
JIT/Methodical/tailcall/deep_value_r
JIT/Methodical/tailcall/deep_virt_r
JIT/Methodical/VT/callconv/jumper4_r
JIT/Methodical/VT/callconv/jumper5_r
JIT/Methodical/VT/callconv/jumps2_r
JIT/Directed/perffix/primitivevt/callconv3_d
JIT/Directed/perffix/primitivevt/callconv3_r
JIT/jit64/localloc/call/call05_dynamic
JIT/jit64/verif/sniff/fg/ver_fg_13
JIT/Methodical/Invoke/25params/25paramMixed_d
JIT/Methodical/Invoke/SEH/catchfault_jmp_d
JIT/Methodical/Invoke/SEH/catchfinally_jmpind_d
JIT/Methodical/Invoke/SEH/catchfinally_jmp_d
JIT/Methodical/tailcall/deep_array_d
JIT/Methodical/tailcall/deep_array_nz_d
JIT/Methodical/tailcall/deep_gc_d
JIT/Methodical/tailcall/deep_inst_d
JIT/Methodical/tailcall/deep_value_d
JIT/Methodical/tailcall/deep_virt_d
JIT/Methodical/VT/callconv/jumps2_d
JIT/SIMD/Vector3Interop_r
JIT/SIMD/Vector3Interop_ro
JIT/HardwareIntrinsics/X86/Sse2.X64/StoreNonTemporal_r
JIT/HardwareIntrinsics/X86/Sse2.X64/StoreNonTemporal_ro
JIT/Directed/coverage/importer/badendfinally
JIT/Directed/coverage/importer/Desktop/badendfinally_d
JIT/Directed/coverage/importer/Desktop/badendfinally_r
JIT/Directed/coverage/importer/ceeillegal
JIT/Directed/coverage/importer/Desktop/ceeillegal_d
JIT/Directed/coverage/importer/Desktop/ceeillegal_r
JIT/HardwareIntrinsics/General/Vector128_1
baseservices/varargs/varargsupport_r
JIT/Methodical/Boxing/misc/tailjump_r
Regressions/coreclr/16354/notimplemented
JIT/Regression/CLR-x86-JIT/V1.2-M01/b13452/b13452
JIT/Regression/CLR-x86-JIT/V2.0-Beta2/b353858/b353858
JIT/Methodical/Boxing/misc/tailjump_d
JIT/Directed/zeroinit/tail_zeroinit
JIT/Directed/callconv/ThisCall/ThisCallTest
JIT/Directed/pinvoke/tail_pinvoke
Loader/classloader/generics/ByRefLike/ValidateNegative
JIT/Regression/JitBlue/Runtime_80731/Runtime_80731
JIT/opt/Devirtualization/Comparer_get_Default
Loader/classloader/DictionaryExpansion/DictionaryExpansion
Loader/classloader/explicitlayout/NestedStructs/case03
Loader/classloader/explicitlayout/NestedStructs/case04
Loader/classloader/explicitlayout/NestedStructs/case05
Loader/classloader/explicitlayout/Regressions/ASURT/ASURT150271/test13
Loader/classloader/MethodImpl/CovariantReturns/ReturnTypeValidation/ImplicitOverrideSameSigAsDecl
Loader/classloader/MethodImpl/CovariantReturns/ReturnTypeValidation/OverrideSameSigAsDecl
Loader/classloader/MethodImpl/CovariantReturns/UnitTest/UnitTest_GVM
Loader/classloader/MethodImpl/generics_override1
Loader/classloader/Statics/Misc/LiteralStatic
Loader/CustomAttributes/DynamicObjects
baseservices/compilerservices/UnsafeAccessors/UnsafeAccessorsTests
baseservices/TieredCompilation/TieredVtableMethodTests
baseservices/varargs/varargsupport
ghost commented 1 year ago

Tagging subscribers to this area: @directhex See info in area-owners.md if you want to be subscribed.

Issue Details
Refer to https://github.com/dotnet/runtime/issues/86324#issuecomment-1549896743, to re-enable the disabled lane.
Author: SamMonoRT
Assignees: directhex, kotlarmilos
Labels: `area-Infrastructure-mono`
Milestone: 9.0.0
kotlarmilos commented 1 year ago

Mono fullAOT linux runtime tests now utilize the new Mariner image. This image does not include binutils, and as a result, GNU as cannot be used as a native assembler. The AOT compiler utilizes llvm opt and ilc to generate the llvm binary file. Then, a native assembler and linker are required to produce the final binary.

One option is to utilize llvm-mc (clang), which can convert assembly into a binary object file. However, I am not sure if there's a 1:1 mapping between the llvm-mc and GNU as. Another option is to ship binutils with the image.

For linux-arm64, either option would require a cross-compiled toolchain for x64 targeting arm64.

/cc: @directhex @lambdageek

kotlarmilos commented 1 year ago

After several attempts to enable fullAOT llvm runtime tests on the extra-platforms, the job consistently hits OOM issue after 2h. To simplify the process, the job compiles only the baseservices/mono and System.Private.CoreLib with the maxcpucount set to 1.

The CI is using an azure instance with 16gb, docker on top of that is limited to less, probably 8gb. At peak, the AOT compilation takes about 16gb to AOT compile the System.Private.CoreLib. When I tried to reproduce the issue locally on the cbl-mariner-2.0-cross-amd64, it completed the compilation successfully within 10min. There is a tracking issue in azdo where we have requested a hardware upgrade https://github.com/dotnet/dnceng/issues/1163.

@steveisok @SamMonoRT While we work on resolving the OOM issue, I would appreciate hearing your thoughts on temporarily disabling the affected jobs to ensure a consistently green CI.

steveisok commented 1 year ago

I'm fine with disabling for the time being.

I wonder if requiring so much memory is a sign we can improve our aot compiler utilization?

kotlarmilos commented 1 year ago

I wonder if requiring so much memory is a sign we can improve our aot compiler utilization?

Yes, let's create a tracking issue so that we can brainstorm our next steps. Additionally, I'll temporarily disable the failing jobs.