Sigsegv in dotnet/msbuild when building repo

[MichalStrehovsky]

Hit in #100945 while building the GenerateHWIntrinsicTests_Arm.csproj project:

2024-04-15T09:34:59.7728930Z     /Users/runner/work/1/s/dotnet.sh: line 27: 91742 Segmentation fault: 11  ${dotnetPath} "$@"
2024-04-15T09:34:59.8872770Z /Users/runner/work/1/s/src/tests/JIT/HardwareIntrinsics/Arm/Directory.Build.targets(15,5): error MSB3073: The command ""/Users/runner/work/1/s/dotnet.sh" build /Users/runner/work/1/s//src/tests/Common/GenerateHWIntrinsicTests/GenerateHWIntrinsicTests_Arm.csproj -c Release --no-restore /p:IntermediateOutputPath=/Users/runner/work/1/s/artifacts/tests/coreclr/obj/osx.x64.Release/Managed/JIT/HardwareIntrinsics/Arm/Sve/Sve_r//GenerateHWIntrinsicTests/ /p:OutputPath=/Users/runner/work/1/s/artifacts/tests/coreclr/osx.x64.Release/JIT/HardwareIntrinsics/Arm/Sve/Sve_r//GenerateHWIntrinsicTests/" exited with code 139. [/Users/runner/work/1/s/src/tests/JIT/HardwareIntrinsics/Arm/Sve/Sve_r.csproj] [/Users/runner/work/1/s/src/tests/build.proj]

We don't have a dump because the infra is not capable of collecting dumps on the build machine. Hit on a macOS machine, so could be macOS specific, but it's also non-deterministic, so...

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=643372 Build error leg or test failing: Build osx-x64 Release AllSubsets_Mono_Minijit_RuntimeTests minijit Pull request: https://github.com/dotnet/runtime/pull/100945

Error Message

{
  "ErrorMessage": "Segmentation fault: 11  ${dotnetPath}",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=643372 Error message validated: [Segmentation fault: 11 ${dotnetPath}] Result validation: :white_check_mark: Known issue matched with the provided build. Validation performed at: 4/15/2024 1:07:45 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
714438	dotnet/runtime	Build product	Log	dotnet/runtime#103772
713090	dotnet/runtime	Build product	Log	dotnet/runtime#103709
712446	dotnet/runtime	Build product	Log	dotnet/runtime#103361
712154	dotnet/runtime	Build product	Log	dotnet/runtime#103361
711056	dotnet/runtime	Build product	Log	dotnet/runtime#103361

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	5

[am11]

It's osx-x64 which seem to have hit it earlier with mono: https://github.com/dotnet/runtime/issues/67405.

[MichalStrehovsky]

It's osx-x64 which seem to have hit it earlier with mono: #67405.

We're crashing when we run dotnet.sh (in both bugs) - I don't think the runtime involved is Mono. This is the dotnet that is used to build the repo and that's CoreCLR-based. I think #67405 was misrouted and should likely be just closed because it's stale.

[akoeplinger]

Also reported in https://github.com/dotnet/runtime/issues/69588 in 7.0 timeframe, but seems to be super rare. Looks like all of the hits were on macOS.

[jkotas]

We need to be able to collect crash dumps for builds to make this crash diagnosable.

[MichalStrehovsky]

Also reported in #69588 in 7.0 timeframe, but seems to be super rare. Looks like all of the hits were on macOS.

My concern was that if we hit this in one of two runs, it might not be super rare in this SDK (or it was just a very unlucky run), but I guess we'll see soon enough and can revert if needed.

[am11]

1 occurrence in a month, so it was an unlucky run. 😄 Now we are on preview.4.