[mono] MiniJIT System.Formats.Nrbf.Tests getting killed

matouskozak commented 1 month ago

The System.Formats.Nrbf.Tests are getting ocassinally killed on CI. Affecting mostly MonoJIT linux_x64_release_mono_minijit_librariestests job and occasionally MonoInterpreterlinux_x64_release_mono_interpreter_librariestests.

7 failures in the last 30 days as per

Log:

Starting:    System.Formats.Nrbf.Tests (parallel test collections = on [2 threads], stop on fail = off)
./RunTests.sh: line 176:    21 Killed                  "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Formats.Nrbf.Tests.runtimeconfig.json --depsfile System.Formats.Nrbf.Tests.deps.json xunit.console.dll System.Formats.Nrbf.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
cat: /proc/sys/kernel/coredump_filter: No such file or directory
dmesg: read kernel buffer failed: Operation not permitted
+ export _commandExitCode=137
+ python /root/helix/work/correlation/reporter/run.py https://dev.azure.com/dnceng-public/ public 21132138 eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6IjdSd2F5dmlYRHFoZnN6MTZSNmxPbXNXWWxTQSJ9.eyJuYW1laWQiOiJjNzczZjJjMi01MTIwLTQyMDctYWZlMi1hZmFmMzVhOGJjMGEiLCJzY3AiOiJhcHBfdG9rZW4iLCJhdWkiOiJiMWU0ZDg3Ni0yZjIwLTQ1NjctODgyMi05OWQ0YTNhZGYxNGEiLCJzaWQiOiI4ODdhMTkyYy0yNTQ3LTQ4YTUtYTEwMS0yOTc4ZmJiNzI0ZWIiLCJCdWlsZElkIjoiY2JiMTgyNjEtYzQ4Zi00YWJiLTg2NTEtOGNkY2I1NDc0NjQ5OzgxNTI0OSIsIkRlZklkIjoiMTI5Iiwiam9icmVmIjoiNDgyZjk0ZWQtYjFjZi00Mjk4LWJmMmMtMDY3NGIyNTRjNzgxOjFkMTA2YTY1LTI2OWItNTI4Mi1lZjMxLWUzMzc4YTVhMTNmMiIsInBwaWQiOiJ2c3RmczovLy9CdWlsZC9CdWlsZC84MTUyNDkiLCJvcmNoaWQiOiI0ODJmOTRlZC1iMWNmLTQyOTgtYmYyYy0wNjc0YjI1NGM3ODEuYnVpbGQuYnVpbGRfbGludXhfeDY0X3JlbGVhc2VfbW9ub19pbnRlcnByZXRlcl9saWJyYXJpZXN0ZXN0cy5fX2RlZmF1bHQiLCJyZXBvSWRzIjoiIiwiaXNzIjoiYXBwLnZzdG9rZW4udmlzdWFsc3R1ZGlvLmNvbSIsImF1ZCI6ImFwcC52c3Rva2VuLnZpc3VhbHN0dWRpby5jb218dnNvOjZmY2M5MmU1LTczYTctNGY4OC04ZDEzLWQ5MDQ1YjQ1ZmIyNyIsIm5iZiI6MTcyNzAzNDk0MSwiZXhwIjoxNzI3MDY0OTQxfQ.om2rr6irEACnRYKhRuUB2ECACuJKmQKGl3wctZ4IVZo8LtGCigyhva_yyKjvZRX5n3xl6SQgNlv01Cl2uKRaVGws1zizT9WsUwKKkLm390V5dSHg3eb1VyDHMM2LTO9mM5Ve4dfn4XhgeUid4xkpH-V6vXrIMu6E3hW-B30oH7xH1kdj8Ftw1sYuL5rsNJRAXjnd3hZgODSSXgK3_dij7BqeH-AZZBqv3ErGWdUYuVd0BRv4M8P51EJPjp8u4CC1jP_rjKRMw8nZm4VvCEL7CyR4X4SbVNqp5KP2F8Kpjnpus_0rMPNyLnyE2KOI1totQRV_WmguZZq7w9V8KMb5FQ
/root/helix/work/workitem/e
----- end Sun 22 Sep 2024 08:43:19 PM UTC ----- exit code 137 ----------------------------------------------------------
exit code 137 means SIGKILL Killed either due to out of memory/resources (see /var/log/messages) or by explicit kill.

Known Issue Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": ["System.Formats.Nrbf.Tests", "exit code 137 means SIGKILL Killed either due to out of memory/resources (see /var/log/messages) or by explicit kill"],
  "BuildRetry": true,
  "ExcludeConsoleLog": false
}

Report

Build	Definition	Test
878000	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution
872567	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution
869114	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution
861628	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution
857281	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution
856935	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution
856325	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution
854889	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
1	2	8

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=825800 Result validation: :warning: Validation could not be done without an Azure DevOps build URL on the issue. Please add it to the "Build: :mag_right:" line. Validation performed at: 10/3/2024 4:08:45 PM UTC

dotnet-policy-service[bot] commented 1 month ago

Tagging subscribers to this area: @lambdageek, @steveisok See info in area-owners.md if you want to be subscribed.

lambdageek commented 1 month ago

There are some interesting "attack tests" that do things with cyclic graphs and potentially large memory usage that might be behaving pathologically on Mono. (for example due to conservative stack scanning)

https://github.com/dotnet/runtime/blob/main/src/libraries/System.Formats.Nrbf/tests/AttackTests.cs

dotnet / runtime