carlossanlop commented 3 months ago

The System.IO.Net5Compat.Tests and the System.IO.Tests test processes are intermittengly getting killed on Linux shortly after starting, and the exit code is 137.

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=627407 Build error leg or test failing: System.IO.Net5Compat.Tests

Error Message

{
  "ErrorPattern": ["Starting:    System\\.IO\\.(Net5Compat\\.)?Tests", "exit code 137"],
  "BuildRetry" : true,
  "ExcludeConsoleLog" : false
}

System.IO.Net5Compat.Tests example

PR: https://github.com/dotnet/runtime/pull/100545
Queue: Libraries Test Run release coreclr Linux_musl x64 Debug
Job result: https://dev.azure.com/dnceng-public/public/_build/results?buildId=627407&view=logs&j=73b7db51-6fbc-594d-25c9-15b2737b0bf6
Log file: https://helixre107v0xd1eu3ibi6ka.blob.core.windows.net/dotnet-runtime-refs-pull-100545-merge-f07349d435f3464e8b/System.IO.Net5Compat.Tests/1/console.71f42a07.log?helixlogtype=result

Output:

===========================================================================================================
/root/helix/work/workitem/e /root/helix/work/workitem/e
Discovering: System.IO.Net5Compat.Tests (method display = ClassAndMethod, method display options = None)
Discovered:  System.IO.Net5Compat.Tests (found 679 of 685 test cases)
Starting:    System.IO.Net5Compat.Tests (parallel test collections = on, max threads = 2)
./RunTests.sh: line 162:    25 Killed                  "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.IO.Net5Compat.Tests.runtimeconfig.json --depsfile System.IO.Net5Compat.Tests.deps.json xunit.console.dll System.IO.Net5Compat.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/root/helix/work/workitem/e
----- end Tue Apr 2 20:20:02 UTC 2024 ----- exit code 137 ----------------------------------------------------------

System.IO.Test example

PR: https://github.com/dotnet/runtime/pull/100545
Queue: Libraries Test Run release mono Linux x64 Debug
Job result: https://dev.azure.com/dnceng-public/public/_build/results?buildId=627407&view=logs&j=b4344b0d-0f92-5d69-ccaf-e0b24fbf14a2&t=65a58faa-a542-5344-b5c8-897139f242cd&l=888
Log file: https://helixre107v0xd1eu3ibi6ka.blob.core.windows.net/dotnet-runtime-refs-pull-100545-merge-f07349d435f3464e8b/System.IO.Tests/1/console.effe990e.log?helixlogtype=result

Output:

===========================================================================================================
/root/helix/work/workitem/e /root/helix/work/workitem/e
Discovering: System.IO.Tests (method display = ClassAndMethod, method display options = None)
Discovered:  System.IO.Tests (found 679 of 685 test cases)
Starting:    System.IO.Tests (parallel test collections = on, max threads = 2)
./RunTests.sh: line 162:    25 Killed                  "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.IO.Tests.runtimeconfig.json --depsfile System.IO.Tests.deps.json xunit.console.dll System.IO.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/root/helix/work/workitem/e
----- end Tue Apr 2 20:20:10 UTC 2024 ----- exit code 137 ----------------------------------------------------------

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=627407 Error message validated: [Starting: System\.IO\.(Net5Compat\.)?Tests exit code 137] Result validation: :white_check_mark: Known issue matched with the provided build. Validation performed at: 4/2/2024 11:08:28 PM UTC

Report

Build	Definition	Test	Pull Request
749490	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#105178
749481	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#105207
749467	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#105036
749156	dotnet/runtime	System.IO.Tests.WorkItemExecution
748981	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#105188
748952	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#102144
748937	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#103585
748848	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#105178
748855	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104811
748791	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104856
748741	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104957
748457	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104955
748425	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#105170
748350	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#105151
748313	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104516
748254	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#105036
748234	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#103504
748206	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#105161
748156	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#105119
748000	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104961
747939	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104835
747923	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#105103
747831	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#103528
747800	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#105106
747813	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#105145
747696	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104959
747638	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104957
747537	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#105109
747528	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#105027
747478	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#103585
747441	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#105073
747431	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104961
747371	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#105131
747315	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#105099
747282	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#105109
747270	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104583
747252	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104848
746748	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104503
746673	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104848
746594	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104856
746546	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#105091
743585	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104829
746243	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104957
745973	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104849
745970	dotnet/runtime	System.IO.Tests.WorkItemExecution
745875	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#105063
745782	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#105057
745761	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104943
745223	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104503
745214	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104998
745209	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#105002
745169	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104856
745142	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#103528
745132	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104134
745123	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104993
745129	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#105007
745071	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#105041
745059	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#105036
745006	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104680
744957	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#105004
744934	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#105003
744873	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104731
744798	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104955
744735	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104856
744740	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104972
744704	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104959
744679	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104800
744641	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104957
744454	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#105017
744406	dotnet/runtime	System.IO.Tests.WorkItemExecution
744359	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#105003
744347	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104680
744333	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#105004
744253	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#102498
743013	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104942
744186	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104998
744174	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104704
744105	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#105003
744082	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104835
743412	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104944
743972	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104758
743901	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104961
743890	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104831
743460	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104934
743443	dotnet/runtime	System.IO.Tests.WorkItemExecution
743262	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104552
743409	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104985
743326	dotnet/runtime	System.IO.Net5Compat.Tests.WorkItemExecution	dotnet/runtime#104829
742919	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104958
742723	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104943
742711	dotnet/runtime	System.IO.Tests.WorkItemExecution
742324	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104503
742263	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104935
742229	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104848
742256	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104934
742253	dotnet/runtime	System.IO.Tests.WorkItemExecution
742122	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104759
742060	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104921
742020	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104534
742030	dotnet/runtime	System.IO.Tests.WorkItemExecution	dotnet/runtime#104925

Displaying 100 of 330 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
4	119	330

dotnet-policy-service[bot] commented 3 months ago

Tagging subscribers to this area: @dotnet/area-system-io See info in area-owners.md if you want to be subscribed.

ericstj commented 3 months ago

@dotnet/area-system-io there are a lot of hits on this and relatively recent. It seems to me to be happening across many configurations. I think it's worth having a look.

adamsitnik commented 3 months ago

The System.IO.Net5Compat.Tests and the System.IO.Tests test processes are intermittengly getting killed on Linux shortly after starting, and the exit code is 137.

137 means out of memory. We have not made any changes to 6.0 in System.IO, so I expect that either there was some infra change (like less memory available) or a bug was introduced in the product itself. The bug would be specific to Linux.

@carlossanlop is it possible to perform some kind of binary search based on the merged PRs and when it started to fail?

carlossanlop commented 6 days ago

@adamsitnik @Jozkee This is one of the most impactful failures in servicing. It only affects System.IO.Tests and System.IO.Net5Compat.Tests. Any chance you can take a look soon?

adamsitnik commented 6 days ago

@carlossanlop sure, but could you please answer the question I've asked in https://github.com/dotnet/runtime/issues/100558#issuecomment-2051457144 ?

carlossanlop commented 5 days ago

Sorry, I missed that question. Yes, you can use Kusto. David has used it many times in the past.

carlossanlop commented 5 days ago

This is the super basic kusto query you can execute if looking via issue:

TestKnownIssues
| union KnownIssues
| where IssueId == ""

This database stores data from the last 4 months so hopefully there's still info from April.

This is the cluster where you would look for that info: https://dataexplorer.azure.com/clusters/dotnetperf.westus/databases/PerformanceData

Unfortunately it seems that failure data is not stored if it's not linked to an issue.

Thanks @AlitzelMendez for the above info.

dotnet / runtime

System.IO.Net5Compat.Tests and System.IO.Tests suddenly exiting with error 137 #100558