search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
14.91k stars 4.63k forks source link

Test failure: AggregateException: Not all processes in process tree could be terminated. #93321

Open v-wenyuxu opened 11 months ago

v-wenyuxu commented 11 months ago

Failed in: runtime-coreclr jitstress 20231009.2

Failed tests:

coreclr osx x64 Checked jitstress2 @ OSX.1200.Amd64.Open
    - baseservices/exceptions/simple/ParallelCrashMainThread/ParallelCrashMainThread.sh
    - baseservices/exceptions/simple/ParallelCrash/ParallelCrash.sh
    - baseservices/exceptions/unhandled/unhandled/unhandled.sh

Error message:

Test Infrastructure Failure: System.AggregateException: Not all processes in process tree could be terminated. (Operation not permitted)
 ---> System.ComponentModel.Win32Exception (1): Operation not permitted
   --- End of inner exception stack trace ---
   at System.Diagnostics.Process.Kill(Boolean entireProcessTree)
   at CoreclrTestLib.CoreclrTestWrapperLib.CollectCrashDumpWithCreateDump(Process process, String crashDumpPath, StreamWriter outputWriter)
   at CoreclrTestLib.CoreclrTestWrapperLib.CollectCrashDump(Process process, String crashDumpPath, StreamWriter outputWriter)
   at CoreclrTestLib.CoreclrTestWrapperLib.RunTest(String executable, String outputFile, String errorFile, String category, String testBinaryBase, String outputDir)
   at baseservices_exceptions._simple_ParallelCraMainThread_ParallelCraMainThread_._simple_ParallelCrashMainThread_ParallelCrashMainThread_sh()
Expected: True
Actual:   False

Stack trace:

   at baseservices_exceptions._simple_ParallelCraMainThread_ParallelCraMainThread_._simple_ParallelCrashMainThread_ParallelCrashMainThread_sh()
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)


Queued | Pipeline | Pipeline_Configuration | OS | Arch | Test | Outcome | Properties | BuildReason -- | -- | -- | -- | -- | -- | -- | -- | -- 2023-10-10T05:50:17.995Z | runtime-coreclr jitstress | Checked-jitstress2 | osx.1200.amd64.open | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"BuildNumber":"20231009.2","DefinitionId":"109","Project":"public","System.PhaseName":"run_test_p1__osx_x64_checked","operatingSystem":"OSX.1200.Amd64.Open","System.JobName":"__default","DefinitionName":"runtime-coreclr jitstress","System.StageName":"Build","BuildId":"432736","architecture":"x64","configuration":"Checked-jitstress2","System.PhaseAttempt":"1","AzurePipelinesTestRunId":"9585790","System.JobAttempt":"1","CollectionUri":"https://dev.azure.com/dnceng-public/","System.JobId":"bf644a0b-2dbe-5a03-d8c6-a86fdd97c823","System.StageAttempt":"1","Reason":"Schedule"} | Schedule 2023-10-10T05:50:17.995Z | runtime-coreclr jitstress | Checked-jitstress2 | osx.1200.amd64.open | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"BuildNumber":"20231009.2","DefinitionId":"109","Project":"public","System.PhaseName":"run_test_p1__osx_x64_checked","operatingSystem":"OSX.1200.Amd64.Open","System.JobName":"__default","DefinitionName":"runtime-coreclr jitstress","System.StageName":"Build","BuildId":"432736","architecture":"x64","configuration":"Checked-jitstress2","System.PhaseAttempt":"1","AzurePipelinesTestRunId":"9585790","System.JobAttempt":"1","CollectionUri":"https://dev.azure.com/dnceng-public/","System.JobId":"bf644a0b-2dbe-5a03-d8c6-a86fdd97c823","System.StageAttempt":"1","Reason":"Schedule"} | Schedule 2023-10-10T05:50:17.995Z | runtime-coreclr jitstress | Checked-jitstress2 | osx.1200.amd64.open | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"BuildNumber":"20231009.2","DefinitionId":"109","Project":"public","System.PhaseName":"run_test_p1__osx_x64_checked","operatingSystem":"OSX.1200.Amd64.Open","System.JobName":"__default","DefinitionName":"runtime-coreclr jitstress","System.StageName":"Build","BuildId":"432736","architecture":"x64","configuration":"Checked-jitstress2","System.PhaseAttempt":"1","AzurePipelinesTestRunId":"9585790","System.JobAttempt":"1","CollectionUri":"https://dev.azure.com/dnceng-public/","System.JobId":"bf644a0b-2dbe-5a03-d8c6-a86fdd97c823","System.StageAttempt":"1","Reason":"Schedule"} | Schedule 2023-10-10T05:50:17.995Z | runtime-coreclr jitstress | Checked-jitstress2 | osx.1200.amd64.open | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"BuildNumber":"20231009.2","DefinitionId":"109","Project":"public","System.PhaseName":"run_test_p1__osx_x64_checked","operatingSystem":"OSX.1200.Amd64.Open","System.JobName":"__default","DefinitionName":"runtime-coreclr jitstress","System.StageName":"Build","BuildId":"432736","architecture":"x64","configuration":"Checked-jitstress2","System.PhaseAttempt":"1","AzurePipelinesTestRunId":"9585790","System.JobAttempt":"1","CollectionUri":"https://dev.azure.com/dnceng-public/","System.JobId":"bf644a0b-2dbe-5a03-d8c6-a86fdd97c823","System.StageAttempt":"1","Reason":"Schedule"} | Schedule 2023-10-08T14:21:17.734Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} | Schedule
More failures Queued | Pipeline | Pipeline_Configuration | OS | Arch | Test | Outcome | Properties | BuildReason -- | -- | -- | -- | -- | -- | -- | -- | -- 2023-10-08T14:21:17.734Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} | Schedule 2023-10-08T14:21:17.734Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} | Schedule 2023-10-08T14:21:17.734Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} | Schedule 2023-10-08T14:21:17.734Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} | Schedule 2023-10-08T14:21:17.734Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} | Schedule 2023-10-08T14:21:17.734Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} | Schedule 2023-10-08T14:21:17.734Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} | Schedule 2023-10-08T14:21:17.734Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} | Schedule 2023-10-08T14:21:17.734Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} | Schedule 2023-10-08T14:21:16.532Z | runtime-coreclr runincontext | Checked | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} | Schedule 2023-10-08T14:21:16.532Z | runtime-coreclr runincontext | Checked | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} | Schedule 2023-10-08T14:21:16.532Z | runtime-coreclr runincontext | Checked | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} | Schedule 2023-10-08T14:21:16.532Z | runtime-coreclr runincontext | Checked | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} | Schedule 2023-10-08T14:21:16.532Z | runtime-coreclr runincontext | Checked | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} | Schedule 2023-10-08T14:21:16.532Z | runtime-coreclr runincontext | Checked | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} | Schedule 2023-10-08T14:21:16.532Z | runtime-coreclr runincontext | Checked | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} | Schedule 2023-10-08T14:21:16.532Z | runtime-coreclr runincontext | Checked | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} | Schedule 2023-10-08T14:21:16.532Z | runtime-coreclr runincontext | Checked | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} | Schedule 2023-10-08T14:21:16.532Z | runtime-coreclr runincontext | Checked | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} | Schedule 2023-10-07T14:20:57.576Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} | Schedule 2023-10-07T14:20:57.576Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} | Schedule 2023-10-07T14:20:57.576Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} | Schedule 2023-10-07T14:20:57.576Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} | Schedule 2023-10-07T14:20:57.576Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} | Schedule 2023-10-07T14:20:57.576Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} | Schedule 2023-10-07T14:20:57.576Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} | Schedule 2023-10-07T14:20:57.576Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} | Schedule 2023-10-07T14:20:57.576Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} | Schedule 2023-10-07T14:20:57.576Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} | Schedule
### Known Issue Error Message Fill the error message using [step by step known issues guidance](https://github.com/dotnet/arcade/blob/main/Documentation/Projects/Build%20Analysis/KnownIssues.md#how-to-fill-out-a-known-issue-error-section). ```json { "ErrorMessage": [ "Not all processes in process tree could be terminated.", "Operation not permitted" ], "ErrorPattern": "", "BuildRetry": false, "ExcludeConsoleLog": false } ``` ### Known issue validation **Build: :mag_right:** **Result validation:** :warning: Provided build not found. Provide a valid build in the "**Build:** :mag_right:" line. **Validation performed at:** 4/2/2024 6:55:27 AM UTC ### Report #### Summary |24-Hour Hit Count|7-Day Hit Count|1-Month Count| |---|---|---| |0|0|0|
ghost commented 11 months ago

Tagging subscribers to this area: @dotnet/area-system-diagnostics-process See info in area-owners.md if you want to be subscribed.

Issue Details
**Failed in:** [runtime-coreclr jitstress 20231009.2](https://dev.azure.com/dnceng-public/public/_build/results?buildId=432736&view=ms.vss-test-web.build-test-results-tab&runId=9585790&resultId=118037&paneView=debug) **Failed tests:** ``` coreclr osx x64 Checked jitstress2 @ OSX.1200.Amd64.Open - baseservices/exceptions/simple/ParallelCrashMainThread/ParallelCrashMainThread.sh - baseservices/exceptions/simple/ParallelCrash/ParallelCrash.sh - baseservices/exceptions/unhandled/unhandled/unhandled.sh ``` **Error message:** ``` Test Infrastructure Failure: System.AggregateException: Not all processes in process tree could be terminated. (Operation not permitted) ---> System.ComponentModel.Win32Exception (1): Operation not permitted --- End of inner exception stack trace --- at System.Diagnostics.Process.Kill(Boolean entireProcessTree) at CoreclrTestLib.CoreclrTestWrapperLib.CollectCrashDumpWithCreateDump(Process process, String crashDumpPath, StreamWriter outputWriter) at CoreclrTestLib.CoreclrTestWrapperLib.CollectCrashDump(Process process, String crashDumpPath, StreamWriter outputWriter) at CoreclrTestLib.CoreclrTestWrapperLib.RunTest(String executable, String outputFile, String errorFile, String category, String testBinaryBase, String outputDir) at baseservices_exceptions._simple_ParallelCraMainThread_ParallelCraMainThread_._simple_ParallelCrashMainThread_ParallelCrashMainThread_sh() Expected: True Actual: False ``` **Stack trace:** ``` at baseservices_exceptions._simple_ParallelCraMainThread_ParallelCraMainThread_._simple_ParallelCrashMainThread_ParallelCrashMainThread_sh() at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor) at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr) ```


Queued | Pipeline | Pipeline_Configuration | OS | Arch | Test | Outcome | Properties | BuildReason -- | -- | -- | -- | -- | -- | -- | -- | -- 2023-10-10T05:50:17.995Z | runtime-coreclr jitstress | Checked-jitstress2 | osx.1200.amd64.open | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"BuildNumber":"20231009.2","DefinitionId":"109","Project":"public","System.PhaseName":"run_test_p1__osx_x64_checked","operatingSystem":"OSX.1200.Amd64.Open","System.JobName":"__default","DefinitionName":"runtime-coreclr jitstress","System.StageName":"Build","BuildId":"432736","architecture":"x64","configuration":"Checked-jitstress2","System.PhaseAttempt":"1","AzurePipelinesTestRunId":"9585790","System.JobAttempt":"1","CollectionUri":"https://dev.azure.com/dnceng-public/","System.JobId":"bf644a0b-2dbe-5a03-d8c6-a86fdd97c823","System.StageAttempt":"1","Reason":"Schedule"} | Schedule 2023-10-10T05:50:17.995Z | runtime-coreclr jitstress | Checked-jitstress2 | osx.1200.amd64.open | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"BuildNumber":"20231009.2","DefinitionId":"109","Project":"public","System.PhaseName":"run_test_p1__osx_x64_checked","operatingSystem":"OSX.1200.Amd64.Open","System.JobName":"__default","DefinitionName":"runtime-coreclr jitstress","System.StageName":"Build","BuildId":"432736","architecture":"x64","configuration":"Checked-jitstress2","System.PhaseAttempt":"1","AzurePipelinesTestRunId":"9585790","System.JobAttempt":"1","CollectionUri":"https://dev.azure.com/dnceng-public/","System.JobId":"bf644a0b-2dbe-5a03-d8c6-a86fdd97c823","System.StageAttempt":"1","Reason":"Schedule"} | Schedule 2023-10-10T05:50:17.995Z | runtime-coreclr jitstress | Checked-jitstress2 | osx.1200.amd64.open | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"BuildNumber":"20231009.2","DefinitionId":"109","Project":"public","System.PhaseName":"run_test_p1__osx_x64_checked","operatingSystem":"OSX.1200.Amd64.Open","System.JobName":"__default","DefinitionName":"runtime-coreclr jitstress","System.StageName":"Build","BuildId":"432736","architecture":"x64","configuration":"Checked-jitstress2","System.PhaseAttempt":"1","AzurePipelinesTestRunId":"9585790","System.JobAttempt":"1","CollectionUri":"https://dev.azure.com/dnceng-public/","System.JobId":"bf644a0b-2dbe-5a03-d8c6-a86fdd97c823","System.StageAttempt":"1","Reason":"Schedule"} | Schedule 2023-10-10T05:50:17.995Z | runtime-coreclr jitstress | Checked-jitstress2 | osx.1200.amd64.open | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"BuildNumber":"20231009.2","DefinitionId":"109","Project":"public","System.PhaseName":"run_test_p1__osx_x64_checked","operatingSystem":"OSX.1200.Amd64.Open","System.JobName":"__default","DefinitionName":"runtime-coreclr jitstress","System.StageName":"Build","BuildId":"432736","architecture":"x64","configuration":"Checked-jitstress2","System.PhaseAttempt":"1","AzurePipelinesTestRunId":"9585790","System.JobAttempt":"1","CollectionUri":"https://dev.azure.com/dnceng-public/","System.JobId":"bf644a0b-2dbe-5a03-d8c6-a86fdd97c823","System.StageAttempt":"1","Reason":"Schedule"} | Schedule 2023-10-08T14:21:17.734Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} | Schedule
More failures Queued | Pipeline | Pipeline_Configuration | OS | Arch | Test | Outcome | Properties | BuildReason -- | -- | -- | -- | -- | -- | -- | -- | -- 2023-10-08T14:21:17.734Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} | Schedule 2023-10-08T14:21:17.734Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} | Schedule 2023-10-08T14:21:17.734Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} | Schedule 2023-10-08T14:21:17.734Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} | Schedule 2023-10-08T14:21:17.734Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} | Schedule 2023-10-08T14:21:17.734Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} | Schedule 2023-10-08T14:21:17.734Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} | Schedule 2023-10-08T14:21:17.734Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} | Schedule 2023-10-08T14:21:17.734Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionName":"runtime-coreclr runincontext","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","configuration":"Checked-no_tiered_compilation","AzurePipelinesTestRunId":"9537288","BuildNumber":"20231008.1","System.JobAttempt":"1","System.JobName":"__default","architecture":"x64","BuildId":"431261","System.StageName":"Build","DefinitionId":"121","Project":"public","Reason":"Schedule"} | Schedule 2023-10-08T14:21:16.532Z | runtime-coreclr runincontext | Checked | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} | Schedule 2023-10-08T14:21:16.532Z | runtime-coreclr runincontext | Checked | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} | Schedule 2023-10-08T14:21:16.532Z | runtime-coreclr runincontext | Checked | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} | Schedule 2023-10-08T14:21:16.532Z | runtime-coreclr runincontext | Checked | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} | Schedule 2023-10-08T14:21:16.532Z | runtime-coreclr runincontext | Checked | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} | Schedule 2023-10-08T14:21:16.532Z | runtime-coreclr runincontext | Checked | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} | Schedule 2023-10-08T14:21:16.532Z | runtime-coreclr runincontext | Checked | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} | Schedule 2023-10-08T14:21:16.532Z | runtime-coreclr runincontext | Checked | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} | Schedule 2023-10-08T14:21:16.532Z | runtime-coreclr runincontext | Checked | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} | Schedule 2023-10-08T14:21:16.532Z | runtime-coreclr runincontext | Checked | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","System.JobAttempt":"1","Reason":"Schedule","configuration":"Checked","DefinitionName":"runtime-coreclr runincontext","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","BuildId":"431261","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","AzurePipelinesTestRunId":"9537286","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","Project":"public","System.JobName":"__default","BuildNumber":"20231008.1","DefinitionId":"121","System.StageAttempt":"1"} | Schedule 2023-10-07T14:20:57.576Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} | Schedule 2023-10-07T14:20:57.576Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} | Schedule 2023-10-07T14:20:57.576Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} | Schedule 2023-10-07T14:20:57.576Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} | Schedule 2023-10-07T14:20:57.576Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} | Schedule 2023-10-07T14:20:57.576Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} | Schedule 2023-10-07T14:20:57.576Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} | Schedule 2023-10-07T14:20:57.576Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} | Schedule 2023-10-07T14:20:57.576Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} | Schedule 2023-10-07T14:20:57.576Z | runtime-coreclr runincontext | Checked-no_tiered_compilation | ubuntu.2204.amd64.open.rt | x64 | baseservices/exceptions/unhandled/unhandled/unhandled.sh | Failed | {"AzurePipelinesTestRunId":"9526074","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"430819","System.PhaseName":"run_test_p1_RunInContext_linux_x64_checked","DefinitionId":"121","System.StageName":"Build","System.JobName":"__default","BuildNumber":"20231007.1","Project":"public","architecture":"x64","System.StageAttempt":"1","System.JobAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-no_tiered_compilation","System.JobId":"83aa44ae-94d8-5d3d-7684-a0cf5cb0d108","System.PhaseAttempt":"1","Reason":"Schedule","DefinitionName":"runtime-coreclr runincontext"} | Schedule
Author: v-wenyuxu
Assignees: -
Labels: `area-System.Diagnostics.Process`, `os-mac-os-x`, `JitStress`, `arch-x64`, `untriaged`, `needs-area-label`
Milestone: -
v-wenyuxu commented 6 months ago

Failed in: runtime-coreclr r2r-extra 20240224.1

Failed tests:

R2R-CG2 linux arm Checked jitstressregs4 @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd

Error message:

 Test Infrastructure Failure: System.AggregateException: Not all processes in process tree could be terminated. (Operation not permitted)
 ---> System.ComponentModel.Win32Exception (1): Operation not permitted
   --- End of inner exception stack trace ---
   at System.Diagnostics.Process.Kill(Boolean entireProcessTree) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.NonUap.cs:line 31
   at CoreclrTestLib.CoreclrTestWrapperLib.CollectCrashDumpWithCreateDump(Process process, String crashDumpPath, StreamWriter outputWriter)
   at CoreclrTestLib.CoreclrTestWrapperLib.RunTest(String executable, String outputFile, String errorFile, String category, String testBinaryBase, String outputDir)
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)

Stack trace:

   at Xunit.Assert.Fail(String message) in /_/src/Microsoft.DotNet.XUnitAssert/src/FailAsserts.cs:line 34
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)
   at Program.<<Main>$>g__TestExecutor251|25_252(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass25_0&)
v-wenyuxu commented 5 months ago

Failed in: runtime-coreclr r2r-extra 20240316.1

Failed tests:

R2R-CG2 linux arm Checked jitstress1 @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd
R2R-CG2 linux arm Checked gcstress0xc @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd

Error message:

 Test Infrastructure Failure: System.AggregateException: Not all processes in process tree could be terminated. (Operation not permitted)
 ---> System.ComponentModel.Win32Exception (1): Operation not permitted
   --- End of inner exception stack trace ---
   at System.Diagnostics.Process.Kill(Boolean entireProcessTree) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.NonUap.cs:line 31
   at CoreclrTestLib.CoreclrTestWrapperLib.CollectCrashDumpWithCreateDump(Process process, String crashDumpPath, StreamWriter outputWriter)
   at CoreclrTestLib.CoreclrTestWrapperLib.RunTest(String executable, String outputFile, String errorFile, String category, String testBinaryBase, String outputDir)
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)

Stack trace:

   at Xunit.Assert.Fail(String message) in /_/src/Microsoft.DotNet.XUnitAssert/src/FailAsserts.cs:line 34
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)
   at Program.<<Main>$>g__TestExecutor251|25_252(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass25_0&)
mikelle-rogers commented 5 months ago

@mangod9, what is the status of this failure?

mangod9 commented 5 months ago

seems to be only affecting jitstress. Did you notice this failure on a PR run?

mikelle-rogers commented 5 months ago

One of the vendors noticed it here, as seen in the above comment.

v-wenyuxu commented 5 months ago

Failed in: runtime-coreclr r2r-extra 20240323.1

Failed tests:

R2R-CG2 linux arm Checked jitstress2_tiered @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd
R2R-CG2 linux arm Checked gcstress0xc @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd

Error message:

 Test Infrastructure Failure: System.AggregateException: Not all processes in process tree could be terminated. (Operation not permitted)
 ---> System.ComponentModel.Win32Exception (1): Operation not permitted
   --- End of inner exception stack trace ---
   at System.Diagnostics.Process.Kill(Boolean entireProcessTree) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.NonUap.cs:line 31
   at CoreclrTestLib.CoreclrTestWrapperLib.CollectCrashDumpWithCreateDump(Process process, String crashDumpPath, StreamWriter outputWriter)
   at CoreclrTestLib.CoreclrTestWrapperLib.RunTest(String executable, String outputFile, String errorFile, String category, String testBinaryBase, String outputDir)
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)

Stack trace:

   at Xunit.Assert.Fail(String message) in /_/src/Microsoft.DotNet.XUnitAssert/src/FailAsserts.cs:line 34
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)
   at Program.<<Main>$>g__TestExecutor251|25_252(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass25_0&)
jkotas commented 5 months ago

This is hit intermittently on both macOS and Linux, different architectures, for different tests. I have just resolved #99885 as a duplicate.

The problem seems to be a race condition in the implementation of System.Diagnostic.Process.Kill(entireProcessTree: true).

The processes in the tree can be starting and exiting while Kill(entireProcessTree: true) is trying to kill them all. There seems to be some specific pattern where the process is found, but it exits on its own as it is getting killed, that results into Operation not permitted error and exception.

v-wenyuxu commented 5 months ago

Failed in: runtime-coreclr r2r 20240401.1

Failed tests:

R2R-CG2 linux arm Checked no_tiered_compilation @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd

Error message:

 Test Infrastructure Failure: System.AggregateException: Not all processes in process tree could be terminated. (Operation not permitted)
 ---> System.ComponentModel.Win32Exception (1): Operation not permitted
   --- End of inner exception stack trace ---
   at System.Diagnostics.Process.Kill(Boolean entireProcessTree) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.NonUap.cs:line 31
   at CoreclrTestLib.CoreclrTestWrapperLib.CollectCrashDumpWithCreateDump(Process process, String crashDumpPath, StreamWriter outputWriter)
   at CoreclrTestLib.CoreclrTestWrapperLib.RunTest(String executable, String outputFile, String errorFile, String category, String testBinaryBase, String outputDir)
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)

Stack trace:

    at Xunit.Assert.Fail(String message) in /_/src/Microsoft.DotNet.XUnitAssert/src/FailAsserts.cs:line 34
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)
   at Program.<<Main>$>g__TestExecutor251|25_252(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass25_0&)
v-wenyuxu commented 5 months ago

Failed in: runtime-coreclr r2r-extra 20240407.1

Failed tests:

R2R-CG2 linux arm Checked jitstressregs0x80 @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd
R2R-CG2 linux arm Checked jitstressregs8 @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd
R2R-CG2 linux arm Checked jitstressregs0x2000 @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd
R2R-CG2 linux arm Checked jitstress2 @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd
R2R-CG2 linux arm Checked forcerelocs @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd
R2R-CG2 linux arm Checked jitstressregs3 @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd
R2R-CG2 linux arm Checked jitminopts @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd
R2R-CG2 linux arm Checked jitstress2_tiered @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd
R2R-CG2 linux arm Checked gcstress0xc @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd

Error message:

 Test Infrastructure Failure: System.AggregateException: Not all processes in process tree could be terminated. (Operation not permitted)
 ---> System.ComponentModel.Win32Exception (1): Operation not permitted
   --- End of inner exception stack trace ---
   at System.Diagnostics.Process.Kill(Boolean entireProcessTree) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.NonUap.cs:line 31
   at CoreclrTestLib.CoreclrTestWrapperLib.CollectCrashDumpWithCreateDump(Process process, String crashDumpPath, StreamWriter outputWriter)
   at CoreclrTestLib.CoreclrTestWrapperLib.RunTest(String executable, String outputFile, String errorFile, String category, String testBinaryBase, String outputDir)
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)

Stack trace:

   at Xunit.Assert.Fail(String message) in /_/src/Microsoft.DotNet.XUnitAssert/src/FailAsserts.cs:line 34
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)
   at Program.<<Main>$>g__TestExecutor253|25_254(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass25_0&)
Queued Pipeline Pipeline_Configuration OS Arch Test Outcome Properties BuildReason
2024-03-31T07:33:02.486Z coreclr-release-outerloop-nightly Release-no_tiered_compilation ubuntu.2004.armarch.open arm Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd Failed {"CollectionUri":"https://dev.azure.com/dnceng-public/","System.StageAttempt":"1","AzurePipelinesTestRunId":"15321360","System.JobId":"c8f31202-1062-5d6a-be4d-45312a30ae25","System.StageName":"Build","configuration":"Release-no_tiered_compilation","System.PhaseAttempt":"1","operatingSystem":"(Alpine.316.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.17-helix-arm32v7","architecture":"arm","BuildNumber":"20240330.1","DefinitionName":"coreclr-release-outerloop-nightly","Project":"public","System.JobName":"__default","Reason":"Schedule","System.JobAttempt":"1","System.PhaseName":"run_test_p1_R2R_linux_musl_arm_release","DefinitionId":"135","BuildId":"625141"} Schedule
2024-03-29T07:32:29.466Z coreclr-release-outerloop-nightly Release-no_tiered_compilation ubuntu.2004.armarch.open arm Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd Failed {"Project":"public","Reason":"Schedule","BuildId":"623537","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseName":"run_test_p1_R2R_linux_musl_arm_release","architecture":"arm","AzurePipelinesTestRunId":"15273954","System.JobName":"__default","DefinitionId":"135","System.PhaseAttempt":"1","System.StageAttempt":"1","BuildNumber":"20240328.1","System.JobAttempt":"1","System.JobId":"c8f31202-1062-5d6a-be4d-45312a30ae25","operatingSystem":"(Alpine.316.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.17-helix-arm32v7","configuration":"Release-no_tiered_compilation","System.StageName":"Build","DefinitionName":"coreclr-release-outerloop-nightly"} Schedule
v-wenyuxu commented 4 months ago

Failed in: runtime-coreclr r2r-extra 20240413.1

Failed tests:

R2R-CG2 linux arm Checked jitstressregs0x1000 @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd
R2R-CG2 linux arm Checked jitstressregs3 @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd
R2R-CG2 linux arm Checked jitstress1_tiered @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd
R2R-CG2 linux arm Checked jitstressregs0x10 @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd
R2R-CG2 linux arm Checked jitstressregs8 @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd
R2R-CG2 linux arm Checked gcstress0xc @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd
R2R-CG2 linux arm Checked jitstress2 @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd

Error message:

 Test Infrastructure Failure: System.AggregateException: Not all processes in process tree could be terminated. (Operation not permitted)
 ---> System.ComponentModel.Win32Exception (1): Operation not permitted
   --- End of inner exception stack trace ---
   at System.Diagnostics.Process.Kill(Boolean entireProcessTree) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.NonUap.cs:line 31
   at CoreclrTestLib.CoreclrTestWrapperLib.CollectCrashDumpWithCreateDump(Process process, String crashDumpPath, StreamWriter outputWriter)
   at CoreclrTestLib.CoreclrTestWrapperLib.RunTest(String executable, String outputFile, String errorFile, String category, String testBinaryBase, String outputDir)
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)

Stack trace:

   at Xunit.Assert.Fail(String message) in /_/src/Microsoft.DotNet.XUnitAssert/src/FailAsserts.cs:line 34
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)
   at Program.<<Main>$>g__TestExecutor253|25_254(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass25_0&)
Queued Pipeline Pipeline_Configuration OS Arch Test Outcome Properties BuildReason
2024-03-31T07:33:02.486Z coreclr-release-outerloop-nightly Release-no_tiered_compilation ubuntu.2004.armarch.open arm Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd Failed {"CollectionUri":"https://dev.azure.com/dnceng-public/","System.StageAttempt":"1","AzurePipelinesTestRunId":"15321360","System.JobId":"c8f31202-1062-5d6a-be4d-45312a30ae25","System.StageName":"Build","configuration":"Release-no_tiered_compilation","System.PhaseAttempt":"1","operatingSystem":"(Alpine.316.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.17-helix-arm32v7","architecture":"arm","BuildNumber":"20240330.1","DefinitionName":"coreclr-release-outerloop-nightly","Project":"public","System.JobName":"__default","Reason":"Schedule","System.JobAttempt":"1","System.PhaseName":"run_test_p1_R2R_linux_musl_arm_release","DefinitionId":"135","BuildId":"625141"} Schedule
2024-03-29T07:32:29.466Z coreclr-release-outerloop-nightly Release-no_tiered_compilation ubuntu.2004.armarch.open arm Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd Failed {"Project":"public","Reason":"Schedule","BuildId":"623537","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseName":"run_test_p1_R2R_linux_musl_arm_release","architecture":"arm","AzurePipelinesTestRunId":"15273954","System.JobName":"__default","DefinitionId":"135","System.PhaseAttempt":"1","System.StageAttempt":"1","BuildNumber":"20240328.1","System.JobAttempt":"1","System.JobId":"c8f31202-1062-5d6a-be4d-45312a30ae25","operatingSystem":"(Alpine.316.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.17-helix-arm32v7","configuration":"Release-no_tiered_compilation","System.StageName":"Build","DefinitionName":"coreclr-release-outerloop-nightly"} Schedule
mikelle-rogers commented 4 months ago

cc @dotnet/area-system-diagnostics-process

v-wenyuxu commented 4 months ago

Failed in: runtime-coreclr r2r 20240415.2

Failed tests:

R2R-CG2 linux arm Checked no_tiered_compilation @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd

Error message:

 Test Infrastructure Failure: System.AggregateException: Not all processes in process tree could be terminated. (Operation not permitted)
 ---> System.ComponentModel.Win32Exception (1): Operation not permitted
   --- End of inner exception stack trace ---
   at System.Diagnostics.Process.Kill(Boolean entireProcessTree) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.NonUap.cs:line 31
   at CoreclrTestLib.CoreclrTestWrapperLib.CollectCrashDumpWithCreateDump(Process process, String crashDumpPath, StreamWriter outputWriter)
   at CoreclrTestLib.CoreclrTestWrapperLib.RunTest(String executable, String outputFile, String errorFile, String category, String testBinaryBase, String outputDir)
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)

Stack trace:

   at Xunit.Assert.Fail(String message) in /_/src/Microsoft.DotNet.XUnitAssert/src/FailAsserts.cs:line 34
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)
   at Program.<<Main>$>g__TestExecutor253|25_254(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass25_0&)
Queued Pipeline Pipeline_Configuration OS Arch Test Outcome Properties BuildReason
2024-03-31T07:33:02.486Z coreclr-release-outerloop-nightly Release-no_tiered_compilation ubuntu.2004.armarch.open arm Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd Failed {"CollectionUri":"https://dev.azure.com/dnceng-public/","System.StageAttempt":"1","AzurePipelinesTestRunId":"15321360","System.JobId":"c8f31202-1062-5d6a-be4d-45312a30ae25","System.StageName":"Build","configuration":"Release-no_tiered_compilation","System.PhaseAttempt":"1","operatingSystem":"(Alpine.316.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.17-helix-arm32v7","architecture":"arm","BuildNumber":"20240330.1","DefinitionName":"coreclr-release-outerloop-nightly","Project":"public","System.JobName":"__default","Reason":"Schedule","System.JobAttempt":"1","System.PhaseName":"run_test_p1_R2R_linux_musl_arm_release","DefinitionId":"135","BuildId":"625141"} Schedule
2024-03-29T07:32:29.466Z coreclr-release-outerloop-nightly Release-no_tiered_compilation ubuntu.2004.armarch.open arm Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd Failed {"Project":"public","Reason":"Schedule","BuildId":"623537","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseName":"run_test_p1_R2R_linux_musl_arm_release","architecture":"arm","AzurePipelinesTestRunId":"15273954","System.JobName":"__default","DefinitionId":"135","System.PhaseAttempt":"1","System.StageAttempt":"1","BuildNumber":"20240328.1","System.JobAttempt":"1","System.JobId":"c8f31202-1062-5d6a-be4d-45312a30ae25","operatingSystem":"(Alpine.316.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.17-helix-arm32v7","configuration":"Release-no_tiered_compilation","System.StageName":"Build","DefinitionName":"coreclr-release-outerloop-nightly"} Schedule
v-wenyuxu commented 4 months ago

Failed in: runtime-coreclr r2r 20240416.1

Failed tests:

R2R-CG2 linux arm Checked no_tiered_compilation @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd

Error message:

 Test Infrastructure Failure: System.AggregateException: Not all processes in process tree could be terminated. (Operation not permitted)
 ---> System.ComponentModel.Win32Exception (1): Operation not permitted
   --- End of inner exception stack trace ---
   at System.Diagnostics.Process.Kill(Boolean entireProcessTree) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.NonUap.cs:line 31
   at CoreclrTestLib.CoreclrTestWrapperLib.CollectCrashDumpWithCreateDump(Process process, String crashDumpPath, StreamWriter outputWriter)
   at CoreclrTestLib.CoreclrTestWrapperLib.RunTest(String executable, String outputFile, String errorFile, String category, String testBinaryBase, String outputDir)
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)

Stack trace:

   at Xunit.Assert.Fail(String message) in /_/src/Microsoft.DotNet.XUnitAssert/src/FailAsserts.cs:line 34
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)
   at Program.<<Main>$>g__TestExecutor253|25_254(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass25_0&)
Queued Pipeline Pipeline_Configuration OS Arch Test Outcome Properties BuildReason
2024-03-31T07:33:02.486Z coreclr-release-outerloop-nightly Release-no_tiered_compilation ubuntu.2004.armarch.open arm Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd Failed {"CollectionUri":"https://dev.azure.com/dnceng-public/","System.StageAttempt":"1","AzurePipelinesTestRunId":"15321360","System.JobId":"c8f31202-1062-5d6a-be4d-45312a30ae25","System.StageName":"Build","configuration":"Release-no_tiered_compilation","System.PhaseAttempt":"1","operatingSystem":"(Alpine.316.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.17-helix-arm32v7","architecture":"arm","BuildNumber":"20240330.1","DefinitionName":"coreclr-release-outerloop-nightly","Project":"public","System.JobName":"__default","Reason":"Schedule","System.JobAttempt":"1","System.PhaseName":"run_test_p1_R2R_linux_musl_arm_release","DefinitionId":"135","BuildId":"625141"} Schedule
2024-03-29T07:32:29.466Z coreclr-release-outerloop-nightly Release-no_tiered_compilation ubuntu.2004.armarch.open arm Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd Failed {"Project":"public","Reason":"Schedule","BuildId":"623537","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseName":"run_test_p1_R2R_linux_musl_arm_release","architecture":"arm","AzurePipelinesTestRunId":"15273954","System.JobName":"__default","DefinitionId":"135","System.PhaseAttempt":"1","System.StageAttempt":"1","BuildNumber":"20240328.1","System.JobAttempt":"1","System.JobId":"c8f31202-1062-5d6a-be4d-45312a30ae25","operatingSystem":"(Alpine.316.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.17-helix-arm32v7","configuration":"Release-no_tiered_compilation","System.StageName":"Build","DefinitionName":"coreclr-release-outerloop-nightly"} Schedule
v-wenyuxu commented 4 months ago

Failed in: runtime-coreclr r2r 20240417.1

Failed tests:

R2R-CG2 linux arm Checked no_tiered_compilation @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd

Error message:

 Test Infrastructure Failure: System.AggregateException: Not all processes in process tree could be terminated. (Operation not permitted)
 ---> System.ComponentModel.Win32Exception (1): Operation not permitted
   --- End of inner exception stack trace ---
   at System.Diagnostics.Process.Kill(Boolean entireProcessTree) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.NonUap.cs:line 31
   at CoreclrTestLib.CoreclrTestWrapperLib.CollectCrashDumpWithCreateDump(Process process, String crashDumpPath, StreamWriter outputWriter)
   at CoreclrTestLib.CoreclrTestWrapperLib.RunTest(String executable, String outputFile, String errorFile, String category, String testBinaryBase, String outputDir)
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)

Stack trace:

   at Xunit.Assert.Fail(String message) in /_/src/Microsoft.DotNet.XUnitAssert/src/FailAsserts.cs:line 34
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)
   at Program.<<Main>$>g__TestExecutor253|25_254(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass25_0&)
Queued Pipeline Pipeline_Configuration OS Arch Test Outcome Properties BuildReason
2024-03-31T07:33:02.486Z coreclr-release-outerloop-nightly Release-no_tiered_compilation ubuntu.2004.armarch.open arm Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd Failed {"CollectionUri":"https://dev.azure.com/dnceng-public/","System.StageAttempt":"1","AzurePipelinesTestRunId":"15321360","System.JobId":"c8f31202-1062-5d6a-be4d-45312a30ae25","System.StageName":"Build","configuration":"Release-no_tiered_compilation","System.PhaseAttempt":"1","operatingSystem":"(Alpine.316.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.17-helix-arm32v7","architecture":"arm","BuildNumber":"20240330.1","DefinitionName":"coreclr-release-outerloop-nightly","Project":"public","System.JobName":"__default","Reason":"Schedule","System.JobAttempt":"1","System.PhaseName":"run_test_p1_R2R_linux_musl_arm_release","DefinitionId":"135","BuildId":"625141"} Schedule
2024-03-29T07:32:29.466Z coreclr-release-outerloop-nightly Release-no_tiered_compilation ubuntu.2004.armarch.open arm Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd Failed {"Project":"public","Reason":"Schedule","BuildId":"623537","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseName":"run_test_p1_R2R_linux_musl_arm_release","architecture":"arm","AzurePipelinesTestRunId":"15273954","System.JobName":"__default","DefinitionId":"135","System.PhaseAttempt":"1","System.StageAttempt":"1","BuildNumber":"20240328.1","System.JobAttempt":"1","System.JobId":"c8f31202-1062-5d6a-be4d-45312a30ae25","operatingSystem":"(Alpine.316.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.17-helix-arm32v7","configuration":"Release-no_tiered_compilation","System.StageName":"Build","DefinitionName":"coreclr-release-outerloop-nightly"} Schedule
v-wenyuxu commented 4 months ago

Failed in: runtime-coreclr r2r 20240419.1

Failed tests:

R2R-CG2 linux arm Checked @ (Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd

Error message:

 Test Infrastructure Failure: System.AggregateException: Not all processes in process tree could be terminated. (Operation not permitted)
 ---> System.ComponentModel.Win32Exception (1): Operation not permitted
   --- End of inner exception stack trace ---
   at System.Diagnostics.Process.Kill(Boolean entireProcessTree) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.NonUap.cs:line 31
   at CoreclrTestLib.CoreclrTestWrapperLib.CollectCrashDumpWithCreateDump(Process process, String crashDumpPath, StreamWriter outputWriter)
   at CoreclrTestLib.CoreclrTestWrapperLib.CollectCrashDump(Process process, String crashDumpPath, StreamWriter outputWriter)
   at CoreclrTestLib.CoreclrTestWrapperLib.RunTest(String executable, String outputFile, String errorFile, String category, String testBinaryBase, String outputDir)
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)

Stack trace:

   at Xunit.Assert.Fail(String message) in /_/src/Microsoft.DotNet.XUnitAssert/src/FailAsserts.cs:line 34
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)
   at Program.<<Main>$>g__TestExecutor253|25_254(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass25_0&)
Queued Pipeline Pipeline_Configuration OS Arch Test Outcome Properties BuildReason
2024-03-31T07:33:02.486Z coreclr-release-outerloop-nightly Release-no_tiered_compilation ubuntu.2004.armarch.open arm Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd Failed {"CollectionUri":"https://dev.azure.com/dnceng-public/","System.StageAttempt":"1","AzurePipelinesTestRunId":"15321360","System.JobId":"c8f31202-1062-5d6a-be4d-45312a30ae25","System.StageName":"Build","configuration":"Release-no_tiered_compilation","System.PhaseAttempt":"1","operatingSystem":"(Alpine.316.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.17-helix-arm32v7","architecture":"arm","BuildNumber":"20240330.1","DefinitionName":"coreclr-release-outerloop-nightly","Project":"public","System.JobName":"__default","Reason":"Schedule","System.JobAttempt":"1","System.PhaseName":"run_test_p1_R2R_linux_musl_arm_release","DefinitionId":"135","BuildId":"625141"} Schedule
2024-03-29T07:32:29.466Z coreclr-release-outerloop-nightly Release-no_tiered_compilation ubuntu.2004.armarch.open arm Interop/NativeLibrary/Callback/CallbackStressTest_TargetUnix/CallbackStressTest_TargetUnix.cmd Failed {"Project":"public","Reason":"Schedule","BuildId":"623537","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseName":"run_test_p1_R2R_linux_musl_arm_release","architecture":"arm","AzurePipelinesTestRunId":"15273954","System.JobName":"__default","DefinitionId":"135","System.PhaseAttempt":"1","System.StageAttempt":"1","BuildNumber":"20240328.1","System.JobAttempt":"1","System.JobId":"c8f31202-1062-5d6a-be4d-45312a30ae25","operatingSystem":"(Alpine.316.Arm32.Open)Ubuntu.2004.ArmArch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.17-helix-arm32v7","configuration":"Release-no_tiered_compilation","System.StageName":"Build","DefinitionName":"coreclr-release-outerloop-nightly"} Schedule
mikelle-rogers commented 4 months ago

@jeffhandley, what is the status of this test failure?

jozkee commented 4 months ago

@jkotas would it be OK to ignore "operation not permitted" errors on Kill(entireProcessTree: true)?

jozkee commented 4 months ago

@mikelle-rogers we currently don't have a fix nor a consistent repro, we could spend more time looking into it but not sure if we could get a repro at all.

jkotas commented 4 months ago

would it be OK to ignore "operation not permitted" errors on Kill(entireProcessTree: true)?

I do not know. We need to understand what leads to the "operation not permitted" errors.

jkotas commented 3 months ago

Delete workaround added in https://github.com/dotnet/runtime/pull/102958 once this gets fixed.