Closed v-wenyuxu closed 1 month ago
Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch See info in area-owners.md if you want to be subscribed.
Does not repro in the latest runs. I don't think this particular test is a top-priority fix, I'll take look after P7 snap.
@jkotas @cshung There is a race condition in nongcheap profiler test and I am not sure whether it's expected and we should relax the test or we should make sure PublishObjectAndNotify
is invoked under the gc lock.
So here is the test: https://github.com/dotnet/runtime/blob/main/src/tests/profiler/native/nongcheap/nongcheap.cpp
Basically, we count all ObjectAllocated
events and sum the number of nongc objects we counted. Then, as part of GarbageCollectionFinished
callback (which is performed under gc lock) we enumerate all the existing nongc objects and check whether it matches with the numbers we recorded via the ObjectAllocated
event.
It seems that we might miss ObjectAllocated
for a newly allocated nongc object when GarbageCollectionFinished
started because of this publish which is not under any lock if I understand it correctly. So, roughly:
GarbageCollectionFinished
in the end of it, still under the lock)Hence, GarbageCollectionFinished
will see our new object while ObjectAllocated
won't.
My understanding that it's a correct behavior and we should just relax the test (e.g just to check that both numbers are not zero). thoughts?
My understanding that it's a correct behavior and we should just relax the test (e.g just to check that both numbers are not zero). thoughts?
I agree that product behavior is fine and the test is where we should adjust.
Profilee STDOUT: FAIL: objectAllocated(77) != _nonGcHeapObjects(76)
Hence, GarbageCollectionFinished will see our new object while ObjectAllocated won't.
I just noticed your description looks inverted relative to the test output, but checking the test code I think the error message printing swapped the order of the two parameters. The current PR doesn't have that message but in case it comes back it would be nice to fix that.
Failed in: runtime-coreclr jitstress 20240715.2
Failed tests:
Error message:
Stack trace:
More failures
Queued | Pipeline | Pipeline_Configuration | OS | Arch | Test | Outcome | Properties | BuildReason -- | -- | -- | -- | -- | -- | -- | -- | -- 2024-06-22T21:12:15.352Z | runtime-coreclr jitstress-isas-x86 | Checked-jitstress_isas_2_x86_nosse3_4 | osx.1200.amd64.open | x64 | profiler/gc/nongcheap/nongcheap.sh | Failed | {"architecture":"x64","System.PhaseName":"run_test_p1__osx_x64_checked","DefinitionId":"115","AzurePipelinesTestRunId":"17954534","configuration":"Checked-jitstress_isas_2_x86_nosse3_4","operatingSystem":"OSX.1200.Amd64.Open","System.StageAttempt":"1","System.JobAttempt":"1","CollectionUri":"https://dev.azure.com/dnceng-public/","Reason":"Schedule","System.StageName":"Build","Project":"public","DefinitionName":"runtime-coreclr jitstress-isas-x86","System.JobName":"__default","BuildNumber":"20240622.1","System.JobId":"bf644a0b-2dbe-5a03-d8c6-a86fdd97c823","System.PhaseAttempt":"1","BuildId":"716994"} | Schedule 2024-06-22T21:04:37.18Z | runtime-coreclr jitstress-isas-x86 | Checked-jitstress_isas_2_x86_nosse3 | osx.1200.amd64.open | x64 | profiler/gc/nongcheap/nongcheap.sh | Failed | {"Reason":"Schedule","System.PhaseAttempt":"1","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"716994","System.StageAttempt":"1","AzurePipelinesTestRunId":"17954002","System.StageName":"Build","Project":"public","System.JobId":"bf644a0b-2dbe-5a03-d8c6-a86fdd97c823","operatingSystem":"OSX.1200.Amd64.Open","System.PhaseName":"run_test_p1__osx_x64_checked","BuildNumber":"20240622.1","DefinitionName":"runtime-coreclr jitstress-isas-x86","System.JobAttempt":"1","configuration":"Checked-jitstress_isas_2_x86_nosse3","DefinitionId":"115","System.JobName":"__default","architecture":"x64"} | Schedule 2024-06-22T21:04:03.512Z | runtime-coreclr jitstress-isas-x86 | Checked-jitstress_isas_2_x86_nosse3 | ubuntu.2204.amd64.open.rt | x64 | profiler/gc/nongcheap/nongcheap.sh | Failed | {"AzurePipelinesTestRunId":"17954122","System.JobAttempt":"1","Reason":"Schedule","Project":"public","System.JobId":"6dfec640-5b96-5d95-2a5b-7c236ea2d5c5","System.PhaseAttempt":"1","DefinitionId":"115","operatingSystem":"Ubuntu.2204.Amd64.Open","configuration":"Checked-jitstress_isas_2_x86_nosse3","DefinitionName":"runtime-coreclr jitstress-isas-x86","System.PhaseName":"run_test_p1__linux_x64_checked","System.StageAttempt":"1","architecture":"x64","BuildId":"716994","BuildNumber":"20240622.1","CollectionUri":"https://dev.azure.com/dnceng-public/","System.StageName":"Build","System.JobName":"__default"} | Schedule 2024-05-09T02:18:47.007Z | runtime-coreclr outerloop | Checked-no_tiered_compilation | ubuntu.2004.armarch.open | arm | profiler/gc/nongcheap/nongcheap.sh | Failed | {"System.JobName":"__default","Reason":"Schedule","System.PhaseAttempt":"1","architecture":"arm","configuration":"Checked-no_tiered_compilation","CollectionUri":"https://dev.azure.com/dnceng-public/","System.JobId":"860c965f-b28e-50d5-65d2-e40bd59f3a69","Project":"public","DefinitionName":"runtime-coreclr outerloop","operatingSystem":"(Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7","BuildId":"670046","System.StageAttempt":"1","System.JobAttempt":"1","System.StageName":"Build","System.PhaseName":"run_test_p1__linux_arm_checked","BuildNumber":"20240508.3","DefinitionId":"108","AzurePipelinesTestRunId":"16612930"} | Schedule 2024-05-09T02:18:45.812Z | runtime-coreclr outerloop | Checked | ubuntu.2004.armarch.open | arm | profiler/gc/nongcheap/nongcheap.sh | Failed | {"DefinitionId":"108","DefinitionName":"runtime-coreclr outerloop","System.StageName":"Build","System.JobAttempt":"1","BuildId":"670046","Reason":"Schedule","System.JobId":"860c965f-b28e-50d5-65d2-e40bd59f3a69","Project":"public","BuildNumber":"20240508.3","operatingSystem":"(Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7","AzurePipelinesTestRunId":"16612920","configuration":"Checked","architecture":"arm","System.PhaseName":"run_test_p1__linux_arm_checked","CollectionUri":"https://dev.azure.com/dnceng-public/","System.StageAttempt":"1","System.PhaseAttempt":"1","System.JobName":"__default"} | Schedule 2024-05-08T20:55:20.943Z | runtime | Checked-no_tiered_compilation | ubuntu.2004.armarch.open | arm | profiler/gc/nongcheap/nongcheap.sh | Failed | {"System.JobAttempt":"1","System.PhaseName":"run_test_p0_coreclr__linux_arm_checked","Reason":"Schedule","System.StageName":"Build","operatingSystem":"(Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7","System.JobId":"5875517b-40f2-5b27-5b45-8b8aba144595","configuration":"Checked-no_tiered_compilation","DefinitionName":"runtime","DefinitionId":"129","System.JobName":"__default","System.PhaseAttempt":"1","BuildId":"669797","architecture":"arm","AzurePipelinesTestRunId":"16607856","BuildNumber":"20240508.40","Project":"public","System.StageAttempt":"1","CollectionUri":"https://dev.azure.com/dnceng-public/"} | Schedule 2024-05-08T20:55:19.474Z | runtime | Checked | ubuntu.2004.armarch.open | arm | profiler/gc/nongcheap/nongcheap.sh | Failed | {"DefinitionName":"runtime","operatingSystem":"(Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7","System.PhaseName":"run_test_p0_coreclr__linux_arm_checked","configuration":"Checked","DefinitionId":"129","BuildNumber":"20240508.40","System.JobAttempt":"1","System.StageName":"Build","BuildId":"669797","CollectionUri":"https://dev.azure.com/dnceng-public/","System.JobName":"__default","Reason":"Schedule","System.StageAttempt":"1","System.JobId":"5875517b-40f2-5b27-5b45-8b8aba144595","Project":"public","System.PhaseAttempt":"1","AzurePipelinesTestRunId":"16607854","architecture":"arm"} | Schedule 2024-05-08T19:20:43.463Z | runtime-coreclr outerloop | Checked-no_tiered_compilation | ubuntu.2004.armarch.open | arm | profiler/gc/nongcheap/nongcheap.sh | Failed | {"AzurePipelinesTestRunId":"16606342","DefinitionName":"runtime-coreclr outerloop","BuildId":"669628","architecture":"arm","configuration":"Checked-no_tiered_compilation","Project":"public","System.JobId":"860c965f-b28e-50d5-65d2-e40bd59f3a69","operatingSystem":"(Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7","System.JobAttempt":"1","System.PhaseName":"run_test_p1__linux_arm_checked","Reason":"Schedule","System.PhaseAttempt":"1","System.StageAttempt":"1","System.JobName":"__default","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildNumber":"20240508.2","DefinitionId":"108","System.StageName":"Build"} | Schedule 2024-05-08T19:20:41.712Z | runtime-coreclr outerloop | Checked | ubuntu.2004.armarch.open | arm | profiler/gc/nongcheap/nongcheap.sh | Failed | {"BuildNumber":"20240508.2","DefinitionId":"108","System.JobId":"860c965f-b28e-50d5-65d2-e40bd59f3a69","System.JobName":"__default","DefinitionName":"runtime-coreclr outerloop","AzurePipelinesTestRunId":"16606340","System.StageName":"Build","Reason":"Schedule","System.PhaseAttempt":"1","configuration":"Checked","architecture":"arm","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"669628","System.JobAttempt":"1","System.StageAttempt":"1","Project":"public","System.PhaseName":"run_test_p1__linux_arm_checked","operatingSystem":"(Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7"} | Schedule 2024-05-08T10:18:34.795Z | runtime-coreclr outerloop | Checked-no_tiered_compilation | ubuntu.2004.armarch.open | arm | profiler/gc/nongcheap/nongcheap.sh | Failed | {"System.StageName":"Build","Project":"public","AzurePipelinesTestRunId":"16596180","System.PhaseName":"run_test_p1__linux_arm_checked","BuildNumber":"20240508.1","System.JobAttempt":"1","System.JobName":"__default","BuildId":"669221","configuration":"Checked-no_tiered_compilation","Reason":"Schedule","operatingSystem":"(Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7","System.StageAttempt":"1","architecture":"arm","DefinitionId":"108","System.PhaseAttempt":"1","DefinitionName":"runtime-coreclr outerloop","System.JobId":"860c965f-b28e-50d5-65d2-e40bd59f3a69","CollectionUri":"https://dev.azure.com/dnceng-public/"} | Schedule 2024-05-08T10:18:33.232Z | runtime-coreclr outerloop | Checked | ubuntu.2004.armarch.open | arm | profiler/gc/nongcheap/nongcheap.sh | Failed | {"System.JobName":"__default","AzurePipelinesTestRunId":"16596178","System.PhaseName":"run_test_p1__linux_arm_checked","architecture":"arm","CollectionUri":"https://dev.azure.com/dnceng-public/","System.StageName":"Build","BuildId":"669221","DefinitionName":"runtime-coreclr outerloop","System.PhaseAttempt":"1","System.JobAttempt":"1","Project":"public","System.JobId":"860c965f-b28e-50d5-65d2-e40bd59f3a69","DefinitionId":"108","BuildNumber":"20240508.1","configuration":"Checked","Reason":"Schedule","System.StageAttempt":"1","operatingSystem":"(Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7"} | Schedule 2024-05-08T09:16:15.601Z | runtime-coreclr pgo | Checked-defaultpgo | ubuntu.2004.armarch.open | arm | profiler/gc/nongcheap/nongcheap.sh | Failed | {"System.JobId":"860c965f-b28e-50d5-65d2-e40bd59f3a69","BuildNumber":"20240508.1","Reason":"Schedule","System.JobAttempt":"1","System.PhaseName":"run_test_p1__linux_arm_checked","System.PhaseAttempt":"1","BuildId":"669200","Project":"public","operatingSystem":"(Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7","System.JobName":"__default","DefinitionId":"144","AzurePipelinesTestRunId":"16595346","DefinitionName":"runtime-coreclr pgo","System.StageAttempt":"1","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"arm","configuration":"Checked-defaultpgo","System.StageName":"Build"} | Schedule 2024-05-08T08:55:12.152Z | runtime | Checked-no_tiered_compilation | ubuntu.2004.armarch.open | arm | profiler/gc/nongcheap/nongcheap.sh | Failed | {"System.StageName":"Build","AzurePipelinesTestRunId":"16594958","System.JobName":"__default","configuration":"Checked-no_tiered_compilation","BuildNumber":"20240508.1","CollectionUri":"https://dev.azure.com/dnceng-public/","System.StageAttempt":"1","DefinitionId":"129","System.JobId":"5875517b-40f2-5b27-5b45-8b8aba144595","System.PhaseName":"run_test_p0_coreclr__linux_arm_checked","architecture":"arm","Reason":"Schedule","System.PhaseAttempt":"1","operatingSystem":"(Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7","DefinitionName":"runtime","BuildId":"669202","System.JobAttempt":"1","Project":"public"} | Schedule 2024-05-08T08:55:10.576Z | runtime | Checked | ubuntu.2004.armarch.open | arm | profiler/gc/nongcheap/nongcheap.sh | Failed | {"System.StageName":"Build","DefinitionId":"129","configuration":"Checked","System.StageAttempt":"1","BuildNumber":"20240508.1","System.JobId":"5875517b-40f2-5b27-5b45-8b8aba144595","AzurePipelinesTestRunId":"16594956","DefinitionName":"runtime","Reason":"Schedule","Project":"public","BuildId":"669202","System.JobAttempt":"1","System.PhaseAttempt":"1","CollectionUri":"https://dev.azure.com/dnceng-public/","operatingSystem":"(Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7","System.PhaseName":"run_test_p0_coreclr__linux_arm_checked","architecture":"arm","System.JobName":"__default"} | Schedule 2024-05-08T07:27:00.809Z | coreclr-release-outerloop-nightly | Release-no_tiered_compilation | ubuntu.2004.armarch.open | arm | profiler/gc/nongcheap/nongcheap.sh | Failed | {"System.PhaseAttempt":"1","architecture":"arm","DefinitionName":"coreclr-release-outerloop-nightly","System.JobName":"__default","AzurePipelinesTestRunId":"16592990","configuration":"Release-no_tiered_compilation","BuildNumber":"20240507.1","BuildId":"669106","System.JobAttempt":"1","DefinitionId":"135","System.StageAttempt":"1","System.StageName":"Build","operatingSystem":"(Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7","Project":"public","System.PhaseName":"run_test_p1_R2R_linux_arm_release","System.JobId":"72dfe29e-04cf-5892-e89c-87a9854dd748","CollectionUri":"https://dev.azure.com/dnceng-public/","Reason":"Schedule"} | Schedule 2024-05-08T07:26:57.972Z | coreclr-release-outerloop-nightly | Release | ubuntu.2004.armarch.open | arm | profiler/gc/nongcheap/nongcheap.sh | Failed | {"BuildId":"669106","AzurePipelinesTestRunId":"16592988","System.PhaseName":"run_test_p1_R2R_linux_arm_release","System.PhaseAttempt":"1","System.JobAttempt":"1","Reason":"Schedule","DefinitionId":"135","architecture":"arm","BuildNumber":"20240507.1","System.StageAttempt":"1","System.JobId":"72dfe29e-04cf-5892-e89c-87a9854dd748","CollectionUri":"https://dev.azure.com/dnceng-public/","Project":"public","System.JobName":"__default","System.StageName":"Build","DefinitionName":"coreclr-release-outerloop-nightly","operatingSystem":"(Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7","configuration":"Release"} | Schedule 2024-05-08T07:23:41.546Z | coreclr-release-outerloop-nightly | Release-no_tiered_compilation | ubuntu.2004.armarch.open | arm | profiler/gc/nongcheap/nongcheap.sh | Failed | {"DefinitionName":"coreclr-release-outerloop-nightly","operatingSystem":"(Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7","System.PhaseName":"run_test_p1__linux_arm_release","Reason":"Schedule","System.StageAttempt":"1","DefinitionId":"135","System.JobName":"__default","BuildId":"669106","System.StageName":"Build","BuildNumber":"20240507.1","System.JobAttempt":"1","System.PhaseAttempt":"1","AzurePipelinesTestRunId":"16592840","architecture":"arm","configuration":"Release-no_tiered_compilation","Project":"public","CollectionUri":"https://dev.azure.com/dnceng-public/","System.JobId":"ae69a820-8040-5321-6217-15d430a2ab03"} | Schedule 2024-05-08T07:23:40.155Z | coreclr-release-outerloop-nightly | Release | ubuntu.2004.armarch.open | arm | profiler/gc/nongcheap/nongcheap.sh | Failed | {"DefinitionId":"135","System.StageName":"Build","DefinitionName":"coreclr-release-outerloop-nightly","System.JobName":"__default","AzurePipelinesTestRunId":"16592838","Reason":"Schedule","configuration":"Release","System.JobId":"ae69a820-8040-5321-6217-15d430a2ab03","System.StageAttempt":"1","System.PhaseName":"run_test_p1__linux_arm_release","BuildNumber":"20240507.1","Project":"public","BuildId":"669106","System.PhaseAttempt":"1","System.JobAttempt":"1","architecture":"arm","CollectionUri":"https://dev.azure.com/dnceng-public/","operatingSystem":"(Ubuntu.1804.Arm32.Open)Ubuntu.2004.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7"} | Schedule 2024-05-06T05:49:19.611Z | runtime-coreclr jitstress | Checked-jitstress1_tiered | osx.1200.amd64.open | x64 | profiler/gc/nongcheap/nongcheap.sh | Failed | {"DefinitionId":"109","Project":"public","AzurePipelinesTestRunId":"16528982","System.JobAttempt":"1","System.JobId":"bf644a0b-2dbe-5a03-d8c6-a86fdd97c823","BuildNumber":"20240505.1","System.PhaseAttempt":"1","Reason":"Schedule","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"666407","System.PhaseName":"run_test_p1__osx_x64_checked","System.StageName":"Build","System.JobName":"__default","operatingSystem":"OSX.1200.Amd64.Open","System.StageAttempt":"1","DefinitionName":"runtime-coreclr jitstress","configuration":"Checked-jitstress1_tiered","architecture":"x64"} | Schedule 2024-05-06T05:49:16.621Z | runtime-coreclr jitstress | Checked-jitstress2 | osx.1200.amd64.open | x64 | profiler/gc/nongcheap/nongcheap.sh | Failed | {"Project":"public","System.PhaseName":"run_test_p1__osx_x64_checked","configuration":"Checked-jitstress2","Reason":"Schedule","System.JobName":"__default","DefinitionName":"runtime-coreclr jitstress","AzurePipelinesTestRunId":"16528984","CollectionUri":"https://dev.azure.com/dnceng-public/","architecture":"x64","operatingSystem":"OSX.1200.Amd64.Open","BuildNumber":"20240505.1","System.StageAttempt":"1","System.JobAttempt":"1","BuildId":"666407","System.StageName":"Build","System.JobId":"bf644a0b-2dbe-5a03-d8c6-a86fdd97c823","System.PhaseAttempt":"1","DefinitionId":"109"} | Schedule 2024-05-06T05:49:15.267Z | runtime-coreclr jitstress | Checked-jitstress1 | osx.1200.amd64.open | x64 | profiler/gc/nongcheap/nongcheap.sh | Failed | {"operatingSystem":"OSX.1200.Amd64.Open","System.JobAttempt":"1","AzurePipelinesTestRunId":"16528986","System.JobName":"__default","Reason":"Schedule","System.PhaseName":"run_test_p1__osx_x64_checked","System.PhaseAttempt":"1","DefinitionId":"109","System.StageName":"Build","System.StageAttempt":"1","DefinitionName":"runtime-coreclr jitstress","architecture":"x64","configuration":"Checked-jitstress1","Project":"public","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"666407","System.JobId":"bf644a0b-2dbe-5a03-d8c6-a86fdd97c823","BuildNumber":"20240505.1"} | Schedule 2024-05-06T05:49:13.771Z | runtime-coreclr jitstress | Checked-jitstress2_tiered | osx.1200.amd64.open | x64 | profiler/gc/nongcheap/nongcheap.sh | Failed | {"configuration":"Checked-jitstress2_tiered","System.PhaseAttempt":"1","architecture":"x64","System.JobAttempt":"1","operatingSystem":"OSX.1200.Amd64.Open","BuildNumber":"20240505.1","Project":"public","DefinitionId":"109","CollectionUri":"https://dev.azure.com/dnceng-public/","System.StageAttempt":"1","DefinitionName":"runtime-coreclr jitstress","BuildId":"666407","System.StageName":"Build","System.JobId":"bf644a0b-2dbe-5a03-d8c6-a86fdd97c823","Reason":"Schedule","System.PhaseName":"run_test_p1__osx_x64_checked","System.JobName":"__default","AzurePipelinesTestRunId":"16528988"} | Schedule 2024-05-06T05:41:52.477Z | runtime-coreclr jitstress | Checked-jitstress1_tiered | osx.1200.arm64.open | arm64 | profiler/gc/nongcheap/nongcheap.sh | Failed | {"AzurePipelinesTestRunId":"16528954","System.PhaseAttempt":"1","System.JobAttempt":"1","DefinitionId":"109","System.JobName":"__default","Reason":"Schedule","System.StageAttempt":"1","operatingSystem":"OSX.1200.ARM64.Open","System.StageName":"Build","System.PhaseName":"run_test_p1__osx_arm64_checked","BuildNumber":"20240505.1","configuration":"Checked-jitstress1_tiered","System.JobId":"765fd5e9-e47f-5cb0-4cbf-499eaaaee93e","architecture":"arm64","DefinitionName":"runtime-coreclr jitstress","CollectionUri":"https://dev.azure.com/dnceng-public/","Project":"public","BuildId":"666407"} | Schedule 2024-05-06T05:41:51.027Z | runtime-coreclr jitstress | Checked-jitstress2_tiered | osx.1200.arm64.open | arm64 | profiler/gc/nongcheap/nongcheap.sh | Failed | {"System.PhaseName":"run_test_p1__osx_arm64_checked","BuildId":"666407","DefinitionName":"runtime-coreclr jitstress","operatingSystem":"OSX.1200.ARM64.Open","configuration":"Checked-jitstress2_tiered","AzurePipelinesTestRunId":"16528952","System.JobAttempt":"1","System.JobId":"765fd5e9-e47f-5cb0-4cbf-499eaaaee93e","BuildNumber":"20240505.1","DefinitionId":"109","System.StageName":"Build","Reason":"Schedule","architecture":"arm64","Project":"public","System.JobName":"__default","System.StageAttempt":"1","System.PhaseAttempt":"1","CollectionUri":"https://dev.azure.com/dnceng-public/"} | Schedule 2024-05-06T05:41:48.324Z | runtime-coreclr jitstress | Checked-jitstress2 | osx.1200.arm64.open | arm64 | profiler/gc/nongcheap/nongcheap.sh | Failed | {"DefinitionName":"runtime-coreclr jitstress","System.JobId":"765fd5e9-e47f-5cb0-4cbf-499eaaaee93e","CollectionUri":"https://dev.azure.com/dnceng-public/","Project":"public","DefinitionId":"109","Reason":"Schedule","System.PhaseName":"run_test_p1__osx_arm64_checked","System.PhaseAttempt":"1","architecture":"arm64","System.StageAttempt":"1","BuildId":"666407","BuildNumber":"20240505.1","System.StageName":"Build","operatingSystem":"OSX.1200.ARM64.Open","configuration":"Checked-jitstress2","System.JobAttempt":"1","System.JobName":"__default","AzurePipelinesTestRunId":"16528950"} | Schedule 2024-05-06T05:41:45.449Z | runtime-coreclr jitstress | Checked-jitstress1 | osx.1200.arm64.open | arm64 | profiler/gc/nongcheap/nongcheap.sh | Failed | {"System.PhaseName":"run_test_p1__osx_arm64_checked","System.JobId":"765fd5e9-e47f-5cb0-4cbf-499eaaaee93e","BuildNumber":"20240505.1","architecture":"arm64","System.JobName":"__default","configuration":"Checked-jitstress1","System.PhaseAttempt":"1","operatingSystem":"OSX.1200.ARM64.Open","DefinitionId":"109","System.JobAttempt":"1","Project":"public","System.StageName":"Build","Reason":"Schedule","CollectionUri":"https://dev.azure.com/dnceng-public/","System.StageAttempt":"1","AzurePipelinesTestRunId":"16528956","BuildId":"666407","DefinitionName":"runtime-coreclr jitstress"} | Schedule 2024-05-06T05:24:02.857Z | runtime-coreclr jitstress | Checked-jitstress1 | ubuntu.2204.amd64.open.rt | x64 | profiler/gc/nongcheap/nongcheap.sh | Failed | {"Reason":"Schedule","Project":"public","System.JobAttempt":"1","configuration":"Checked-jitstress1","CollectionUri":"https://dev.azure.com/dnceng-public/","DefinitionId":"109","DefinitionName":"runtime-coreclr jitstress","architecture":"x64","System.PhaseName":"run_test_p1__linux_x64_checked","System.StageName":"Build","System.JobId":"6dfec640-5b96-5d95-2a5b-7c236ea2d5c5","BuildId":"666407","AzurePipelinesTestRunId":"16528844","System.PhaseAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","BuildNumber":"20240505.1","System.JobName":"__default","System.StageAttempt":"1"} | Schedule 2024-05-06T05:24:01.384Z | runtime-coreclr jitstress | Checked-jitstress2_tiered | ubuntu.2204.amd64.open.rt | x64 | profiler/gc/nongcheap/nongcheap.sh | Failed | {"configuration":"Checked-jitstress2_tiered","System.StageAttempt":"1","AzurePipelinesTestRunId":"16528852","System.JobName":"__default","CollectionUri":"https://dev.azure.com/dnceng-public/","Reason":"Schedule","System.JobAttempt":"1","BuildNumber":"20240505.1","System.JobId":"6dfec640-5b96-5d95-2a5b-7c236ea2d5c5","System.PhaseName":"run_test_p1__linux_x64_checked","operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageName":"Build","System.PhaseAttempt":"1","DefinitionId":"109","BuildId":"666407","DefinitionName":"runtime-coreclr jitstress","architecture":"x64","Project":"public"} | Schedule 2024-05-06T05:24:00.009Z | runtime-coreclr jitstress | Checked-jitstress2 | ubuntu.2204.amd64.open.rt | x64 | profiler/gc/nongcheap/nongcheap.sh | Failed | {"Project":"public","BuildNumber":"20240505.1","System.JobName":"__default","Reason":"Schedule","System.StageName":"Build","operatingSystem":"Ubuntu.2204.Amd64.Open","System.PhaseName":"run_test_p1__linux_x64_checked","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"666407","AzurePipelinesTestRunId":"16528842","configuration":"Checked-jitstress2","DefinitionId":"109","System.StageAttempt":"1","architecture":"x64","System.JobAttempt":"1","DefinitionName":"runtime-coreclr jitstress","System.JobId":"6dfec640-5b96-5d95-2a5b-7c236ea2d5c5","System.PhaseAttempt":"1"} | Schedule 2024-05-06T05:23:57.087Z | runtime-coreclr jitstress | Checked-jitstress1_tiered | ubuntu.2204.amd64.open.rt | x64 | profiler/gc/nongcheap/nongcheap.sh | Failed | {"architecture":"x64","System.StageAttempt":"1","System.PhaseName":"run_test_p1__linux_x64_checked","Reason":"Schedule","operatingSystem":"Ubuntu.2204.Amd64.Open","System.JobAttempt":"1","BuildId":"666407","AzurePipelinesTestRunId":"16528846","System.JobName":"__default","DefinitionName":"runtime-coreclr jitstress","Project":"public","System.PhaseAttempt":"1","DefinitionId":"109","BuildNumber":"20240505.1","System.JobId":"6dfec640-5b96-5d95-2a5b-7c236ea2d5c5","System.StageName":"Build","configuration":"Checked-jitstress1_tiered","CollectionUri":"https://dev.azure.com/dnceng-public/"} | Schedule 2024-05-06T03:47:41.1Z | runtime-coreclr r2r-extra | Checked-jitstress1_tiered | osx.1200.arm64.open | arm64 | profiler/gc/nongcheap/nongcheap.sh | Failed | {"System.JobId":"99f15f99-448d-5593-a271-dbdf553789a6","operatingSystem":"OSX.1200.ARM64.Open","AzurePipelinesTestRunId":"16528370","System.StageName":"Build","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseAttempt":"1","System.JobName":"__default","BuildId":"666362","DefinitionName":"runtime-coreclr r2r-extra","System.StageAttempt":"1","DefinitionId":"114","Reason":"Schedule","Project":"public","BuildNumber":"20240505.1","configuration":"Checked-jitstress1_tiered","System.JobAttempt":"1","architecture":"arm64","System.PhaseName":"run_test_p1_R2R_osx_arm64_checked"} | Schedule 2024-05-06T03:47:37.241Z | runtime-coreclr r2r-extra | Checked-jitstress2 | osx.1200.arm64.open | arm64 | profiler/gc/nongcheap/nongcheap.sh | Failed | {"Reason":"Schedule","configuration":"Checked-jitstress2","operatingSystem":"OSX.1200.ARM64.Open","System.JobAttempt":"1","System.PhaseName":"run_test_p1_R2R_osx_arm64_checked","BuildId":"666362","System.StageName":"Build","System.StageAttempt":"1","Project":"public","AzurePipelinesTestRunId":"16528374","DefinitionName":"runtime-coreclr r2r-extra","BuildNumber":"20240505.1","architecture":"arm64","DefinitionId":"114","CollectionUri":"https://dev.azure.com/dnceng-public/","System.JobName":"__default","System.JobId":"99f15f99-448d-5593-a271-dbdf553789a6","System.PhaseAttempt":"1"} | Schedule 2024-05-06T03:47:23.446Z | runtime-coreclr r2r-extra | Checked-jitstress2_tiered | osx.1200.arm64.open | arm64 | profiler/gc/nongcheap/nongcheap.sh | Failed | {"configuration":"Checked-jitstress2_tiered","System.PhaseName":"run_test_p1_R2R_osx_arm64_checked","DefinitionId":"114","operatingSystem":"OSX.1200.ARM64.Open","System.PhaseAttempt":"1","System.StageName":"Build","Reason":"Schedule","BuildId":"666362","DefinitionName":"runtime-coreclr r2r-extra","System.JobId":"99f15f99-448d-5593-a271-dbdf553789a6","System.JobAttempt":"1","AzurePipelinesTestRunId":"16528362","System.JobName":"__default","System.StageAttempt":"1","CollectionUri":"https://dev.azure.com/dnceng-public/","Project":"public","BuildNumber":"20240505.1","architecture":"arm64"} | Schedule 2024-05-06T03:47:18.634Z | runtime-coreclr r2r-extra | Checked-jitstress1 | osx.1200.arm64.open | arm64 | profiler/gc/nongcheap/nongcheap.sh | Failed | {"System.StageAttempt":"1","architecture":"arm64","Project":"public","System.PhaseName":"run_test_p1_R2R_osx_arm64_checked","operatingSystem":"OSX.1200.ARM64.Open","DefinitionId":"114","System.JobAttempt":"1","AzurePipelinesTestRunId":"16528348","System.JobId":"99f15f99-448d-5593-a271-dbdf553789a6","BuildId":"666362","System.PhaseAttempt":"1","BuildNumber":"20240505.1","System.StageName":"Build","DefinitionName":"runtime-coreclr r2r-extra","CollectionUri":"https://dev.azure.com/dnceng-public/","System.JobName":"__default","Reason":"Schedule","configuration":"Checked-jitstress1"} | Schedule 2024-05-06T03:31:02.455Z | runtime-coreclr r2r-extra | Checked-jitstress1_tiered | ubuntu.2204.amd64.open.rt | x64 | profiler/gc/nongcheap/nongcheap.sh | Failed | {"AzurePipelinesTestRunId":"16527936","BuildNumber":"20240505.1","System.StageName":"Build","System.JobName":"__default","DefinitionId":"114","System.PhaseName":"run_test_p1_R2R_linux_x64_checked","System.JobAttempt":"1","architecture":"x64","Project":"public","CollectionUri":"https://dev.azure.com/dnceng-public/","System.JobId":"45a39349-90d5-51c8-4429-a48b2907fbdd","DefinitionName":"runtime-coreclr r2r-extra","configuration":"Checked-jitstress1_tiered","System.StageAttempt":"1","operatingSystem":"Ubuntu.2204.Amd64.Open","Reason":"Schedule","System.PhaseAttempt":"1","BuildId":"666362"} | Schedule 2024-05-06T03:30:58.033Z | runtime-coreclr r2r-extra | Checked-jitstress2_tiered | ubuntu.2204.amd64.open.rt | x64 | profiler/gc/nongcheap/nongcheap.sh | Failed | {"System.JobId":"45a39349-90d5-51c8-4429-a48b2907fbdd","AzurePipelinesTestRunId":"16527918","System.StageAttempt":"1","System.JobName":"__default","System.PhaseName":"run_test_p1_R2R_linux_x64_checked","CollectionUri":"https://dev.azure.com/dnceng-public/","Reason":"Schedule","architecture":"x64","BuildId":"666362","DefinitionName":"runtime-coreclr r2r-extra","System.PhaseAttempt":"1","System.StageName":"Build","operatingSystem":"Ubuntu.2204.Amd64.Open","BuildNumber":"20240505.1","Project":"public","DefinitionId":"114","System.JobAttempt":"1","configuration":"Checked-jitstress2_tiered"} | Schedule 2024-05-06T03:30:47.385Z | runtime-coreclr r2r-extra | Checked-jitstress2 | ubuntu.2204.amd64.open.rt | x64 | profiler/gc/nongcheap/nongcheap.sh | Failed | {"operatingSystem":"Ubuntu.2204.Amd64.Open","Project":"public","DefinitionName":"runtime-coreclr r2r-extra","System.PhaseAttempt":"1","Reason":"Schedule","CollectionUri":"https://dev.azure.com/dnceng-public/","System.PhaseName":"run_test_p1_R2R_linux_x64_checked","BuildId":"666362","AzurePipelinesTestRunId":"16527924","System.StageAttempt":"1","architecture":"x64","BuildNumber":"20240505.1","configuration":"Checked-jitstress2","System.JobId":"45a39349-90d5-51c8-4429-a48b2907fbdd","DefinitionId":"114","System.JobName":"__default","System.JobAttempt":"1","System.StageName":"Build"} | Schedule 2024-05-06T03:30:45.845Z | runtime-coreclr r2r-extra | Checked-jitstress1 | ubuntu.2204.amd64.open.rt | x64 | profiler/gc/nongcheap/nongcheap.sh | Failed | {"operatingSystem":"Ubuntu.2204.Amd64.Open","System.StageAttempt":"1","DefinitionId":"114","System.StageName":"Build","configuration":"Checked-jitstress1","AzurePipelinesTestRunId":"16527914","System.PhaseName":"run_test_p1_R2R_linux_x64_checked","System.JobAttempt":"1","architecture":"x64","System.JobName":"__default","DefinitionName":"runtime-coreclr r2r-extra","Project":"public","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildId":"666362","System.PhaseAttempt":"1","Reason":"Schedule","System.JobId":"45a39349-90d5-51c8-4429-a48b2907fbdd","BuildNumber":"20240505.1"} | Schedule