Test failure: CoreMangLib/system/span/RefStructWithSpan/RefStructWithSpan.cmd

[v-wenyuxu]

Failed in: runtime-coreclr outerloop 20240617.4

Failed tests:

coreclr windows arm64 Checked no_tiered_compilation @ Windows.11.Arm64.Open
    - CoreMangLib/system/span/RefStructWithSpan/RefStructWithSpan.cmd

Error message:

cmdLine:C:hwB88F09C1wC2F00A46eCoreMangLibCoreMangLib../system/span/RefStructWithSpan/RefStructWithSpan.cmd Timed Out (timeout in milliseconds: 600000 from variable __TestTimeout, start: 6/18/2024 2:47:58 AM, end: 6/18/2024 2:57:58 AM)

Return code:      -100
Raw output file:      C:hwB88F09C1wC2F00A46uploadssystemspanRefStructWithSpanoutput.txt
Raw output:
BEGIN EXECUTION
 "C:hwB88F09C1pcorerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false" -p "System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true"  RefStructWithSpan.dll 

cmdLine:C:hwB88F09C1wC2F00A46eCoreMangLibCoreMangLib../system/span/RefStructWithSpan/RefStructWithSpan.cmd Timed Out (timeout in milliseconds: 600000 from variable __TestTimeout, start: 6/18/2024 2:47:58 AM, end: 6/18/2024 2:57:58 AM)
Test Harness Exitcode is : -100
To run the test:
Set up CORE_ROOT and run.
> C:hwB88F09C1wC2F00A46eCoreMangLibCoreMangLib../system/span/RefStructWithSpan/RefStructWithSpan.cmd

Stack trace:

   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 146
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath, String testPathPrefix)
   at Program.<<Main>$>g__TestExecutor46|0_47(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

[dotnet-policy-service[bot]]

Tagging subscribers to this area: @dotnet/area-system-runtime See info in area-owners.md if you want to be subscribed.

[jkotas]

RefStructWithSpan test was introduced by https://github.com/dotnet/coreclr/pull/16231

[v-wenyuxu]

Failed in: runtime-coreclr outerloop 20240619.4

Failed tests:

coreclr windows x86 Checked @ Windows.10.Amd64.Open
    - CoreMangLib/system/span/RefStructWithSpan/RefStructWithSpan.cmd

Error message:

cmdLine:C:hwA7080977wA8580954eCoreMangLibCoreMangLib../system/span/RefStructWithSpan/RefStructWithSpan.cmd Timed Out (timeout in milliseconds: 600000 from variable __TestTimeout, start: 6/20/2024 2:40:26 AM, end: 6/20/2024 2:50:26 AM)

Return code:      -100
Raw output file:      C:hwA7080977wA8580954uploadssystemspanRefStructWithSpanoutput.txt
Raw output:
BEGIN EXECUTION
 "C:hwA7080977pcorerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false" -p "System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true"  RefStructWithSpan.dll 

cmdLine:C:hwA7080977wA8580954eCoreMangLibCoreMangLib../system/span/RefStructWithSpan/RefStructWithSpan.cmd Timed Out (timeout in milliseconds: 600000 from variable __TestTimeout, start: 6/20/2024 2:40:26 AM, end: 6/20/2024 2:50:26 AM)
Test Harness Exitcode is : -100
To run the test:
Set up CORE_ROOT and run.
> C:hwA7080977wA8580954eCoreMangLibCoreMangLib../system/span/RefStructWithSpan/RefStructWithSpan.cmd

Stack trace:

   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 146
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath, String testPathPrefix)
   at Program.<<Main>$>g__TestExecutor46|0_47(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

[v-wenyuxu]

Failed in: runtime-coreclr outerloop 20240620.2

Failed tests:

coreclr windows x64 Checked @ Windows.10.Amd64.Open
    - JIT/Regression/JitBlue/GitHub_27924/GitHub_27924/GitHub_27924.cmd

Error message:

cmdLine:C:hwAE210920wB9380A20eJITRegressionRegression_3../JitBlue/GitHub_27924/GitHub_27924/GitHub_27924.cmd Timed Out (timeout in milliseconds: 600000 from variable __TestTimeout, start: 6/20/2024 7:40:51 PM, end: 6/20/2024 7:50:51 PM)

Return code:      -100
Raw output file:      C:hwAE210920wB9380A20uploadsJitBlueGitHub_27924GitHub_27924output.txt
Raw output:
BEGIN EXECUTION
 "C:hwAE210920pcorerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false" -p "System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true"  GitHub_27924.dll 

cmdLine:C:hwAE210920wB9380A20eJITRegressionRegression_3../JitBlue/GitHub_27924/GitHub_27924/GitHub_27924.cmd Timed Out (timeout in milliseconds: 600000 from variable __TestTimeout, start: 6/20/2024 7:40:51 PM, end: 6/20/2024 7:50:51 PM)
Test Harness Exitcode is : -100
To run the test:
Set up CORE_ROOT and run.
> C:hwAE210920wB9380A20eJITRegressionRegression_3../JitBlue/GitHub_27924/GitHub_27924/GitHub_27924.cmd

Stack trace:

   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 146
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath, String testPathPrefix)
   at Program.<<Main>$>g__TestExecutor344|0_345(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

[EgorBo]

In this pipeline (x86) multiple test failed with this symptom - timeout. https://dev.azure.com/dnceng-public/public/_build/results?buildId=715523&view=ms.vss-test-web.build-test-results-tab&runId=17909268&resultId=122250&paneView=debug

I was able to repro it locally but unfortunately didn't collect a dump

[EgorBo]

Stack-trace:

[0x0]   ntdll!NtWaitForSingleObject+0x14   0xd20177f0f8   0x7ffc2535952e   
[0x1]   KERNELBASE!WaitForSingleObjectEx+0x8e   0xd20177f100   0x7ffb1a92a0d3   
[0x2]   coreclr!WKS::GCHeap::WaitUntilGCComplete+0x83   0xd20177f1a0   0x7ffb1a6f8539   
[0x3]   coreclr!Thread::RareDisablePreemptiveGC+0x469   0xd20177f1e0   0x7ffb1a08e0d9   
[0x4]   coreclr!Thread::DisablePreemptiveGC+0xd9   0xd20177f300   0x7ffb1a08f98b   
[0x5]   coreclr!GCHolderBase::EnterInternalCoop+0x15b   0xd20177f340   0x7ffb1a076e46   
[0x6]   coreclr!GCCoop::GCCoop+0x56   0xd20177f380   0x7ffb1a3d724d   
[0x7]   coreclr!Thread::DetachThread+0x23d   0xd20177f3d0   0x7ffb1ab08035   
[0x8]   coreclr!TlsDestructionMonitor::~TlsDestructionMonitor+0x155   0xd20177f4c0   0x7ffb1acc5266   

[EgorBo]

Minimal repro (Win-x64): https://gist.github.com/EgorBo/295ab4df2f337306d3d0afd5e158ac93

• run it and wait for 30 sec or so - if it stuck - it hits the bug (100% reproducible for me)

I run it with R2R=0, TC=0 but likely reproes with default args too

[EgorBo]

My bisect script points to https://github.com/dotnet/runtime/pull/103055 cc @jkoritzinsky @jkotas

[dotnet-policy-service[bot]]

Tagging subscribers to this area: @mangod9 See info in area-owners.md if you want to be subscribed.

[JulieLeeMSFT]

It also failed in runtime-coreclr gcstress-extra / 20240625.2.

coreclr windows arm64 Checked heapverify1 @ Windows.11.Arm64.Open

• CoreMangLib/system/span/RefStructWithSpan/RefStructWithSpan.cmd

[AndyAyersMS]

Ah this looks like what I saw in https://github.com/dotnet/runtime/issues/104057 too.