Test failure baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.cmd - Githubissues

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.

https://docs.microsoft.com/dotnet/core/

MIT License

14.93k stars 4.64k forks source link

Test failure baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.cmd #83658

Open v-wenyuxu opened 1 year ago

v-wenyuxu commented 1 year ago

Failed in Run: runtime-coreclr outerloop 20230318.3

Failed tests:

 R2R-CG2 windows x64 Checked no_tiered_compilation @ Windows.10.Amd64.Open
     - baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.cmd

Error message:


Return code:      1
Raw output file:      C:hwAC570944wB68409ADuploads
egressions2164oreground-shutdownoutput.txt
Raw output:
BEGIN EXECUTION
foreground-shutdown.dll
        1 file(s) copied.
20:26:15.68
Response file: C:hwAC570944wB68409ADeaseservices   hreading
egressions2164oreground-shutdownoreground-shutdown.dll.rsp
C:hwAC570944wB68409ADeaseservices  hreading
egressions2164oreground-shutdownIL-CG2oreground-shutdown.dll
-o:C:hwAC570944wB68409ADeaseservices   hreading
egressions2164oreground-shutdownoreground-shutdown.dll
--targetarch:x64
--targetos:windows
--verify-type-and-field-layout
--method-layout:random
-r:C:hwAC570944pSystem.*.dll
-r:C:hwAC570944pMicrosoft.*.dll
-r:C:hwAC570944p\xunit.*.dll
-r:C:hwAC570944pmscorlib.dll
-r:C:hwAC570944p
etstandard.dll
-O
" "dotnet" "C:hwAC570944pcrossgen2crossgen2.dll" @"C:hwAC570944wB68409ADeaseservices   hreading
egressions2164oreground-shutdownoreground-shutdown.dll.rsp"   -r:C:hwAC570944wB68409ADeaseservices   hreading    hreading_group1IL-CG2*.dll  -r:C:hwAC570944wB68409ADeaseservices   hreading
egressions2164oreground-shutdownIL-CG2*.dll"
Emitting R2R PE file: C:hwAC570944wB68409ADeaseservices    hreading
egressions2164oreground-shutdownoreground-shutdown.dll
" "dotnet" "C:hwAC570944p
2rdump
2rdump.dll" --header --sc --in C:hwAC570944wB68409ADeaseservices   hreading
egressions2164oreground-shutdownoreground-shutdown.dll --out C:hwAC570944wB68409ADeaseservices   hreading
egressions2164oreground-shutdownoreground-shutdown.dll.r2rdump --val"
20:26:17.71
 "C:hwAC570944pcorerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false"  foreground-shutdown.dll 
Xunit.Sdk.EqualException: Assert.Equal() Failure
Expected: 100
Actual:   101
   at Xunit.Assert.Equal[T](T expected, T actual, IEqualityComparer`1 comparer) in /_/src/xunit.assert/Asserts/EqualityAsserts.cs:line 96
   at __GeneratedMainWrapper.Main()
Expected: 100
Actual: 101
END EXECUTION - FAILED
FAILED
Test failed. Trying to see if dump file was created in C:cores since 3/18/2023 8:26:15 PM
Test Harness Exitcode is : 1
To run the test:
Set up CORE_ROOT and run.
> C:hwAC570944wB68409ADeaseservices    hreading    hreading_group1../regressions/2164/foreground-shutdown/foreground-shutdown.cmd
Expected: True
Actual:   False

Stack trace:

   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/xunit.assert/Asserts/BooleanAsserts.cs:line 132
   at Xunit.Assert.True(Boolean condition, String userMessage) in /_/src/xunit.assert/Asserts/BooleanAsserts.cs:line 116
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String basePath, String assemblyPath)
   at Program.<<Main>$>g__TestExecutor2|0_1(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

Queued	Pipeline	Pipeline_Configuration	OS	Arch	Test	Outcome	Properties	BuildReason
2022-11-19T20:51:51.061Z	runtime-coreclr ilasm	Checked-ilasmroundtrip	ubuntu.1804.amd64.open.rt	x64	baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.sh	Failed	{"System.StageName":"Build","DefinitionId":"140","BuildId":"88711","Project":"public","System.JobAttempt":"1","configuration":"Checked-ilasmroundtrip","System.StageAttempt":"1","AzurePipelinesTestRunId":"1867338","CollectionUri":"https://dev.azure.com/dnceng-public/","DefinitionName":"runtime-coreclr ilasm","BuildNumber":"20221119.1","System.PhaseName":"run_test_p1Linux_x64_checked","operatingSystem":"Ubuntu.1804.Amd64.Open","System.JobName":"default","System.PhaseAttempt":"1","architecture":"x64","System.JobId":"6dfec640-5b96-5d95-2a5b-7c236ea2d5c5","Reason":"Schedule"}	Schedule
2022-11-19T20:51:17.451Z	runtime-coreclr ilasm	Checked-ilasmroundtrip	ubuntu.1804.armarch.open	arm64	baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.sh	Failed	{"architecture":"arm64","System.JobId":"34b2f53b-add1-5e74-7a39-5a11483297fc","BuildId":"88711","Project":"public","System.PhaseName":"run_test_p1Linux_arm64_checked","DefinitionId":"140","System.JobName":"default","CollectionUri":"https://dev.azure.com/dnceng-public/","AzurePipelinesTestRunId":"1867336","Reason":"Schedule","operatingSystem":"(Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8","System.StageAttempt":"1","System.PhaseAttempt":"1","BuildNumber":"20221119.1","System.StageName":"Build","System.JobAttempt":"1","configuration":"Checked-ilasmroundtrip","DefinitionName":"runtime-coreclr ilasm"}	Schedule
2022-11-19T20:48:25.822Z	runtime-coreclr ilasm	Checked-ilasmroundtrip	ubuntu.1804.armarch.open	arm	baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.sh	Failed	{"BuildNumber":"20221119.1","operatingSystem":"(Ubuntu.1804.Arm32.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7","AzurePipelinesTestRunId":"1867310","DefinitionId":"140","System.StageName":"Build","System.JobId":"860c965f-b28e-50d5-65d2-e40bd59f3a69","architecture":"arm","CollectionUri":"https://dev.azure.com/dnceng-public/","System.JobAttempt":"1","DefinitionName":"runtime-coreclr ilasm","System.PhaseName":"run_test_p1Linux_arm_checked","BuildId":"88711","System.JobName":"default","Reason":"Schedule","System.StageAttempt":"1","Project":"public","System.PhaseAttempt":"1","configuration":"Checked-ilasmroundtrip"}	Schedule

dotnet-issue-labeler[bot] commented 1 year ago

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

ghost commented 1 year ago

Tagging subscribers to this area: @mangod9 See info in area-owners.md if you want to be subscribed.

Issue Details

Failed in Run: [runtime-coreclr outerloop 20230318.3](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=209759) **Failed tests:** ``` R2R-CG2 windows x64 Checked no_tiered_compilation @ Windows.10.Amd64.Open - baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.cmd ``` **Error message:** ``` Return code: 1 Raw output file: C:hwAC570944wB68409ADuploads egressions2164oreground-shutdownoutput.txt Raw output: BEGIN EXECUTION foreground-shutdown.dll 1 file(s) copied. 20:26:15.68 Response file: C:hwAC570944wB68409ADeaseservices hreading egressions2164oreground-shutdownoreground-shutdown.dll.rsp C:hwAC570944wB68409ADeaseservices hreading egressions2164oreground-shutdownIL-CG2oreground-shutdown.dll -o:C:hwAC570944wB68409ADeaseservices hreading egressions2164oreground-shutdownoreground-shutdown.dll --targetarch:x64 --targetos:windows --verify-type-and-field-layout --method-layout:random -r:C:hwAC570944pSystem.*.dll -r:C:hwAC570944pMicrosoft.*.dll -r:C:hwAC570944p\xunit.*.dll -r:C:hwAC570944pmscorlib.dll -r:C:hwAC570944p etstandard.dll -O " "dotnet" "C:hwAC570944pcrossgen2crossgen2.dll" @"C:hwAC570944wB68409ADeaseservices hreading egressions2164oreground-shutdownoreground-shutdown.dll.rsp" -r:C:hwAC570944wB68409ADeaseservices hreading hreading_group1IL-CG2*.dll -r:C:hwAC570944wB68409ADeaseservices hreading egressions2164oreground-shutdownIL-CG2*.dll" Emitting R2R PE file: C:hwAC570944wB68409ADeaseservices hreading egressions2164oreground-shutdownoreground-shutdown.dll " "dotnet" "C:hwAC570944p 2rdump 2rdump.dll" --header --sc --in C:hwAC570944wB68409ADeaseservices hreading egressions2164oreground-shutdownoreground-shutdown.dll --out C:hwAC570944wB68409ADeaseservices hreading egressions2164oreground-shutdownoreground-shutdown.dll.r2rdump --val" 20:26:17.71 "C:hwAC570944pcorerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false" foreground-shutdown.dll Xunit.Sdk.EqualException: Assert.Equal() Failure Expected: 100 Actual: 101 at Xunit.Assert.Equal[T](T expected, T actual, IEqualityComparer`1 comparer) in /_/src/xunit.assert/Asserts/EqualityAsserts.cs:line 96 at __GeneratedMainWrapper.Main() Expected: 100 Actual: 101 END EXECUTION - FAILED FAILED Test failed. Trying to see if dump file was created in C:cores since 3/18/2023 8:26:15 PM Test Harness Exitcode is : 1 To run the test: Set up CORE_ROOT and run. > C:hwAC570944wB68409ADeaseservices hreading hreading_group1../regressions/2164/foreground-shutdown/foreground-shutdown.cmd Expected: True Actual: False ``` **Stack trace:** ``` at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/xunit.assert/Asserts/BooleanAsserts.cs:line 132 at Xunit.Assert.True(Boolean condition, String userMessage) in /_/src/xunit.assert/Asserts/BooleanAsserts.cs:line 116 at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String basePath, String assemblyPath) at Program.<

$>g__TestExecutor2|0_1(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&) ``` Queued | Pipeline | Pipeline_Configuration | OS | Arch | Test | Outcome | Properties | BuildReason -- | -- | -- | -- | -- | -- | -- | -- | -- 2022-11-19T20:51:51.061Z | runtime-coreclr ilasm | Checked-ilasmroundtrip | ubuntu.1804.amd64.open.rt | x64 | baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.sh | Failed | {"System.StageName":"Build","DefinitionId":"140","BuildId":"88711","Project":"public","System.JobAttempt":"1","configuration":"Checked-ilasmroundtrip","System.StageAttempt":"1","AzurePipelinesTestRunId":"1867338","CollectionUri":"https://dev.azure.com/dnceng-public/","DefinitionName":"runtime-coreclr ilasm","BuildNumber":"20221119.1","System.PhaseName":"run_test_p1__Linux_x64_checked","operatingSystem":"Ubuntu.1804.Amd64.Open","System.JobName":"__default","System.PhaseAttempt":"1","architecture":"x64","System.JobId":"6dfec640-5b96-5d95-2a5b-7c236ea2d5c5","Reason":"Schedule"} | Schedule 2022-11-19T20:51:17.451Z | runtime-coreclr ilasm | Checked-ilasmroundtrip | ubuntu.1804.armarch.open | arm64 | baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.sh | Failed | {"architecture":"arm64","System.JobId":"34b2f53b-add1-5e74-7a39-5a11483297fc","BuildId":"88711","Project":"public","System.PhaseName":"run_test_p1__Linux_arm64_checked","DefinitionId":"140","System.JobName":"__default","CollectionUri":"https://dev.azure.com/dnceng-public/","AzurePipelinesTestRunId":"1867336","Reason":"Schedule","operatingSystem":"(Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8","System.StageAttempt":"1","System.PhaseAttempt":"1","BuildNumber":"20221119.1","System.StageName":"Build","System.JobAttempt":"1","configuration":"Checked-ilasmroundtrip","DefinitionName":"runtime-coreclr ilasm"} | Schedule 2022-11-19T20:48:25.822Z | runtime-coreclr ilasm | Checked-ilasmroundtrip | ubuntu.1804.armarch.open | arm | baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.sh | Failed | {"BuildNumber":"20221119.1","operatingSystem":"(Ubuntu.1804.Arm32.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7","AzurePipelinesTestRunId":"1867310","DefinitionId":"140","System.StageName":"Build","System.JobId":"860c965f-b28e-50d5-65d2-e40bd59f3a69","architecture":"arm","CollectionUri":"https://dev.azure.com/dnceng-public/","System.JobAttempt":"1","DefinitionName":"runtime-coreclr ilasm","System.PhaseName":"run_test_p1__Linux_arm_checked","BuildId":"88711","System.JobName":"__default","Reason":"Schedule","System.StageAttempt":"1","Project":"public","System.PhaseAttempt":"1","configuration":"Checked-ilasmroundtrip"} | Schedule

Author:	v-wenyuxu
Assignees:	-
Labels:	`area-System.Threading`, `os-windows`, `arch-x64`, `untriaged`, `blocking-outerloop`
Milestone:	-

v-wenyuxu commented 1 year ago

Failed again in: runtime-coreclr r2r-extra 20230326.1

Failed test:

R2R-CG2 windows x64 Checked jitstressregs0x80 @ Windows.10.Amd64.Open

- baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.cmd

R2R-CG2 windows x64 Checked jitstressregs1 @ Windows.10.Amd64.Open

- baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.cmd

R2R-CG2 windows x64 Checked jitstressregs4 @ Windows.10.Amd64.Open

- baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.cmd

R2R-CG2 windows x64 Checked jitstressregs3 @ Windows.10.Amd64.Open

- baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.cmd

R2R-CG2 windows x64 Checked jitstress2 @ Windows.10.Amd64.Open

- baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.cmd

R2R-CG2 windows x64 Checked jitstressregs2 @ Windows.10.Amd64.Open

- baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.cmd

Error message:

Return code:      1
Raw output file:      C:hwA9FF0918wB31A09C1uploads
egressions2164oreground-shutdownoutput.txt
Raw output:
BEGIN EXECUTION
foreground-shutdown.dll
1 file(s) copied.
4:04:36.29
Response file: C:hwA9FF0918wB31A09C1easeservices   hreading
egressions2164oreground-shutdownoreground-shutdown.dll.rsp
C:hwA9FF0918wB31A09C1easeservices  hreading
egressions2164oreground-shutdownIL-CG2oreground-shutdown.dll
-o:C:hwA9FF0918wB31A09C1easeservices   hreading
egressions2164oreground-shutdownoreground-shutdown.dll
--targetarch:x64
--targetos:windows
--verify-type-and-field-layout
--method-layout:random
-r:C:hwA9FF0918pSystem..dll
-r:C:hwA9FF0918pMicrosoft..dll
-r:C:hwA9FF0918p\xunit..dll
-r:C:hwA9FF0918pmscorlib.dll
-r:C:hwA9FF0918p
etstandard.dll
-O
" "dotnet" "C:hwA9FF0918pcrossgen2crossgen2.dll" @"C:hwA9FF0918wB31A09C1easeservices   hreading
egressions2164oreground-shutdownoreground-shutdown.dll.rsp"   -r:C:hwA9FF0918wB31A09C1easeservices   hreading    hreading_group1IL-CG2.dll  -r:C:hwA9FF0918wB31A09C1easeservices    hreading
egressions2164oreground-shutdownIL-CG2*.dll"
Emitting R2R PE file: C:hwA9FF0918wB31A09C1easeservices    hreading
egressions2164oreground-shutdownoreground-shutdown.dll
" "dotnet" "C:hwA9FF0918p
2rdump
2rdump.dll" --header --sc --in C:hwA9FF0918wB31A09C1easeservices   hreading
egressions2164oreground-shutdownoreground-shutdown.dll --out C:hwA9FF0918wB31A09C1easeservices   hreading
egressions2164oreground-shutdownoreground-shutdown.dll.r2rdump --val"
4:04:38.15
"C:hwA9FF0918pcorerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false"  foreground-shutdown.dll
Xunit.Sdk.EqualException: Assert.Equal() Failure
Expected: 100
Actual:   101
at Xunit.Assert.Equal[T](T expected, T actual, IEqualityComparer`1 comparer) in /_/src/xunit.assert/Asserts/EqualityAsserts.cs:line 96
at __GeneratedMainWrapper.Main()
Expected: 100
Actual: 101
END EXECUTION - FAILED
FAILED
Test failed. Trying to see if dump file was created in C:cores since 3/27/2023 4:04:36 AM
Test Harness Exitcode is : 1
To run the test:
Set up CORE_ROOT and run.

C:hwA9FF0918wB31A09C1easeservices  hreading    hreading_group1../regressions/2164/foreground-shutdown/foreground-shutdown.cmd
Expected: True
Actual:   False

Stack trace
   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/xunit.assert/Asserts/BooleanAsserts.cs:line 132
   at Xunit.Assert.True(Boolean condition, String userMessage) in /_/src/xunit.assert/Asserts/BooleanAsserts.cs:line 116
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String basePath, String assemblyPath)
   at Program.<<Main>$>g__TestExecutor2|0_1(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

markples commented 1 year ago

fyi - This test was switched to running in a merged test group around the time of this failure. However, it sets RequiresProcessIsolation (because of the Environment.Exit), so there is very little change in behavior. There is a new autogenerated wrapper method that calls the test method rather than the test method itself being the process entry point. I don't think this is the cause, but it's probably something to keep in mind.

BruceForstall commented 1 year ago

@mangod9

mangod9 commented 1 year ago

Is this an intermittent failure? @kouvel if he has any insights into this test.

JulieLeeMSFT commented 1 year ago

Failed in runtime-coreclr pgostress pipeline Failed test

v-wenyuxu commented 1 year ago

Failed again in: runtime-coreclr jitstress 20230402.1

Failed test:

coreclr windows x64 Checked jitstress1_tiered @ Windows.10.Amd64.Open

- baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.cmd

Error message:

Return code:      1
Raw output file:      C:hwB5D10A03wA87F08D9uploads
egressions2164oreground-shutdownoutput.txt
Raw output:
BEGIN EXECUTION
"C:hwB5D10A03pcorerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false"  foreground-shutdown.dll
Xunit.Sdk.EqualException: Assert.Equal() Failure
Expected: 100
Actual:   101
at Xunit.Assert.Equal[T](T expected, T actual, IEqualityComparer`1 comparer) in //src/xunit.assert/Asserts/EqualityAsserts.cs:line 96
at Xunit.Assert.Equal[T](T expected, T actual) in //src/xunit.assert/Asserts/EqualityAsserts.cs:line 63
at __GeneratedMainWrapper.Main()
Expected: 100
Actual: 101
END EXECUTION - FAILED
FAILED
Test failed. Trying to see if dump file was created in C:cores since 4/3/2023 5:47:01 AM
Test Harness Exitcode is : 1
To run the test:
Set up CORE_ROOT and run.

C:hwB5D10A03wA87F08D9easeservices  hreading    hreading_group1../regressions/2164/foreground-shutdown/foreground-shutdown.cmd
Expected: True
Actual:   False

Stack trace
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String basePath, String assemblyPath)
   at Program.<<Main>$>g__TestExecutor2|0_1(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

BruceForstall commented 1 year ago

Probably dup of https://github.com/dotnet/runtime/issues/84006

v-wenyuxu commented 1 year ago

Failed again in: runtime-coreclr outerloop 20230417.2

Failed test:

R2R-CG2 windows x64 Checked @ Windows.10.Amd64.Open
baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.cmd

Error message:

Return code:      1
Raw output file:      C:hwA6920911wAA1D0918uploads
egressions2164oreground-shutdownoutput.txt
Raw output:
BEGIN EXECUTION
foreground-shutdown.dll
1 file(s) copied.
20:07:42.69
Response file: C:hwA6920911wAA1D0918easeservices   hreading
egressions2164oreground-shutdownoreground-shutdown.dll.rsp
C:hwA6920911wAA1D0918easeservices  hreading
egressions2164oreground-shutdownIL-CG2oreground-shutdown.dll
-o:C:hwA6920911wAA1D0918easeservices   hreading
egressions2164oreground-shutdownoreground-shutdown.dll
--targetarch:x64
--targetos:windows
--verify-type-and-field-layout
--method-layout:random
-r:C:hwA6920911pSystem..dll
-r:C:hwA6920911pMicrosoft..dll
-r:C:hwA6920911p\xunit..dll
-r:C:hwA6920911pmscorlib.dll
-r:C:hwA6920911p
etstandard.dll
-O
" "dotnet" "C:hwA6920911pcrossgen2crossgen2.dll" @"C:hwA6920911wAA1D0918easeservices   hreading
egressions2164oreground-shutdownoreground-shutdown.dll.rsp"   -r:C:hwA6920911wAA1D0918easeservices   hreading    hreading_group1IL-CG2.dll  -r:C:hwA6920911wAA1D0918easeservices    hreading
egressions2164oreground-shutdownIL-CG2*.dll"
Emitting R2R PE file: C:hwA6920911wAA1D0918easeservices    hreading
egressions2164oreground-shutdownoreground-shutdown.dll
" "dotnet" "C:hwA6920911p
2rdump
2rdump.dll" --header --sc --in C:hwA6920911wAA1D0918easeservices   hreading
egressions2164oreground-shutdownoreground-shutdown.dll --out C:hwA6920911wAA1D0918easeservices   hreading
egressions2164oreground-shutdownoreground-shutdown.dll.r2rdump --val"
20:07:44.03
"C:hwA6920911pcorerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false" -p "System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true"  foreground-shutdown.dll
Xunit.Sdk.EqualException: Assert.Equal() Failure
Expected: 100
Actual:   101
at Xunit.Assert.Equal[T](T expected, T actual, IEqualityComparer`1 comparer) in //src/xunit.assert/Asserts/EqualityAsserts.cs:line 96
at Xunit.Assert.Equal[T](T expected, T actual) in //src/xunit.assert/Asserts/EqualityAsserts.cs:line 63
at __GeneratedMainWrapper.Main()
Expected: 100
Actual: 101
END EXECUTION - FAILED
FAILED
Test failed. Trying to see if dump file was created in C:cores since 4/17/2023 8:07:42 PM
Test Harness Exitcode is : 1
To run the test:
Set up CORE_ROOT and run.
> C:hwA6920911wAA1D0918easeservices    hreading    hreading_group1../regressions/2164/foreground-shutdown/foreground-shutdown.cmd
Expected: True
Actual:   False

Stack trace
   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/xunit.assert/Asserts/BooleanAsserts.cs:line 132
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String basePath, String assemblyPath)
   at Program.&lt;&lt;Main&gt;$&gt;g__TestExecutor2|0_1(StreamWriter tempLogSw, StreamWriter statsCsvSw, &lt;&gt;c__DisplayClass0_0&amp;)

v-wenyuxu commented 1 year ago

Failed again in: runtime-coreclr jitstress 20230425.5

Failed test:

coreclr windows x64 Checked jitstress1_tiered @ Windows.10.Amd64.Open

- baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.cmd

Error message:

Return code:      1
Raw output file:      C:hwAE770A0Dw94A60839uploads
egressions2164oreground-shutdownoutput.txt
Raw output:
BEGIN EXECUTION
"C:hwAE770A0Dpcorerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false" -p "System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true"  foreground-shutdown.dll
Xunit.Sdk.EqualException: Assert.Equal() Failure
Expected: 100
Actual:   101
at Xunit.Assert.Equal[T](T expected, T actual, IEqualityComparer`1 comparer) in //src/xunit.assert/Asserts/EqualityAsserts.cs:line 96
at Xunit.Assert.Equal[T](T expected, T actual) in //src/xunit.assert/Asserts/EqualityAsserts.cs:line 63
at __GeneratedMainWrapper.Main()
Expected: 100
Actual: 101
END EXECUTION - FAILED
FAILED
Test failed. Trying to see if dump file was created in C:cores since 4/26/2023 6:01:24 AM
Test Harness Exitcode is : 1
To run the test:
Set up CORE_ROOT and run.
> C:hwAE770A0Dw94A60839easeservices    hreading    hreading_group1../regressions/2164/foreground-shutdown/foreground-shutdown.cmd
Expected: True
Actual:   False

Stack trace
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String basePath, String assemblyPath)
   at Program.&lt;&lt;Main&gt;$&gt;g__TestExecutor2|0_1(StreamWriter tempLogSw, StreamWriter statsCsvSw, &lt;&gt;c__DisplayClass0_0&amp;)

BruceForstall commented 1 year ago

Failed again in outerloop:

https://dev.azure.com/dnceng-public/public/_build/results?buildId=258972&view=ms.vss-test-web.build-test-results-tab&runId=5063396&resultId=116486&paneView=debug

@kouvel @mangod9 Any progress on this? Should it be disabled?

mangod9 commented 1 year ago

yeah probably needs to be disabled. @eduardo-vp since you are running threading/TP tests recently, could you please take a look at this?

v-wenyuxu commented 1 year ago

Failed again in: runtime-coreclr jitstress 20230507.2

Failed tests:

coreclr windows x64 Checked jitstress1_tiered @ Windows.10.Amd64.Open
    - baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.cmd

Error message:


Return code:      1
Raw output file:      C:hwA0B3088AwB11109B5uploads
egressions2164oreground-shutdownoutput.txt
Raw output:
BEGIN EXECUTION
 &quot;C:hwA0B3088Apcorerun.exe&quot; -p &quot;System.Reflection.Metadata.MetadataUpdater.IsSupported=false&quot; -p &quot;System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true&quot;  foreground-shutdown.dll 
Xunit.Sdk.EqualException: Assert.Equal() Failure
Expected: 100
Actual:   101
   at Xunit.Assert.Equal[T](T expected, T actual, IEqualityComparer`1 comparer) in /_/src/xunit.assert/Asserts/EqualityAsserts.cs:line 96
   at Xunit.Assert.Equal[T](T expected, T actual) in /_/src/xunit.assert/Asserts/EqualityAsserts.cs:line 63
   at __GeneratedMainWrapper.Main()
Expected: 100
Actual: 101
END EXECUTION - FAILED
FAILED
Test failed. Trying to see if dump file was created in C:cores since 5/8/2023 5:50:09 AM
Test Harness Exitcode is : 1
To run the test:
Set up CORE_ROOT and run.
&gt; C:hwA0B3088AwB11109B5easeservices hreading    hreading_group1../regressions/2164/foreground-shutdown/foreground-shutdown.cmd
Expected: True
Actual:   False

Stack trace:

    at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String basePath, String assemblyPath)
   at Program.&lt;&lt;Main&gt;$&gt;g__TestExecutor2|0_1(StreamWriter tempLogSw, StreamWriter statsCsvSw, &lt;&gt;c__DisplayClass0_0&amp;)

v-wenyuxu commented 1 year ago

Failed again in: runtime-coreclr jitstress-isas-x86 20230610.1

Failed tests:

coreclr osx x64 Checked jitstress_isas_x86_nosse @ OSX.1200.Amd64.Open
    - baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.cmd

Error message:


Return code:      1
Raw output file:      /tmp/helix/working/B48C09AB/w/A920094F/uploads/regressions/2164/foreground-shutdown/output.txt
Raw output:
BEGIN EXECUTION
/tmp/helix/working/B48C09AB/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true foreground-shutdown.dll ''
Xunit.Sdk.EqualException: Assert.Equal() Failure
Expected: 100
Actual:   101
   at Xunit.Assert.Equal[T](T expected, T actual, IEqualityComparer`1 comparer) in /_/src/xunit.assert/Asserts/EqualityAsserts.cs:line 96
   at __GeneratedMainWrapper.Main()
Expected: 100
Actual: 101
END EXECUTION - FAILED
Test failed. Trying to see if dump file was created in /cores since 6/10/2023 2:57:01 PM
Test Harness Exitcode is : 1
To run the test:
Set up CORE_ROOT and run.
> /private/tmp/helix/working/B48C09AB/w/A920094F/e/baseservices/threading/threading_group1/../regressions/2164/foreground-shutdown/foreground-shutdown.sh
Expected: True
Actual:   False

Stack trace:

    at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/xunit.assert/Asserts/BooleanAsserts.cs:line 132
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)
   at Program.<<Main>$>g__TestExecutor78|0_79(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

v-wenyuxu commented 1 year ago

Failed again in: runtime-coreclr jitstressregs-x86 20230702.1

Failed tests:

coreclr linux x64 Checked jitstressregs2_x86_noavx @ Ubuntu.1804.Amd64.Open
    - baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.cmd

Error message:


Return code:      1
Raw output file:      /datadisks/disk1/work/CBC10A93/w/B4210977/uploads/regressions/2164/foreground-shutdown/output.txt
Raw output:
BEGIN EXECUTION
/datadisks/disk1/work/CBC10A93/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true foreground-shutdown.dll ''
Xunit.Sdk.EqualException: Assert.Equal() Failure
Expected: 100
Actual:   101
   at Xunit.Assert.Equal[T](T expected, T actual, IEqualityComparer`1 comparer) in /_/src/xunit.assert/Asserts/EqualityAsserts.cs:line 96
   at __GeneratedMainWrapper.Main()
Expected: 100
Actual: 101
END EXECUTION - FAILED
Test failed. Trying to see if dump file was created in /home/helixbot/dotnetbuild/dumps since 7/2/2023 8:40:16 PM
Test Harness Exitcode is : 1
To run the test:
Set up CORE_ROOT and run.
> /datadisks/disk1/work/CBC10A93/w/B4210977/e/baseservices/threading/threading_group1/../regressions/2164/foreground-shutdown/foreground-shutdown.sh
Expected: True
Actual:   False

Stack trace:

   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/xunit.assert/Asserts/BooleanAsserts.cs:line 132
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)
   at Program.<<Main>$>g__TestExecutor78|0_79(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

v-wenyuxu commented 1 year ago

Failed again in: runtime-coreclr jitstress2-jitstressregs 20230722.1

Failed tests:

coreclr osx x64 Checked jitstress2_jitstressregs0x10 @ OSX.1200.Amd64.Open
    - baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.cmd

Error message:


Return code:      1
Raw output file:      /tmp/helix/working/9AD10864/w/C0790A7A/uploads/regressions/2164/foreground-shutdown/output.txt
Raw output:
BEGIN EXECUTION
/tmp/helix/working/9AD10864/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true foreground-shutdown.dll ''
Xunit.Sdk.EqualException: Assert.Equal() Failure
Expected: 100
Actual:   101
   at Xunit.Assert.Equal[T](T expected, T actual, IEqualityComparer`1 comparer) in /_/src/xunit.assert/Asserts/EqualityAsserts.cs:line 96
   at __GeneratedMainWrapper.Main()
Expected: 100
Actual: 101
END EXECUTION - FAILED
Test failed. Trying to see if dump file was created in /cores since 7/22/2023 3:12:36 PM
Test Harness Exitcode is : 1
To run the test:
Set up CORE_ROOT and run.
> /private/tmp/helix/working/9AD10864/w/C0790A7A/e/baseservices/threading/threading_group1/../regressions/2164/foreground-shutdown/foreground-shutdown.sh
Expected: True
Actual:   False

Stack trace:

   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/xunit.assert/Asserts/BooleanAsserts.cs:line 132
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)
   at Program.<<Main>$>g__TestExecutor78|0_79(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

v-wenyuxu commented 1 year ago

Failed again in: runtime-coreclr r2r-extra 20230814.1

Failed tests:

R2R-CG2 linux arm Checked gcstress0xc @ (Ubuntu.1804.Arm32.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.cmd

Error message:

/root/helix/work/workitem/e/baseservices/threading/threading_group1/../regressions/2164/foreground-shutdown/foreground-shutdown.sh: line 304: -r:/root/helix/work/workitem/e/baseservices/threading/regressions/2164/foreground-shutdown/IL-CG2/*.dll: No such file or directory

Return code:      1
Raw output file:      /root/helix/work/workitem/uploads/regressions/2164/foreground-shutdown/output.txt
Raw output:
BEGIN EXECUTION
in takeLock
/root/helix/work/workitem/e/baseservices/threading/regressions/2164/foreground-shutdown/IL-CG2/foreground-shutdown.dll
07:59:26
Response file: /root/helix/work/workitem/e/baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.dll.rsp
/root/helix/work/workitem/e/baseservices/threading/regressions/2164/foreground-shutdown/IL-CG2/foreground-shutdown.dll
-o:/root/helix/work/workitem/e/baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.dll
-r:/root/helix/work/correlation/System.*.dll
-r:/root/helix/work/correlation/Microsoft.*.dll
-r:/root/helix/work/correlation/xunit.*.dll
-r:/root/helix/work/correlation/mscorlib.dll
--verify-type-and-field-layout
--method-layout:random
--targetarch:arm
--targetos:linux
Running CrossGen2:  dotnet /root/helix/work/correlation/crossgen2/crossgen2.dll @/root/helix/work/workitem/e/baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.dll.rsp  
Emitting R2R PE file: /root/helix/work/workitem/e/baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.dll
Running R2RDump:  dotnet /root/helix/work/correlation/R2RDump/R2RDump.dll --header --sc --in /root/helix/work/workitem/e/baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.dll --out /root/helix/work/workitem/e/baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.dll.r2rdump --val
07:59:28
in ReleaseLock
in takeLock
/root/helix/work/correlation/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true foreground-shutdown.dll ''
Xunit.Sdk.EqualException: Assert.Equal() Failure
Expected: 100
Actual:   101
   at Xunit.Assert.Equal[T](T expected, T actual, IEqualityComparer`1 comparer) in /_/src/xunit.assert/Asserts/EqualityAsserts.cs:line 96
   at __GeneratedMainWrapper.Main()
in ReleaseLock
Expected: 100
Actual: 101
END EXECUTION - FAILED
Test failed. Trying to see if dump file was created in /home/helixbot/dotnetbuild/dumps since 8/15/2023 7:59:26 AM
Test Harness Exitcode is : 1
To run the test:
Set up CORE_ROOT and run.
> /root/helix/work/workitem/e/baseservices/threading/threading_group1/../regressions/2164/foreground-shutdown/foreground-shutdown.sh
Expected: True
Actual:   False

Stack trace:

   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/xunit.assert/Asserts/BooleanAsserts.cs:line 132
   at Xunit.Assert.True(Boolean condition, String userMessage) in /_/src/xunit.assert/Asserts/BooleanAsserts.cs:line 116
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)
   at Program.<<Main>$>g__TestExecutor78|0_79(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

v-wenyuxu commented 1 year ago

Failed again in: runtime-coreclr pgostress 20230825.1

Failed tests:

coreclr linux arm Checked fullpgo @ (Ubuntu.1804.Arm32.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7
    - baseservices/threading/regressions/2164/foreground-shutdown/foreground-shutdown.cmd

Error message:


Return code:      1
Raw output file:      /root/helix/work/workitem/uploads/regressions/2164/foreground-shutdown/output.txt
Raw output:
BEGIN EXECUTION
/root/helix/work/correlation/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true foreground-shutdown.dll ''
Xunit.Sdk.EqualException: Assert.Equal() Failure
Expected: 100
Actual:   101
   at Xunit.Assert.Equal[T](T expected, T actual, IEqualityComparer`1 comparer) in /_/src/xunit.assert/Asserts/EqualityAsserts.cs:line 96
   at Xunit.Assert.Equal[T](T expected, T actual) in /_/src/xunit.assert/Asserts/EqualityAsserts.cs:line 63
   at __GeneratedMainWrapper.Main()
Expected: 100
Actual: 101
END EXECUTION - FAILED
Test failed. Trying to see if dump file was created in /home/helixbot/dotnetbuild/dumps since 8/26/2023 7:49:05 AM
Test Harness Exitcode is : 1
To run the test:
Set up CORE_ROOT and run.
> /root/helix/work/workitem/e/baseservices/threading/threading_group1/../regressions/2164/foreground-shutdown/foreground-shutdown.sh
Expected: True
Actual:   False

Stack trace:

   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/xunit.assert/Asserts/BooleanAsserts.cs:line 132
   at Xunit.Assert.True(Boolean condition, String userMessage) in /_/src/xunit.assert/Asserts/BooleanAsserts.cs:line 116
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath)
   at Program.<<Main>$>g__TestExecutor78|0_79(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

kouvel commented 1 year ago

It's not clear yet why this is happening. This issue seems to be different from https://github.com/dotnet/runtime/issues/84006, which is a hang and I have a fix for. For this issue I've only seen it occur one time locally in many hours of running the tests. It'll probably need more testing with additional logging in the runtime to get more info when it does happen. For now, I've moved this issue to 9.0 and I'll disable the test.