False positive test Interop/Interop/Interop.dll

[t-mustafin]

Test Interop/Interop/Interop.dll returns 100 even if problems exist. ./Interop/Interop/Interop.dll was implemented by https://github.com/dotnet/runtime/pull/94109. @jkoritzinsky, please take a look.

Test suite for this launch was built with BuildAsStandalone=true environment variable.

~/main_23may_7284f17/coreclr-tc$ ./coreroot/corerun ./Interop/Interop/Interop.dll 
12:25:42.155 Running test: global::WinRT.Program.CannotLoadWinRTType()
12:25:42.689 Passed test: global::WinRT.Program.CannotLoadWinRTType()
12:25:42.795 Running test: global::Program.TestEntryPoint()
Unexpected Size for DefaultLayoutDefaultPacking`1[System.Runtime.Intrinsics.Vector256`1[System.Byte]].
     Expected: 64; Actual: 48
Unexpected Offset for DefaultLayoutDefaultPacking`1[System.Runtime.Intrinsics.Vector256`1[System.Byte]].Value.
     Expected: 32; Actual: 16

Unexpected Size for SequentialLayoutDefaultPacking`1[System.Runtime.Intrinsics.Vector256`1[System.Byte]].
     Expected: 64; Actual: 48
Unexpected Offset for SequentialLayoutDefaultPacking`1[System.Runtime.Intrinsics.Vector256`1[System.Byte]].Value.
     Expected: 32; Actual: 16

Unexpected Size for SequentialLayoutMaxPacking`1[System.Runtime.Intrinsics.Vector256`1[System.Byte]].
     Expected: 64; Actual: 48
Unexpected Offset for SequentialLayoutMaxPacking`1[System.Runtime.Intrinsics.Vector256`1[System.Byte]].Value.
     Expected: 32; Actual: 16

Unexpected Size for AutoLayoutDefaultPacking`1[System.Runtime.Intrinsics.Vector256`1[System.Byte]].
     Expected: 64; Actual: 48
Unexpected Offset for AutoLayoutDefaultPacking`1[System.Runtime.Intrinsics.Vector256`1[System.Byte]].Value.
     Expected: 32; Actual: 16

Unexpected Size for AutoLayoutMinPacking`1[System.Runtime.Intrinsics.Vector256`1[System.Byte]].
     Expected: 64; Actual: 48
Unexpected Offset for AutoLayoutMinPacking`1[System.Runtime.Intrinsics.Vector256`1[System.Byte]].Value.
     Expected: 32; Actual: 16

Unexpected Size for AutoLayoutMaxPacking`1[System.Runtime.Intrinsics.Vector256`1[System.Byte]].
     Expected: 64; Actual: 48
Unexpected Offset for AutoLayoutMaxPacking`1[System.Runtime.Intrinsics.Vector256`1[System.Byte]].Value.
     Expected: 32; Actual: 16

Xunit.Sdk.EqualException: Assert.Equal() Failure: Values differ
Expected: 100
Actual:   0
   at Xunit.Assert.Equal[T](T expected, T actual, IEqualityComparer`1 comparer) in /_/src/Microsoft.DotNet.XUnitAssert/src/EqualityAsserts.cs:line 168
   at Xunit.Assert.Equal[T](T expected, T actual) in /_/src/Microsoft.DotNet.XUnitAssert/src/EqualityAsserts.cs:line 90
   at Program.<<Main>$>g__TestExecutor2|25_3(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass25_0&)
12:25:43.972 Failed test: global::Program.TestEntryPoint()

... <cut big log here> ...

12:25:51.811 Running test: Interop/UnmanagedCallersOnly/UnmanagedCallersOnlyTest/UnmanagedCallersOnlyTest.cmd
/home/tmustafin/main_23may_7284f17/coreclr-tc/Interop/Interop/../UnmanagedCallersOnly/UnmanagedCallersOnlyTest/UnmanagedCallersOnlyTest.sh: line 350: /corerun: No such file or directory

Return code:      1
Raw output file:      /home/tmustafin/main_23may_7284f17/UnmanagedCallersOnly/UnmanagedCallersOnlyTest/output.txt
Raw output:
BEGIN EXECUTION
/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true UnmanagedCallersOnlyTest.dll ''
Expected: 100
Actual: 127
END EXECUTION - FAILED
Test Harness Exitcode is : 1
To run the test:
Set up CORE_ROOT and run.
> /home/tmustafin/main_23may_7284f17/coreclr-tc/Interop/Interop/../UnmanagedCallersOnly/UnmanagedCallersOnlyTest/UnmanagedCallersOnlyTest.sh
Xunit.Sdk.TrueException: /home/tmustafin/main_23may_7284f17/coreclr-tc/Interop/Interop/../UnmanagedCallersOnly/UnmanagedCallersOnlyTest/UnmanagedCallersOnlyTest.sh: line 350: /corerun: No such file or directory

Return code:      1
Raw output file:      /home/tmustafin/main_23may_7284f17/UnmanagedCallersOnly/UnmanagedCallersOnlyTest/output.txt
Raw output:
BEGIN EXECUTION
/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true UnmanagedCallersOnlyTest.dll ''
Expected: 100
Actual: 127
END EXECUTION - FAILED
Test Harness Exitcode is : 1
To run the test:
Set up CORE_ROOT and run.
> /home/tmustafin/main_23may_7284f17/coreclr-tc/Interop/Interop/../UnmanagedCallersOnly/UnmanagedCallersOnlyTest/UnmanagedCallersOnlyTest.sh
   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 146
   at Xunit.Assert.True(Boolean condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 128
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath, String testPathPrefix)
   at Program.<<Main>$>g__TestExecutor225|25_226(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass25_0&)
12:25:51.857 Failed test: Interop/UnmanagedCallersOnly/UnmanagedCallersOnlyTest/UnmanagedCallersOnlyTest.cmd
12:25:51.865 Running test: Interop/UnmanagedCallersOnlyBasic/UnmanagedCallersOnlyBasicTest/UnmanagedCallersOnlyBasicTest.cmd
/home/tmustafin/main_23may_7284f17/coreclr-tc/Interop/Interop/../UnmanagedCallersOnlyBasic/UnmanagedCallersOnlyBasicTest/UnmanagedCallersOnlyBasicTest.sh: line 350: /corerun: No such file or directory

Return code:      1
Raw output file:      /home/tmustafin/main_23may_7284f17/UnmanagedCallersOnlyBasic/UnmanagedCallersOnlyBasicTest/output.txt
Raw output:
BEGIN EXECUTION
/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true UnmanagedCallersOnlyBasicTest.dll ''
Expected: 100
Actual: 127
END EXECUTION - FAILED
Test Harness Exitcode is : 1
To run the test:
Set up CORE_ROOT and run.
> /home/tmustafin/main_23may_7284f17/coreclr-tc/Interop/Interop/../UnmanagedCallersOnlyBasic/UnmanagedCallersOnlyBasicTest/UnmanagedCallersOnlyBasicTest.sh
Xunit.Sdk.TrueException: /home/tmustafin/main_23may_7284f17/coreclr-tc/Interop/Interop/../UnmanagedCallersOnlyBasic/UnmanagedCallersOnlyBasicTest/UnmanagedCallersOnlyBasicTest.sh: line 350: /corerun: No such file or directory

Return code:      1
Raw output file:      /home/tmustafin/main_23may_7284f17/UnmanagedCallersOnlyBasic/UnmanagedCallersOnlyBasicTest/output.txt
Raw output:
BEGIN EXECUTION
/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true UnmanagedCallersOnlyBasicTest.dll ''
Expected: 100
Actual: 127
END EXECUTION - FAILED
Test Harness Exitcode is : 1
To run the test:
Set up CORE_ROOT and run.
> /home/tmustafin/main_23may_7284f17/coreclr-tc/Interop/Interop/../UnmanagedCallersOnlyBasic/UnmanagedCallersOnlyBasicTest/UnmanagedCallersOnlyBasicTest.sh
   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 146
   at Xunit.Assert.True(Boolean condition, String userMessage) in /_/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 128
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath, String testPathPrefix)
   at Program.<<Main>$>g__TestExecutor226|25_227(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass25_0&)
12:25:51.913 Failed test: Interop/UnmanagedCallersOnlyBasic/UnmanagedCallersOnlyBasicTest/UnmanagedCallersOnlyBasicTest.cmd
~/main_23may_7284f17/coreclr-tc$ echo $?
100

Full Interop.dll launch log: Interop_false_positive_log.txt

Runtime and tests state: main branch, 23may 7284f17efe5f1ad7382f65ef3276ac89d594d3d3

cc @gbalykov @clamp03

[jkoritzinsky]

Please don't use the BuildAsStandalone=true environment variable nowadays. That's no longer a supported testing scenario.

Please set BuildAllTestsAsStandalone=true environment variable instead to get into a supported scenario.

[clamp03]

CC @dotnet/samsung

[t-mustafin]

Please don't use the BuildAsStandalone=true environment variable nowadays. That's no longer a supported testing scenario.

Please set BuildAllTestsAsStandalone=true environment variable instead to get into a supported scenario.

Thanks, with updated BuildAllTestsAsStandalone=true test behavior is the same, returns 100 even if some tests fails.

[jkoritzinsky]

It looks like even with BuildAllTestsAsStandalone=true, it's still building using the "merged runner" model that emits a test results file instead of standalone. Not sure why (might need to delete the test bin/obj artifacts when switching between these settings?).

For the merged runner model, it's expected that the runner will return "success", as returning otherwise indicates a failure in the test harness itself, not in a test, in our distributed testing system.

Can you try doing a clean rebuild of the tests and see if this reproduces?

[t-mustafin]

Can you try doing a clean rebuild of the tests and see if this reproduces?

The check I made above included rm -rf artifacts before rebuild tests.

For the merged runner model, it's expected that the runner will return "success", as returning otherwise indicates a failure in the test harness itself, not in a test, in our distributed testing system.

It means that test launcher need to parse log of merged Interop.dll instead of simple check of return value.