Open RikkiGibson opened 4 years ago
This is not my experience. For example, this is what I get if I put a Debug.Fail("Failed");
in a test and run it via run tests in context:
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.1-pre.build.4059 (64-bit .NET 5.0.0-preview.8.20407.11)
[xUnit.net 00:00:00.69] Discovering: Microsoft.CodeAnalysis.CSharp.Symbol.UnitTests
[xUnit.net 00:00:01.64] Discovered: Microsoft.CodeAnalysis.CSharp.Symbol.UnitTests
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.1-pre.build.4059 (64-bit .NET 5.0.0-preview.8.20407.11)
[xUnit.net 00:00:00.73] Starting: Microsoft.CodeAnalysis.CSharp.Symbol.UnitTests
The active test run was aborted. Reason: Test host process crashed : Process terminated. Assertion failed.
Failed
at Microsoft.CodeAnalysis.CSharp.UnitTests.CompilationErrorTests.ParameterAndReturnTypesAreStaticClassesWarning() in /home/fred/git/roslyn/src/Compilers/CSharp/Test/Symbol/Symbols/SymbolErrorTests.cs:line 13377
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
at Xunit.Sdk.TestInvoker`1.CallTestMethod(Object testClassInstance) in C:\projects\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs:line 150
at Xunit.Sdk.TestInvoker`1.<>c__DisplayClass48_1.<<InvokeTestMethodAsync>b__1>d.MoveNext() in C:\projects\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs:line 257
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Xunit.Sdk.TestInvoker`1.<>c__DisplayClass48_1.<InvokeTestMethodAsync>b__1()
at Xunit.Sdk.ExecutionTimer.AggregateAsync(Func`1 asyncAction) in C:\projects\xunit\src\xunit.execution\Sdk\Frameworks\ExecutionTimer.cs:line 48
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Xunit.Sdk.ExecutionTimer.AggregateAsync(Func`1 asyncAction)
at Xunit.Sdk.TestInvoker`1.<>c__DisplayClass48_1.<InvokeTestMethodAsync>b__0() in C:\projects\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs:line 242
at Xunit.Sdk.ExceptionAggregator.RunAsync(Func`1 code) in C:\projects\xunit\src\xunit.core\Sdk\ExceptionAggregator.cs:line 90
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Xunit.Sdk.ExceptionAggregator.RunAsync(Func`1 code)
at Xunit.Sdk.TestInvoker`1.InvokeTestMethodAsync(Object testClassInstance) in C:\projects\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs:line 241
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Xunit.Sdk.TestInvoker`1.InvokeTestMethodAsync(Object testClassInstance)
at Xunit.Sdk.XunitTestInvoker.InvokeTestMethodAsync(Object testClassInstance) in C:\projects\xunit\src\xunit.execution\Sdk\Frameworks\Runners\XunitTestInvoker.cs:line 112
at Xunit.Sdk.TestInvoker`1.<RunAsync>b__47_0() in C:\projects\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs:line 206
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Xunit.Sdk.TestInvoker`1.<RunAsync>b__47_0()
at Xunit.Sdk.ExceptionAggregator.RunAsync[T](Func`1 code) in C:\projects\xunit\src\xunit.core\Sdk\ExceptionAggregator.cs:line 107
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Xunit.Sdk.ExceptionAggregator.RunAsync[T](Func`1 code)
at Xunit.Sdk.TestInvoker`1.RunAsync() in C:\projects\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs:line 189
at Xunit.Sdk.XunitTestRunner.InvokeTestMethodAsync(ExceptionAggregator aggregator) in C:\projects\xunit\src\xunit.execution\Sdk\Frameworks\Runners\XunitTestRunner.cs:line 84
at Xunit.Sdk.XunitTestRunner.InvokeTestAsync(ExceptionAggregator aggregator) in C:\projects\xunit\src\xunit.execution\Sdk\Frameworks\Runners\XunitTestRunner.cs:line 67
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Xunit.Sdk.XunitTestRunner.InvokeTestAsync(ExceptionAggregator aggregator)
at Xunit.Sdk.TestRunner`1.<>c__DisplayClass43_0.<RunAsync>b__0() in C:\projects\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestRunner.cs:line 149
at Xunit.Sdk.ExceptionAggregator.RunAsync[T](Func`1 code) in C:\projects\xunit\src\xunit.core\Sdk\ExceptionAggregator.cs:line 107
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Xunit.Sdk.ExceptionAggregator.RunAsync[T](Func`1 code)
at Xunit.Sdk.TestRunner`1.RunAsync() in C:\projects\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestRunner.cs:line 149
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Xunit.Sdk.TestRunner`1.RunAsync()
at Xunit.Sdk.XunitTestCaseRunner.RunTestAsync() in C:\projects\xunit\src\xunit.execution\Sdk\Frameworks\Runners\XunitTestCaseRunner.cs:line 139
at Xunit.Sdk.TestCaseRunner`1.RunAsync() in C:\projects\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestCaseRunner.cs:line 82
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Xunit.Sdk.TestCaseRunner`1.RunAsync()
at Xunit.Sdk.XunitTestCase.RunAsync(IMessageSink diagnosticMessageSink, IMessageBus messageBus, Object[] constructorArguments, ExceptionAggregator aggregator, CancellationTokenSource cancellationTokenSource) in C:\projects\xunit\src\xunit.execution\Sdk\Frameworks\XunitTestCase.cs:line 162
at Xunit.Sdk.XunitTestMethodRunner.RunTestCaseAsync(IXunitTestCase testCase) in C:\projects\xunit\src\xunit.execution\Sdk\Frameworks\Runners\XunitTestMethodRunner.cs:line 45
at Xunit.Sdk.TestMethodRunner`1.RunTestCasesAsync() in C:\projects\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestMethodRunner.cs:line 136
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Xunit.Sdk.TestMethodRunner`1.RunTestCasesAsync()
at Xunit.Sdk.TestMethodRunner`1.RunAsync() in C:\projects\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestMethodRunner.cs:line 106
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Xunit.Sdk.TestMethodRunner`1.RunAsync()
at Xunit.Sdk.XunitTestClassRunner.RunTestMethodAsync(ITestMethod testMethod, IReflectionMethodInfo method, IEnumerable`1 testCases, Object[] constructorArguments) in C:\projects\xunit\src\xunit.execution\Sdk\Frameworks\Runners\XunitTestClassRunner.cs:line 168
at Xunit.Sdk.TestClassRunner`1.RunTestMethodsAsync() in C:\projects\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestClassRunner.cs:line 213
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Xunit.Sdk.TestClassRunner`1.RunTestMethodsAsync()
at Xunit.Sdk.TestClassRunner`1.RunAsync() in C:\projects\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestClassRunner.cs:line 171
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Xunit.Sdk.TestClassRunner`1.RunAsync()
at Xunit.Sdk.XunitTestCollectionRunner.RunTestClassAsync(ITestClass testClass, IReflectionTypeInfo class, IEnumerable`1 testCases) in C:\projects\xunit\src\xunit.execution\Sdk\Frameworks\Runners\XunitTestCollectionRunner.cs:line 158
at Xunit.Sdk.TestCollectionRunner`1.RunTestClassesAsync() in C:\projects\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestCollectionRunner.cs:line 130
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Xunit.Sdk.TestCollectionRunner`1.RunTestClassesAsync()
at Xunit.Sdk.TestCollectionRunner`1.RunAsync() in C:\projects\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestCollectionRunner.cs:line 101
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at Xunit.Sdk.TestCollectionRunner`1.RunAsync()
at Xunit.Sdk.XunitTestAssemblyRunner.RunTestCollectionAsync(IMessageBus messageBus, ITestCollection testCollection, IEnumerable`1 testCases, CancellationTokenSource cancellationTokenSource) in C:\projects\xunit\src\xunit.execution\Sdk\Frameworks\Runners\XunitTestAssemblyRunner.cs:line 235
at Xunit.Sdk.TestAssemblyRunner`1.RunTestCollectionsAsync(IMessageBus messageBus, CancellationTokenSource cancellationTokenSource) in C:\projects\xunit
----- Test Execution Summary -----
Total tests: 0. Passed: 0. Failed: 0. Skipped: 0
If that's not what you're seeing, I would suspect that this is something to do with the test runner itself, not omnisharp or the particular command in question.
I see now that I was just drowning in output and didn't spot the exception:
(there are tons of entries for UnconstrainedTypeParameter_06, but I think it's expected due to CombinatorialValuesAttributes.)
It feels like maybe the test host crashing on a test should be reported as a test failure. Where does that bug belong?
Probably something in here: https://github.com/OmniSharp/omnisharp-roslyn/blob/master/src/OmniSharp.DotNetTest/VSTestManager.cs#L396-L427
If you run tests via the Run Tests in Context command, and the test runner crashes, for example by failing a Debug.Assert, it just reports the result for the subset of the tests that finished up to that point, and doesn't let you know that an assert failed.
I picked up on this when I saw "all the Roslyn NullableReferenceTypesTests passed" in my output window, and it showed about 300 successes instead of about 3000. 😉 @333fred