search

stryker-mutator / stryker-net

Mutation testing for .NET core and .NET framework!
https://stryker-mutator.io
Apache License 2.0
1.78k stars 187 forks source link

No tests found in projects using MSTest.TestAdapter > 2.2.3 #1599

Closed rodri-oliveira-dev closed 2 years ago

rodri-oliveira-dev commented 3 years ago

Describe the bug Erro on execute dotnet stryker, configure in file att

Logs at Stryker.Core.MutationTest.MutationTestProcess.Test(IEnumerable1 mutantsToTest) in D:\a\1\s\src\Stryker.Core\Stryker.Core\MutationTest\MutationTestProcess.cs:line 90 at Stryker.Core.StrykerRunner.RunMutationTest(StrykerOptions options, IEnumerable1 initialLogMessages) in D:\a\1\s\src\Stryker.Core\Stryker.Core\StrykerRunner.cs:line 103 [14:44:06 INF] Time Elapsed 00:05:13.8203633 Unhandled exception. System.AggregateException: One or more errors occurred. (Sequence contains no matching element) ---> System.InvalidOperationException: Sequence contains no matching element at System.Linq.ThrowHelper.ThrowNoMatchException() at System.Linq.Enumerable.First[TSource](IEnumerable1 source, Func2 predicate) at Stryker.Core.TestRunners.VsTest.VsTestRunner.<>cDisplayClass25_0.b8(TestDescription t) in D:\a\1\s\src\Stryker.Core\Stryker.Core\TestRunners\VsTest\VsTestRunner.cs:line 121 at System.Linq.Enumerable.SelectEnumerableIterator2.ToList() at Stryker.Core.TestRunners.VsTest.VsTestRunner.TestMultipleMutants(Nullable1 timeoutMs, IReadOnlyList1 mutants, TestUpdateHandler update) in D:\a\1\s\src\Stryker.Core\Stryker.Core\TestRunners\VsTest\VsTestRunner.cs:line 121 at Stryker.Core.TestRunners.VsTest.VsTestRunnerPool.TestMultipleMutants(Nullable1 timeoutMs, IReadOnlyList1 mutants, TestUpdateHandler update) in D:\a\1\s\src\Stryker.Core\Stryker.Core\TestRunners\VsTest\VsTestRunnerPool.cs:line 51 at Stryker.Core.MutationTest.MutationTestExecutor.RunTestSession(IList1 mutantsToTest, Int32 timeoutMs, TestUpdateHandler updateHandler, Boolean forceSingle) in D:\a\1\s\src\Stryker.Core\Stryker.Core\MutationTest\MutationTestExecutor.cs:line 80 at Stryker.Core.MutationTest.MutationTestExecutor.Test(IList`1 mutantsToTest, Int32 timeoutMs, TestUpdateHandler updateHandler) in D:\a\1\s\src\Stryker.Core\Stryker.Core\MutationTest\MutationTestExecutor.cs:line 35 at Stryker.Core.MutationTest.MutationTestProcess.b14_0(List`1 mutants) in D:\a\1\s\src\Stryker.Core\Stryker.Core\MutationTest\MutationTestProcess.cs:line 124 at System.Threading.Tasks.Parallel.<>cDisplayClass33_02.<ForEachWorker>b__0(Int32 i) at System.Threading.Tasks.Parallel.<>c__DisplayClass19_01.b1(RangeWorker& currentWorker, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source) at System.Threading.Tasks.Parallel.<>cDisplayClass19_01.<ForWorker>b__1(RangeWorker& currentWorker, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion) at System.Threading.Tasks.TaskReplicator.Replica1.ExecuteAction(Boolean& yieldedBeforeCompletion) at System.Threading.Tasks.TaskReplicator.Replica.Execute() --- End of inner exception stack trace --- at System.Threading.Tasks.TaskReplicator.Run[TState](ReplicatableUserAction1 action, ParallelOptions options, Boolean stopOnFirstFailure) at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action1 body, Action2 bodyWithState, Func4 bodyWithLocal, Func1 localInit, Action1 localFinally) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source) at System.Threading.Tasks.Parallel.ThrowSingleCancellationExceptionOrOtherException(ICollection exceptions, CancellationToken cancelToken, Exception otherException) at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action1 body, Action2 bodyWithState, Func4 bodyWithLocal, Func1 localInit, Action1 localFinally) at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IList1 list, ParallelOptions parallelOptions, Action1 body, Action2 bodyWithState, Action3 bodyWithStateAndIndex, Func4 bodyWithStateAndLocal, Func5 bodyWithEverything, Func1 localInit, Action1 localFinally) at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IEnumerable1 source, ParallelOptions parallelOptions, Action1 body, Action2 bodyWithState, Action3 bodyWithStateAndIndex, Func4 bodyWithStateAndLocal, Func5 bodyWithEverything, Func1 localInit, Action1 localFinally) at System.Threading.Tasks.Parallel.ForEach[TSource](IEnumerable1 source, ParallelOptions parallelOptions, Action1 body) at Stryker.Core.MutationTest.MutationTestProcess.TestMutants(IEnumerable1 mutantsToTest) in D:\a\1\s\src\Stryker.Core\Stryker.Core\MutationTest\MutationTestProcess.cs:line 103 at Stryker.Core.MutationTest.MutationTestProcess.Test(IEnumerable1 mutantsToTest) in D:\a\1\s\src\Stryker.Core\Stryker.Core\MutationTest\MutationTestProcess.cs:line 90 at Stryker.Core.StrykerRunner.RunMutationTest(StrykerOptions options, IEnumerable1 initialLogMessages) in D:\a\1\s\src\Stryker.Core\Stryker.Core\StrykerRunner.cs:line 103 at Stryker.CLI.StrykerCLI.RunStryker(StrykerOptions options) in D:\a\1\s\src\Stryker.CLI\Stryker.CLI\StrykerCLI.cs:line 136 at Stryker.CLI.StrykerCLI.<>cDisplayClass7_0.b0() in D:\a\1\s\src\Stryker.CLI\Stryker.CLI\StrykerCLI.cs:line 125 at McMaster.Extensions.CommandLineUtils.CommandLineApplication.<>cDisplayClass146_0.b0(CancellationToken _) at McMaster.Extensions.CommandLineUtils.CommandLineApplication.ExecuteAsync(String[] args, CancellationToken cancellationToken) at McMaster.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args) at Stryker.CLI.StrykerCLI.Run(String[] args) in D:\a\1\s\src\Stryker.CLI\Stryker.CLI\StrykerCLI.cs:line 128 at Stryker.CLI.Program.Main(String[] args) in D:\a\1\s\src\Stryker.CLI\Stryker.CLI\Program.cs:line 16

Expected behavior

Desktop :

Additional context { "stryker-config": { "reporters": [ "progress", "html", "cleartext" ], "log-level": "info", "log-file": true, "timeout-ms": 10000, "test-runner": "vstest", "project-file": "Itau.DW6.RoteirizadorTipoTransacao.Business.csproj", "max-concurrent-test-runners": 1, "threshold-high": 80, "threshold-low": 70, "threshold-break": 60, "mutation-level": "Standard", "git-diff-target": "develop" } }

rouke-broersma commented 3 years ago

Hi @RodrigoDotNet,

Could you send us the console logging as well? Particularly the logging about the discovered tests. Or better yet, could you run stryker with -f and send us the complete logfile from /StrykerOutput/

bcollamore commented 3 years ago

We ran into this exception as well. It's looks like upgrading MSTest.TestAdapter from 2.2.3 to 2.2.4 causes Stryker to erroneously find no tests (reporting success), then upgrading to 2.2.5 causes the Sequence exception (failure). Note: this only affected 1 of our ~70 repos.

rouke-broersma commented 3 years ago

Nice investigative work @bcollamore

So it is caused by upgrading MsTest in the test project? Is this still happening in the latest stryker release where we should have the latest vstest bindings?

bcollamore commented 3 years ago

That's correct @rouke-broersma - upgrading MsTest in the test project triggered it. It was observed on "Version: 0.22.8 (beta)".

Below is the output. Let me know if you would like the Debug level logging!

/ | | | | | \ | | | | | ( | | _ _ _| | ___ _ | | | | | |
_ | | '| | | | |/ / \ '| | . ` | | | |
____) | || | | || | < / | | |\ | |__ | |
|__/ _|| _, ||__|| ()|| \|____| ||
/ |
|/

Version: 0.22.8 (beta)

[08:58:58 ERR] An error occurred during the mutation test run System.AggregateException: One or more errors occurred. (Sequence contains no matching element) (Sequence contains no matching element) ---> System.InvalidOperationException: Sequence contains no matching element at System.Linq.ThrowHelper.ThrowNoMatchException() at System.Linq.Enumerable.First[TSource](IEnumerable1 source, Func2 predicate) at Stryker.Core.TestRunners.VsTest.VsTestRunner.<>cDisplayClass25_0.b8(TestDescription t) at System.Linq.Enumerable.SelectEnumerableIterator2.ToList() at System.Linq.Enumerable.ToList[TSource](IEnumerable1 source) at Stryker.Core.TestRunners.VsTest.VsTestRunner.TestMultipleMutants(Nullable1 timeoutMs, IReadOnlyList1 mutants, TestUpdateHandler update) at Stryker.Core.TestRunners.VsTest.VsTestRunnerPool.TestMultipleMutants(Nullable1 timeoutMs, IReadOnlyList1 mutants, TestUpdateHandler update) at Stryker.Core.MutationTest.MutationTestExecutor.RunTestSession(IList1 mutantsToTest, Int32 timeoutMs, TestUpdateHandler updateHandler, Boolean forceSingle) at Stryker.Core.MutationTest.MutationTestExecutor.Test(IList1 mutantsToTest, Int32 timeoutMs, TestUpdateHandler updateHandler) at Stryker.Core.MutationTest.MutationTestProcess.b14_0(List`1 mutants) at System.Threading.Tasks.Parallel.<>cDisplayClass33_02.<ForEachWorker>b__0(Int32 i) at System.Threading.Tasks.Parallel.<>c__DisplayClass19_01.b1(RangeWorker& currentWorker, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source) at System.Threading.Tasks.Parallel.<>cDisplayClass19_01.<ForWorker>b__1(RangeWorker& currentWorker, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion) at System.Threading.Tasks.TaskReplicator.Replica1.ExecuteAction(Boolean& yieldedBeforeCompletion) at System.Threading.Tasks.TaskReplicator.Replica.Execute() --- End of inner exception stack trace --- at System.Threading.Tasks.TaskReplicator.Run[TState](ReplicatableUserAction1 action, ParallelOptions options, Boolean stopOnFirstFailure) at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action1 body, Action2 bodyWithState, Func4 bodyWithLocal, Func1 localInit, Action1 localFinally) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source) at System.Threading.Tasks.Parallel.ThrowSingleCancellationExceptionOrOtherException(ICollection exceptions, CancellationToken cancelToken, Exception otherException) at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action1 body, Action2 bodyWithState, Func4 bodyWithLocal, Func1 localInit, Action1 localFinally) at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IList1 list, ParallelOptions parallelOptions, Action1 body, Action2 bodyWithState, Action3 bodyWithStateAndIndex, Func4 bodyWithStateAndLocal, Func5 bodyWithEverything, Func1 localInit, Action1 localFinally) at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IEnumerable1 source, ParallelOptions parallelOptions, Action1 body, Action2 bodyWithState, Action3 bodyWithStateAndIndex, Func4 bodyWithStateAndLocal, Func5 bodyWithEverything, Func1 localInit, Action1 localFinally) at System.Threading.Tasks.Parallel.ForEach[TSource](IEnumerable1 source, ParallelOptions parallelOptions, Action1 body) at Stryker.Core.MutationTest.MutationTestProcess.TestMutants(IEnumerable1 mutantsToTest) at Stryker.Core.MutationTest.MutationTestProcess.Test(IEnumerable1 mutantsToTest) at Stryker.Core.StrykerRunner.RunMutationTest(StrykerOptions options, IEnumerable1 initialLogMessages) ---> (Inner Exception #1) System.InvalidOperationException: Sequence contains no matching element at System.Linq.ThrowHelper.ThrowNoMatchException() at System.Linq.Enumerable.First[TSource](IEnumerable1 source, Func2 predicate) at Stryker.Core.TestRunners.VsTest.VsTestRunner.<>cDisplayClass25_0.b8(TestDescription t) at System.Linq.Enumerable.SelectEnumerableIterator2.ToList() at System.Linq.Enumerable.ToList[TSource](IEnumerable1 source) at Stryker.Core.TestRunners.VsTest.VsTestRunner.TestMultipleMutants(Nullable1 timeoutMs, IReadOnlyList1 mutants, TestUpdateHandler update) at Stryker.Core.TestRunners.VsTest.VsTestRunnerPool.TestMultipleMutants(Nullable1 timeoutMs, IReadOnlyList1 mutants, TestUpdateHandler update) at Stryker.Core.MutationTest.MutationTestExecutor.RunTestSession(IList1 mutantsToTest, Int32 timeoutMs, TestUpdateHandler updateHandler, Boolean forceSingle) at Stryker.Core.MutationTest.MutationTestExecutor.Test(IList1 mutantsToTest, Int32 timeoutMs, TestUpdateHandler updateHandler) at Stryker.Core.MutationTest.MutationTestProcess.b14_0(List`1 mutants) at System.Threading.Tasks.Parallel.<>cDisplayClass33_02.<ForEachWorker>b__0(Int32 i) at System.Threading.Tasks.Parallel.<>c__DisplayClass19_01.b1(RangeWorker& currentWorker, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source) at System.Threading.Tasks.Parallel.<>cDisplayClass19_01.<ForWorker>b__1(RangeWorker& currentWorker, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion) at System.Threading.Tasks.TaskReplicator.Replica1.ExecuteAction(Boolean& yieldedBeforeCompletion) at System.Threading.Tasks.TaskReplicator.Replica.Execute()<--- Unhandled exception. System.AggregateException: One or more errors occurred. (Sequence contains no matching element) (Sequence contains no matching element) ---> System.InvalidOperationException: Sequence contains no matching element at System.Linq.ThrowHelper.ThrowNoMatchException() at System.Linq.Enumerable.First[TSource](IEnumerable1 source, Func2 predicate) at Stryker.Core.TestRunners.VsTest.VsTestRunner.<>cDisplayClass25_0.b8(TestDescription t) at System.Linq.Enumerable.SelectEnumerableIterator2.ToList() at System.Linq.Enumerable.ToList[TSource](IEnumerable1 source) at Stryker.Core.TestRunners.VsTest.VsTestRunner.TestMultipleMutants(Nullable1 timeoutMs, IReadOnlyList1 mutants, TestUpdateHandler update) at Stryker.Core.TestRunners.VsTest.VsTestRunnerPool.TestMultipleMutants(Nullable1 timeoutMs, IReadOnlyList1 mutants, TestUpdateHandler update) at Stryker.Core.MutationTest.MutationTestExecutor.RunTestSession(IList1 mutantsToTest, Int32 timeoutMs, TestUpdateHandler updateHandler, Boolean forceSingle) at Stryker.Core.MutationTest.MutationTestExecutor.Test(IList1 mutantsToTest, Int32 timeoutMs, TestUpdateHandler updateHandler) at Stryker.Core.MutationTest.MutationTestProcess.b14_0(List`1 mutants) at System.Threading.Tasks.Parallel.<>cDisplayClass33_02.<ForEachWorker>b__0(Int32 i) at System.Threading.Tasks.Parallel.<>c__DisplayClass19_01.b1(RangeWorker& currentWorker, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source) at System.Threading.Tasks.Parallel.<>cDisplayClass19_01.<ForWorker>b__1(RangeWorker& currentWorker, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion) at System.Threading.Tasks.TaskReplicator.Replica1.ExecuteAction(Boolean& yieldedBeforeCompletion) at System.Threading.Tasks.TaskReplicator.Replica.Execute() --- End of inner exception stack trace --- at System.Threading.Tasks.TaskReplicator.Run[TState](ReplicatableUserAction1 action, ParallelOptions options, Boolean stopOnFirstFailure) at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action1 body, Action2 bodyWithState, Func4 bodyWithLocal, Func1 localInit, Action1 localFinally) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source) at System.Threading.Tasks.Parallel.ThrowSingleCancellationExceptionOrOtherException(ICollection exceptions, CancellationToken cancelToken, Exception otherException) at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int32 toExclusive, ParallelOptions parallelOptions, Action1 body, Action2 bodyWithState, Func4 bodyWithLocal, Func1 localInit, Action1 localFinally) at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IList1 list, ParallelOptions parallelOptions, Action1 body, Action2 bodyWithState, Action3 bodyWithStateAndIndex, Func4 bodyWithStateAndLocal, Func5 bodyWithEverything, Func1 localInit, Action1 localFinally) at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IEnumerable1 source, ParallelOptions parallelOptions, Action1 body, Action2 bodyWithState, Action3 bodyWithStateAndIndex, Func4 bodyWithStateAndLocal, Func5 bodyWithEverything, Func1 localInit, Action1 localFinally) at System.Threading.Tasks.Parallel.ForEach[TSource](IEnumerable1 source, ParallelOptions parallelOptions, Action1 body) at Stryker.Core.MutationTest.MutationTestProcess.TestMutants(IEnumerable1 mutantsToTest) at Stryker.Core.MutationTest.MutationTestProcess.Test(IEnumerable1 mutantsToTest) at Stryker.Core.StrykerRunner.RunMutationTest(StrykerOptions options, IEnumerable1 initialLogMessages) at Stryker.CLI.StrykerCLI.RunStryker(StrykerOptions options) in D:\a\1\s\src\Stryker.CLI\Stryker.CLI\StrykerCLI.cs:line 136 at Stryker.CLI.StrykerCLI.<>cDisplayClass7_0.b0() in D:\a\1\s\src\Stryker.CLI\Stryker.CLI\StrykerCLI.cs:line 125 at McMaster.Extensions.CommandLineUtils.CommandLineApplication.<>cDisplayClass146_0.b0(CancellationToken _) at McMaster.Extensions.CommandLineUtils.CommandLineApplication.ExecuteAsync(String[] args, CancellationToken cancellationToken) at McMaster.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args) at Stryker.CLI.StrykerCLI.Run(String[] args) in D:\a\1\s\src\Stryker.CLI\Stryker.CLI\StrykerCLI.cs:line 128 at Stryker.CLI.Program.Main(String[] args) in D:\a\1\s\src\Stryker.CLI\Stryker.CLI\Program.cs:line 16 ---> (Inner Exception #1) System.InvalidOperationException: Sequence contains no matching element at System.Linq.ThrowHelper.ThrowNoMatchException() at System.Linq.Enumerable.First[TSource](IEnumerable1 source, Func2 predicate) at Stryker.Core.TestRunners.VsTest.VsTestRunner.<>cDisplayClass25_0.b8(TestDescription t) at System.Linq.Enumerable.SelectEnumerableIterator2.ToList() at System.Linq.Enumerable.ToList[TSource](IEnumerable1 source) at Stryker.Core.TestRunners.VsTest.VsTestRunner.TestMultipleMutants(Nullable1 timeoutMs, IReadOnlyList1 mutants, TestUpdateHandler update) at Stryker.Core.TestRunners.VsTest.VsTestRunnerPool.TestMultipleMutants(Nullable1 timeoutMs, IReadOnlyList1 mutants, TestUpdateHandler update) at Stryker.Core.MutationTest.MutationTestExecutor.RunTestSession(IList1 mutantsToTest, Int32 timeoutMs, TestUpdateHandler updateHandler, Boolean forceSingle) at Stryker.Core.MutationTest.MutationTestExecutor.Test(IList1 mutantsToTest, Int32 timeoutMs, TestUpdateHandler updateHandler) at Stryker.Core.MutationTest.MutationTestProcess.b14_0(List`1 mutants) at System.Threading.Tasks.Parallel.<>cDisplayClass33_02.<ForEachWorker>b__0(Int32 i) at System.Threading.Tasks.Parallel.<>c__DisplayClass19_01.b1(RangeWorker& currentWorker, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion) --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw(Exception source) at System.Threading.Tasks.Parallel.<>cDisplayClass19_01.<ForWorker>b__1(RangeWorker& currentWorker, Int32 timeout, Boolean& replicationDelegateYieldedBeforeCompletion) at System.Threading.Tasks.TaskReplicator.Replica1.ExecuteAction(Boolean& yieldedBeforeCompletion) at System.Threading.Tasks.TaskReplicator.Replica.Execute()<---

rouke-broersma commented 3 years ago

Hi @bcollamore I don't think we will learn more from debug logging. This sounds like an integration issue between the vstest client, vstest and mstest. I think we will have to hope to find a reproduction project and report this issue to microsoft for a fix.

rouke-broersma commented 2 years ago

In the next release we will ship a new version of vstest that might fix this issue. Please let us know if that is the case.

dupdob commented 2 years ago

Closed as it is likely to be fixed