search

stryker-mutator / stryker-net

Mutation testing for .NET core and .NET framework!
https://stryker-mutator.io
Apache License 2.0
1.78k stars 188 forks source link

Error during test execution #1724

Closed BrunoRM closed 3 years ago

BrunoRM commented 3 years ago

Describe the bug I'm trying to run mutation tests in my local environment and in Azure Pipelines, both throws an exception and exits with error. The exception description is:

An error occurred during the mutation test run
System.AggregateException: One or more errors occurred. (Sequence contains no matching element)
 ---> System.InvalidOperationException: Sequence contains no matching element
   at System.Linq.ThrowHelper.ThrowNoMatchException()
   at System.Linq.Enumerable.First[TSource](IEnumerable1 source, Func2 predicate)
   at Stryker.Core.TestRunners.VsTest.VsTestRunner.<>cDisplayClass25_0.<TestMultipleMutants>b8(TestDescription t) in D:\a\1\s\src\Stryker.Core\Stryker.Core\TestRunners\VsTest\VsTestRunner.cs:line 121
   at System.Linq.Enumerable.SelectEnumerableIterator2.ToList()
   at Stryker.Core.TestRunners.VsTest.VsTestRunner.TestMultipleMutants(Nullable1 timeoutMs, IReadOnlyList1 mutants, TestUpdateHandler update) in D:\a\1\s\src\Stryker.Core\Stryker.Core\TestRunners\VsTest\VsTestRunner.cs:line 121
   at Stryker.Core.TestRunners.VsTest.VsTestRunnerPool.TestMultipleMutants(Nullable1 timeoutMs, IReadOnlyList1 mutants, TestUpdateHandler update) in D:\a\1\s\src\Stryker.Core\Stryker.Core\TestRunners\VsTest\VsTestRunnerPool.cs:line 51
   at Stryker.Core.MutationTest.MutationTestExecutor.RunTestSession(IList1 mutantsToTest, Int32 timeoutMs, TestUpdateHandler updateHandler, Boolean forceSingle) in D:\a\1\s\src\Stryker.Core\Stryker.Core\MutationTest\MutationTestExecutor.cs:line 80

Doing a checkout to previous commit before the feature I implemented, tests executes successfuly, with my changes, it starts to break.

Logs VsTest-log.txt log-20211027.txt

Desktop (please complete the following information):

Additional context I upgraded Stryker to version 1.0.0-rc3 in my local machine and it works. But i can't do this in the pipeline now.

Another logs that I see on start of the run is:

[16:55:08 WRN] Analyzer assembly C:\Program Files\dotnet\sdk\5.0.301\Sdks\Microsoft.NET.Sdk\targets..\analyzers\Microsoft.CodeAnalysis.CSharp.NetAnalyzers.dll could not be loaded.
Generated source code may be missing.
System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.CodeAnalysis.NetAnalyzers, Version=5.0.4.22903, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. O sistema não pode encontrar o arquivo especificado.
File name: 'Microsoft.CodeAnalysis.NetAnalyzers, Version=5.0.4.22903, Culture=neutral, PublicKeyToken=31bf3856ad364e35'
   at System.Reflection.RuntimeAssembly.GetExportedTypes()
   at System.Reflection.Assembly.get_ExportedTypes()
   at Stryker.Core.Initialisation.Buildalyzer.IAnalyzerResultExtensions.GetSourceGenerators(IAnalyzerResult analyzerResult, ILogger logger) in D:\a\1\s\src\Stryker.Core\Stryker.Core\Initialisation\Buildalyzer\IAnalyzerResultExtensions.cs:line 114

[16:55:08 WRN] Analyzer assembly C:\Program Files\dotnet\sdk\5.0.301\Sdks\Microsoft.NET.Sdk\targets..\analyzers\Microsoft.CodeAnalysis.NetAnalyzers.dll could not be loaded.
Generated source code may be missing.
System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.CodeAnalysis.Workspaces, Version=3.3.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. O sistema não pode encontrar o arquivo especificado.
File name: 'Microsoft.CodeAnalysis.Workspaces, Version=3.3.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'
   at System.Reflection.RuntimeAssembly.GetExportedTypes()
   at System.Reflection.Assembly.get_ExportedTypes()
   at Stryker.Core.Initialisation.Buildalyzer.IAnalyzerResultExtensions.GetSourceGenerators(IAnalyzerResult analyzerResult, ILogger logger) in D:\a\1\s\src\Stryker.Core\Stryker.Core\Initialisation\Buildalyzer\IAnalyzerResultExtensions.cs:line 114

This "Fail to load assembly" error can contribute to the test execution error? This same logs is present in the run of previous commit that I mentioned, and not affected the execution.

Can anyone help on how to solve this problem?

BrunoRM commented 3 years ago

The problem are related to a specific test that was using Faker (from bogus lib) in a MemberData test theory. The MemberData looks like this:

public static IEnumerable<object[]> Test_Data = new List<object[]>
{
      new object[] { new Faker().Random.String(1) }
}

After some tests I figured out that when new Faker() is removed (and replaced simple by a string with length 1), stryker runs the mutation tests successfuly.

rouke-broersma commented 3 years ago

Sounds like a bug with the dynamic test cases detection, thank you for reporting! I see that you reported that the bug is fixed in the rc release, thank you for checking!

BrunoRM commented 3 years ago

Apparently is a problem with the instantiation of Faker inside of MemberData. We have some other tests that uses MemberData without Faker, and Stryker can run it successfully.

Yes, after the Stryker upgrade to 1.0.0-rc3 the tests pass and reports is generated correctly, so seems that the bug is fixed.