search

stryker-mutator / stryker-net

Mutation testing for .NET core and .NET framework!
https://stryker-mutator.io
Apache License 2.0
1.76k stars 175 forks source link

Unhandled exception in Core.Initialisation.ProjectOrchestrator.MutateProjects #2621

Closed tommysor closed 1 year ago

tommysor commented 1 year ago

Describe the bug Unhandled exception during initialization. This error does not happen when using Stryker 3.8.2 (completes successfully). Stryker 3.9.0 and 3.10.0 fail

Unhandled exception. System.InvalidOperationException: Sequence contains no elements
   at System.Linq.ThrowHelper.ThrowNoElementsException()
   at System.Linq.Enumerable.First[TSource](IEnumerable`1 source)
   at Stryker.Core.Initialisation.ProjectOrchestrator.MutateProjects(StrykerOptions options, IReporter reporters, ITestRunner runner)+MoveNext()

Logs Console output (shortened) for success (v3.8.2), then failure (v3.10.0)

@tommysor ➜ /workspaces/SimplifiedSearch (main) $ dotnet stryker --reporter Dots
[.. snip ..]
Version: 3.8.2

A new version of Stryker.NET (3.10.0) is available. Please consider upgrading using `dotnet tool update -g dotnet-stryker`

[20:55:04 INF] Identifying projects to mutate in /workspaces/SimplifiedSearch/SimplifiedSearch.sln. This can take a while.
[20:55:11 INF] Found 1 source projects
[20:55:11 INF] Found 2 test projects
[20:55:18 INF] Found project /workspaces/SimplifiedSearch/src/SimplifiedSearch/SimplifiedSearch.csproj to mutate.
[20:55:22 INF] Total number of tests found: 114.
[20:55:22 INF] Initial testrun started.
[20:55:32 INF] 180 mutants created
[20:55:32 INF] Capture mutant coverage using 'CoverageBasedTest' mode.
[.. snip ..]
[20:55:38 INF] 59    total mutants are skipped for the above mentioned reasons
[20:55:38 INF] 121   total mutants will be tested
........S...............S.....................................................S.............S............................
[21:00:17 INF] Time Elapsed 00:05:12.9085227
[21:00:17 INF] The final mutation score is 95.90 %

@tommysor ➜ /workspaces/SimplifiedSearch (main) $ dotnet tool restore
Tool 'dotnet-stryker' (version '3.10.0') was restored. Available commands: dotnet-stryker
Restore was successful.

@tommysor ➜ /workspaces/SimplifiedSearch (main) $ dotnet stryker --reporter Dots --log-to-file
[.. snip ..]
Version: 3.10.0

[21:13:11 INF] Analysis starting.
[21:13:11 INF] Identifying projects to mutate in /workspaces/SimplifiedSearch/SimplifiedSearch.sln. This can take a while.
[21:13:18 INF] Found 1 source projects
[21:13:18 INF] Found 2 test projects
[21:13:18 INF] Analysis complete.
[21:13:18 INF] Building solution SimplifiedSearch.sln
[21:13:30 INF] Time Elapsed 00:00:19.1481713
Unhandled exception. System.InvalidOperationException: Sequence contains no elements
   at System.Linq.ThrowHelper.ThrowNoElementsException()
   at System.Linq.Enumerable.First[TSource](IEnumerable`1 source)
   at Stryker.Core.Initialisation.ProjectOrchestrator.MutateProjects(StrykerOptions options, IReporter reporters, ITestRunner runner)+MoveNext()
   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
   at Stryker.Core.StrykerRunner.RunMutationTest(IStrykerInputs inputs, ILoggerFactory loggerFactory, IProjectOrchestrator projectOrchestrator)
   at Stryker.CLI.StrykerCli.RunStryker(IStrykerInputs inputs) in /_/src/Stryker.CLI/Stryker.CLI/StrykerCLI.cs:line 93
   at Stryker.CLI.StrykerCli.<>c__DisplayClass10_0.<Run>b__0() in /_/src/Stryker.CLI/Stryker.CLI/StrykerCLI.cs:line 68
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.<>c__DisplayClass143_0.<OnExecute>b__0(CancellationToken _)
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.ExecuteAsync(String[] args, CancellationToken cancellationToken)
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args)
   at Stryker.CLI.StrykerCli.Run(String[] args) in /_/src/Stryker.CLI/Stryker.CLI/StrykerCLI.cs:line 74
   at Stryker.CLI.Program.Main(String[] args) in /_/src/Stryker.CLI/Stryker.CLI/Program.cs:line 14

Log from failing run

2023-08-02T21:13:11.4665395+00:00  [DBG] Stryker started with options: StrykerOptions { MsBuildPath: null, DevMode: False, ProjectPath: "/workspaces/SimplifiedSearch", IsSolutionContext: True, WorkingDirectory: "/workspaces/SimplifiedSearch", OutputPath: "/workspaces/SimplifiedSearch/StrykerOutput/2023-08-02.21-13-11", ReportPath: "/workspaces/SimplifiedSearch/StrykerOutput/2023-08-02.21-13-11/reports", ReportFileName: "mutation-report", SolutionPath: "/workspaces/SimplifiedSearch/SimplifiedSearch.sln", TargetFramework: null, LogOptions: LogOptions { LogToFile: True, LogLevel: Information }, MutationLevel: Standard, Thresholds: Thresholds { High: 97, Low: 90, Break: 90 }, AdditionalTimeout: 5000, LanguageVersion: Default, Concurrency: 1, SourceProjectName: "./src/SimplifiedSearch/SimplifiedSearch.csproj", TestProjects: [], TestCaseFilter: "", Reporters: [Dots], WithBaseline: False, BaselineProvider: Disk, AzureFileStorageUrl: "", AzureFileStorageSas: "", DashboardUrl: "https://dashboard.stryker-mutator.io", DashboardApiKey: null, Since: False, SinceTarget: "master", DiffIgnoreChanges: [], FallbackVersion: "master", ModuleName: "", ReportTypeToOpen: null, Mutate: [FilePattern { Glob: Glob { Tokens: [WildcardDirectoryToken { TrailingPathSeparator: PathSeparatorToken { Value: / }, LeadingPathSeparator: null }, WildcardToken {  }] }, IsExclude: False, TextSpans: [TextSpan { Start: 0, End: 2147483647, Length: 2147483647, IsEmpty: False }] }], IgnoredMethods: [Regex { Options: IgnoreCase, RightToLeft: False, MatchTimeout: -00:00:00.0010000 }], ExcludedMutations: [], ExcludedLinqExpressions: [], OptimizationMode: CoverageBasedTest, ProjectName: "github.com/tommysor/SimplifiedSearch", ProjectVersion: "", BreakOnInitialTestFailure: False } (fac41b16)
2023-08-02T21:13:11.5520477+00:00  [INF] Analysis starting. (7076a364)
2023-08-02T21:13:11.5648238+00:00  [INF] Identifying projects to mutate in "/workspaces/SimplifiedSearch/SimplifiedSearch.sln". This can take a while. (352fa797)
2023-08-02T21:13:11.6871508+00:00  [DBG] Analyzing 3 projects. (a101d45b)
2023-08-02T21:13:11.6943903+00:00  [DBG] Analyzing "tests/SimplifiedSearch.Tests.Internal/SimplifiedSearch.Tests.Internal.csproj" (1defaf37)
2023-08-02T21:13:11.6943893+00:00  [DBG] Analyzing "src/SimplifiedSearch/SimplifiedSearch.csproj" (1defaf37)
2023-08-02T21:13:11.6944367+00:00  [DBG] Analyzing "tests/SimplifiedSearch.Tests/SimplifiedSearch.Tests.csproj" (1defaf37)
2023-08-02T21:13:18.6214158+00:00  [DBG] Analysis of project "tests/SimplifiedSearch.Tests.Internal/SimplifiedSearch.Tests.Internal.csproj" succeeded. (adaf2e80)
2023-08-02T21:13:18.6217615+00:00  [DBG] Analysis of project "tests/SimplifiedSearch.Tests/SimplifiedSearch.Tests.csproj" succeeded. (adaf2e80)
2023-08-02T21:13:18.6267073+00:00  [DBG] Analysis of project "src/SimplifiedSearch/SimplifiedSearch.csproj" succeeded. (adaf2e80)
2023-08-02T21:13:18.6392840+00:00  [INF] Found 1 source projects (c1ce5c69)
2023-08-02T21:13:18.6548894+00:00  [INF] Found 2 test projects (6f0fd09d)
2023-08-02T21:13:18.6667779+00:00  [INF] Analysis complete. (f039f576)
2023-08-02T21:13:18.6738288+00:00  [INF] Building solution "SimplifiedSearch.sln" (abcf88bf)
2023-08-02T21:13:18.6753358+00:00  [DBG] Started initial build using "dotnet build" (a4e1a5e2)
2023-08-02T21:13:18.6753793+00:00  [DBG] Initial build using path: "/workspaces/SimplifiedSearch/SimplifiedSearch.sln" (657c8664)
2023-08-02T21:13:30.4284223+00:00  [DBG] Initial build successful (61599426)
2023-08-02T21:13:30.4842564+00:00  [INF] Time Elapsed 00:00:19.1481713 (f4427d77)

Expected behavior Mutation test completes

Desktop (please complete the following information):

Additional context Repro (not minimal) https://github.com/tommysor/SimplifiedSearch

Location of error I believe the error happens on ProjectOrchestrator.cs:57 (projectInfos.First()) https://github.com/stryker-mutator/stryker-net/blob/adfe7c2590ab09d1e1442982a17a9dc80513398a/src/Stryker.Core/Stryker.Core/Initialisation/ProjectOrchestrator.cs#L45-L57

projectInfos gets its value from https://github.com/stryker-mutator/stryker-net/blob/adfe7c2590ab09d1e1442982a17a9dc80513398a/src/Stryker.Core/Stryker.Core/Initialisation/InitialisationProcess.cs#L52-L58

And then here. I assume something goes wrong here, but I can't tell where. https://github.com/stryker-mutator/stryker-net/blob/adfe7c2590ab09d1e1442982a17a9dc80513398a/src/Stryker.Core/Stryker.Core/Initialisation/InputFileResolver.cs#L49-L89

dupdob commented 1 year ago

thanks for opening this issue. I have been able to reproduce the problem via your project. I confirm this is bug, likely due to the massive redesign of the so called 'solution mode' in V3.9. The good news is that I have a workaround solution for you (see next comment to understand why I no longer see this as a workaround). please remove reference to relative path for the 'project' parameters. Like this:

{
    "stryker-config": {
        "project": "src/SimplifiedSearch/SimplifiedSearch.csproj",
        "ignore-methods": [
            "ConfigureAwait"
        ],
...

The problem comes from how this parameter is interpreted. I will work on a fix, but the workaround should be durable, i.e. still valid once the fix is released.

dupdob commented 1 year ago

I analyzed a bit further to under why there is a regression: it looks to me it is because the project parameter was ignored when ran in solution mode, i.e. when Stryker is run from the solution directory. In V 3.8.x and earlier, Stryker identified test project(s) (using solution analysis and configured test projects, if any); it then mutated any project tested by one or more of these projects, disregarding the project option. Now this option is taken into account, but it is supposed to host the project file name not the project path (see the doc).

So I will close this issue, as the problem is (more or less) in older version which should have warned about this unneeded option. NB : will see how to improve error reporting anyway, since this was an abrupt crash in all cases