Cannot specify test project(s) in solution mode (tp parameters as no effect)

[CzBuCHi]

Describe the bug Calling stryker with -s C:\Path\To\Solution.sln -p Project.csproj -tp C:\Path\To\Project.Tests\Projects.Tests.csproj from C:\Path\To\ directory will fail to find test project.

Logs (reduced)

[16:55:34 INF] Logging enabled at level Debug
Version: 3.8.0

[16:55:34 DBG] Stryker started with options: {"MsBuildPath": null, "DevMode": false, "ProjectPath": "c:\\projects\\dotnet\\Zeus\\5.0", "IsSolutionContext": true, "WorkingDirectory": "c:\\projects\\dotnet\\Zeus\\5.0", "OutputPath": "c:\\projects\\dotnet\\Zeus\\5.0\\StrykerOutput\\2023-05-16.16-55-34", "ReportPath": "c:\\projects\\dotnet\\Zeus\\5.0\\StrykerOutput\\2023-05-16.16-55-34\\reports", "ReportFileName": "mutation-report", "SolutionPath": "c:\\projects\\dotnet\\Zeus\\5.0\\Zeus.sln", "TargetFramework": null, "LogOptions": {"LogToFile": false, "LogLevel": "Debug", "$type": "LogOptions"}, "MutationLevel": "Standard", "Thresholds": {"High": 80, "Low": 60, "Break": 0, "$type": "Thresholds"}, "AdditionalTimeout": 5000, "LanguageVersion": "Default", "Concurrency": 8, "SourceProjectName": "Zeus.csproj", "TestProjects": ["c:\\projects\\dotnet\\Zeus\\5.0\\Zeus._Tests\\Zeus._Tests.csproj"], "TestCaseFilter": "", "Reporters": ["Progress", "Html"], "WithBaseline": false, "BaselineProvider": "Disk", "AzureFileStorageUrl": "", "AzureFileStorageSas": "", "DashboardUrl": "https://dashboard.stryker-mutator.io", "DashboardApiKey": null, "Since": false, "SinceTarget": "master", "DiffIgnoreChanges": [], "FallbackVersion": "master", "ModuleName": "", "ReportTypeToOpen": null, "Mutate": [{"Glob": {"Tokens": [{"TrailingPathSeparator": {"Value": "\\", "$type": "PathSeparatorToken"}, "LeadingPathSeparator": null, "$type": "WildcardDirectoryToken"}, {"$type": "WildcardToken"}], "$type": "Glob"}, "IsExclude": false, "TextSpans": [{"Start": 0, "End": 2147483647, "Length": 2147483647, "IsEmpty": false, "$type": "TextSpan"}], "$type": "FilePattern"}], "IgnoredMethods": [], "ExcludedMutations": [], "ExcludedLinqExpressions": [], "OptimizationMode": "CoverageBasedTest", "ProjectName": "", "ProjectVersion": "", "BreakOnInitialTestFailure": false, "$type": "StrykerOptions"}
[16:55:34 INF] Identifying projects to mutate in C:\Path\To\Solution.sln. This can take a while.
[16:55:34 DBG] Analysing 2 projects
[16:55:34 DBG] Analysing C:\Path\To\Project\Project.csproj
[16:55:37 DBG] Analysing C:\Path\To\Project.Tests\Project.Tests.csproj
[16:55:38 DBG] Analysis of project C:\Path\To\Project\Project.csproj succeeded
[16:55:39 DBG] Analysis of project C:\Path\To\Project.Tests\Project.Tests.csproj succeeded
[16:55:40 INF] Found 2 source projects
[16:55:40 INF] Found 0 test projects
[16:55:40 DBG] Started initial build using msbuild.exe
[16:55:40 DBG] Using vswhere.exe to locate msbuild
[16:55:40 DBG] Msbuild executable path found at C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Current\Bin\MSBuild.exe
[16:55:46 DBG] Initial build successful
[16:55:46 WRN] No test projects could be found for C:\Path\To\Project\Project.csproj
[16:55:46 WRN] No test projects could be found for C:\Path\To\Project.Tests\Project.Tests.csproj
[16:55:46 INF] Time Elapsed 00:00:12.0432130
No test projects found. Please add a test project to your solution or fix your stryker config.

Expected behavior Same behavior as when calling stryker with -s C:\Path\To\Solution.sln -p Project.csproj from C:\Path\To\Project.Tests\ directory

• aka use C:\Path\To\Project.Tests\Project.Tests.csproj as test project and C:\Path\To\Project\Project.csproj as project under test

Desktop (please complete the following information):

• OS: Windows 11
• Type of project class library
• Framework Version net48
• Stryker Version 3.8.0

Additional context Cannot upgrade to 3.9.0 because of #2528

[dupdob]

Running Stryker.Net from the solution folder triggers the so called 'solution mode; Stryker then identifies test projects in the solution and mutate the others. In solution mode, the tp parameter is ignored. To put it another way: Stryker is expected to perform everything automatically in solution mode. But this feature was limited to simple solutions pre V3.9, when we massively rewrote the initialization phase to bring solution mode to full maturity.

Here it appears that Project.Tests is not identified as a test project. To be identified as a test project, a project must either has been created as one in VS, it then bears a specific project type guid -3AC096D0-A1C2-E12C-1390-A8335801FDAB- recognized by Stryker. Or it must have the property IsTestProject set to true, which is usually set by the unit test frameworks.

We have identified that older testprojects may be missing both characteristics, so we added detection of known test frameworks (xUnit, NUnit and MsTest) as another criteria.

To work around this, you can either: 1) change the ProjectTypeGuids property of Project.Tests to 3AC096D0-A1C2-E12C-1390-A8335801FDAB (by manually editing the csproj file). 2) try V3.9, which should work in solution mode 3) add the IsTestProject property to Project.Tests.csproj file

[CzBuCHi]

added both <ProjectTypeGuids>3AC096D0-A1C2-E12C-1390-A8335801FDAB</ProjectTypeGuids> and <IsTestProject>True</IsTestProject> to test projects and ran dotnet stryker from solution dir and stryker found all test projects and now is mutating ...

PS: my original idea was to generate reports for each test/tested [project and then merge them into one, but now i discovered that stryker can run multiple tests in solution (didnt found any mention of this in doc) so i dont need to use -p + -tp at all ...