stryker-mutator / stryker-net

Mutation testing for .NET core and .NET framework!
https://stryker-mutator.io
Apache License 2.0
1.75k stars 176 forks source link

When running stryker against a solution, no baseline is found #2441

Open MischaVreeburg opened 1 year ago

MischaVreeburg commented 1 year ago

When running stryker against a multiproject solution, stryker will consistently tell that no baseline is found, even though the file is in the baseline folder.

Stryker is not finding the baseline. After debugging it looks at a strange path namely:

\.csproj\StrykerOutput\baseline\feature\\stryker-report.json It looks like stryker expects that there is a folder named for the current projectfile it is testing and it expects the baseline in this folder. This is especially strange when running stryker for the full solution. Then stryker makes the baseline folder relative to the solution file: |- solutionfile |- StrykerOutput |- baseline |- feature |- |- stryker-report.json I believe that in the load method of the Diskbaselineprovider, the _options.projectpath should be _options.WorkingDirectory, at least for solutions.
MischaVreeburg commented 1 year ago

The bug appears to be in the ProjectOrchestrator class, in the MutateSolution method. Where the options are copied to a new project options. There the projectPath is filled with the path to the project file, in stead of keeping the projectpath for the solution.

I'll prepare a pull request with this fix.

rouke-broersma commented 1 year ago

Output path should be used from options instead of trying to figure it out in the provider: https://github.com/stryker-mutator/stryker-net/blob/master/src/Stryker.Core/Stryker.Core/Options/StrykerOptions.cs#L46

MischaVreeburg commented 1 year ago

Using the output path in the diskbaselineprovider gives the following path: < PathToSolution >\StrykerOutput\2023-03-20.11-41-03\StrykerOutput\baseline\feature\< BranchName >\stryker-report.json instead of < PathToSolution >\StrykerOutput\baseline\feature< branchName >\stryker-report.json

The output path doesn't look like a viable option.

rouke-broersma commented 1 year ago

You're right OutputPath as it stands does not work, however WorkingDirectory is also not exactly correct because then we're 1 creating output outside of the path specified by the user and 2 the reports might be placed in a folder that does not contain a gitignore so would show up in git.

I think the correct approach is to split the OutputPath into the OutputPath as given by the user and the generated 'current run session folder', and use the OutputPath for the baseline disk provider.

MischaVreeburg commented 1 year ago

I think that the best place to change would be the ProjectOrchestrator, where the erroneous information is placed in the ProjectPath property. This would solve this issue. And is only a single property assignment change.

If you would describe the intended workings of the outputpath in this case, I can change the code to reflect this.

rouke-broersma commented 1 year ago

The project path in the child project copy must be the project path for the child path because all further actions are from the context of the project and not the solution. So we can't change that property.