Urgent <> Unable to get any mutation score and stryker is skipping all run - Stryker dot net

[inishantmishra]

Hi I am running following script to run in my pipeline- dotnet-stryker --test-project "*/Test.csproj" -m --with-baseline:$(System.PullRequest.TargetBranch) --version $(System.PullRequest.SourceBranch) --open-report --reporter "html" --reporter "progress" --reporter "cleartext" --break-at 80

But I am not getting the score and not getting any result. It seems all mutants are ignore but i haven't added any exclusion logic neither in comments nor in command.

Here are some snippet from logs

[17:30:58 WRN] Failed to load analyzer 'Microsoft.Extensions.Logging.Generators': Exception System.IO.FileLoadException: Assembly with same name is already loaded at System.Runtime.Loader.AssemblyLoadContext.LoadFromAssemblyPath(String assemblyPath) at System.Reflection.Assembly.LoadFrom(String assemblyFile) at Stryker.Core.Initialisation.Buildalyzer.IAnalyzerResultExtensions.AnalyzerAssemblyLoader.LoadFromPath(String fullPath) in //src/Stryker.Core/Stryker.Core/Initialisation/Buildalyzer/IAnalyzerResultExtensions.cs:line 126 at Microsoft.CodeAnalysis.Diagnostics.AnalyzerFileReference.GetAssembly() at Microsoft.CodeAnalysis.Diagnostics.AnalyzerFileReference.Extensions1.AddExtensions(Builder builder, String language, Func2 shouldInclude). [17:30:58 WRN] Failed to load analyzer 'Microsoft.Extensions.Options.SourceGeneration': Assembly with same name is already loaded (error : UnableToLoadAnalyzer, analyzer: All). [17:30:58 WRN] Failed to load analyzer 'Microsoft.Extensions.Options.SourceGeneration': Exception System.IO.FileLoadException: Assembly with same name is already loaded at System.Runtime.Loader.AssemblyLoadContext.LoadFromAssemblyPath(String assemblyPath) at System.Reflection.Assembly.LoadFrom(String assemblyFile) at Stryker.Core.Initialisation.Buildalyzer.IAnalyzerResultExtensions.AnalyzerAssemblyLoader.LoadFromPath(String fullPath) in //src/Stryker.Core/Stryker.Core/Initialisation/Buildalyzer/IAnalyzerResultExtensions.cs:line 126 at Microsoft.CodeAnalysis.Diagnostics.AnalyzerFileReference.GetAssembly() at Microsoft.CodeAnalysis.Diagnostics.AnalyzerFileReference.Extensions1.AddExtensions(Builder builder, String language, Func2 shouldInclude). [17:31:34 INF] 6417 mutants created [17:31:34 INF] Capture mutant coverage using 'CoverageBasedTest' mode. [17:34:36 INF] 160 mutants got status CompileError. Reason: Mutant caused compile errors [17:34:36 INF] 596 mutants got status Ignored. Reason: Removed by exclude from code coverage filter [17:34:36 INF] 5661 mutants got status Ignored. Reason: Removed by mutate filter [17:34:36 INF] 6417 total mutants are skipped for the above mentioned reasons [17:34:36 INF] 0 total mutants will be tested [17:34:36 WRN] It looks like all mutants with tests were ignored. Try a re-run with less ignoring! [17:34:36 WRN] It looks like all mutants resulted in compile errors. Mutants sure are strange!

Killed: 0 Survived: 0 Timeout: 0

All mutants have been tested, and your mutation score has been calculated

Please help and let me know what is happening or is there anything wrong am i doing?

[rouke-broersma]

You're supplying the option -m without argument, so nothing is mutation tested.

[inishantmishra]

Now I am getting this [16:36:30 INF] We could not locate a baseline for branch refs/heads/LocalDevelopment, now trying fallback version refs/heads/develop [16:36:30 INF] We could not locate a baseline report for the current branch, version or fallback version. Now running a complete test to establish a fresh baseline. [16:36:30 INF] Time Elapsed 00:09:47.9098005 Stryker.NET failed to mutate your project. For more information see the logs below:

No branch or tag or commit found with given target refs/heads/develop. Please provide a different GitDiffTarget.

[rouke-broersma]

Your pipeline is probably doing shallow fetch so does not include all refs and this Stryker cannot find the ref you specified

[inishantmishra]

@rouke-broersma I just want to run mutation test on the feature branch which i have raised as PullRequest to merge the develop branch which is target branch.

There is no mutation ran ever on develop and there is no report stored anywhere. Is it important to store the baseline somewhere? Can I store it in the repo itself?

Can you please explain me in steps what exactly should i do and how can I run mutation tests only for the changes i have done in my feature branch instead of whole develop branch? So that i can test the newly changes not the old source codes.

[rouke-broersma]

If you don't care about storing the report, use the since feature not the baseline feature

[inishantmishra]

So I just need to pass feature branch name or the develop branch name? Will following syntax work? --since localDevelopment

[inishantmishra]

@rouke-broersma Also I am not able to understand your comment

Your pipeline is probably doing shallow fetch so does not include all refs and this Stryker cannot find the ref you specified

[dupdob]

@inishantmishra : CI pipeline works on a light copy of the git repository (via this option https://git-scm.com/docs/git-fetch#Documentation/git-fetch.txt---depthltdepthgt). Which makes sense from a CI pipeline which only builds the software, but if git history is missing, Stryker is not able to identify the reference point. And no way to identify which parts of the code have been changed. Stryker can't do anything about that

You need to look on how to fetch a full git history (or a deep enough one) on your pipeline.

Is it clearer?

[inishantmishra]

@dupdob @rouke-broersma

I was able to run the striker run. but need help on these 2 queries- 1> Why lot of mutations are getting ignored either because of block or because of equality operator etc. How can I make sure these should not get ignored.

2. How can I use the since operator to run the mutations only on the files which are changed in feature branch which I have raised as PullRequest to merge in target branch-develop. I want to run this as PR Build on feature branch

[dupdob]

Regarding ignored mutations: Stryker automatically ignores block mutations (replacing a block with an empty block) when there are mutations in the block; the assumption is that there would be no situation where mutations inside the block are killed while the block mutation survives. This implies these mutations have little marginal value so Stryker skips them to save on test time.

[inishantmishra]

@dupdob @rouke-broersma

Stryker takes more than 9 minutes to run the mutation. And I think once the repo or test cases will increase it will take more. How can we reduce the time? How can we make it fast?

How does it run by default?

[dupdob]

9 minutes is quite fast actually.

Mutation testing is slow by essence.

The best approach is to look on how to make tests faster.

Alternatively, you can restrict which mutators are active and reduce the # of mutations, but it has obvious downsides

[inishantmishra]

Any way to make it faster? Cant we do some byte code change not the actual code change in the mutation process to make it faster?

[inishantmishra]

And how to filter out time taking tests like integration tests from running mutations on it?

[rouke-broersma]

Any way to make it faster? Cant we do some byte code change not the actual code change in the mutation process to make it faster?

Byte code change would not make this any faster. We place all mutants at once. Mutation testing takes a lot of time because we have to run your unit tests multiple times, that's inherent to mutation testing. This would not be different for byte code mutations.

[rouke-broersma]

And how to filter out time taking tests like integration tests from running mutations on it?

Sure, use test filters like you normally would. Stryker supports this.

[inishantmishra]

@rouke-broersma Any reference, how to use test filters?

[rouke-broersma]

@rouke-broersma Any reference, how to use test filters?

https://stryker-mutator.io/docs/stryker-net/configuration/#test-case-filter-string