--diff does not work

[eranrodrig-tipalti]

Hello, I've been trying to integrate .NET Stryker as a task in my team's TFS build pipeline, but it seems like the --diff argument doesn't take affect. I wanted to be absolutely certain that it doesn't work, so I created a dummy project with very simple code, and a test project for it. Then I branched out of master, made a very small change (Console.WriteLine somewhere) and ran the following commands in the terminal: 'dotnet stryker --git-source "master" --diff', 'dotnet stryker' and got the exact same results for both. Attached are the log files of the test results, with the diff argument and without it and the "git diff" between the branches. Here's a link to the repository with the dummy project and test project: https://github.com/eranrodrig-tipalti/MutationTestingDummy. The stryker-config file can be found in the repository under the "someBranch" branch, in the test project. I'm working with .NET Core and Xunit. gitDiff.txt log-20200825(no-diff).txt log-20200825 (diff).txt

[rouke-broersma]

Hi @eranrodrig-tipalti! From your diff log, it looks like you have changes in your test project, is that correct? The current iteration of the feature has been built to test the complete project when a change in test files is detected, because we cannot determine based on the changed test files which source files are affected.

We are working on improving this behavior in #1101

[eranrodrig-tipalti]

So what you're saying is that I need to merge my tests into master, and only then run .NET Stryker to get the diff functionality?

[rouke-broersma]

So what you're saying is that I need to merge my tests into master, and only then run .NET Stryker to get the diff functionality?

The diff feature is working as expected. When tests have changed we currently cannot determine which mutations are affected by that change so we have to test all mutations instead of only the mutations actually changed in your diff. This is simply a limitation in our first version of the diff feature. We are working to improve this so the diff feature is more useful.

There is no workaround at this time. Even if you do what you're suggesting, you're just faking the result. The unit tests you changed might have an affect on other mutations that are not in the changed files, but you won't know as those changed unit tests might never be run when using the diff feature.

[eranrodrig-tipalti]

I thought that was the point of the diff feature... You want to check how the tests you added to the test suite perform on the code you added to the project. Can you think of a more elegant way to do this than the solution I offered?

[rouke-broersma]

I thought that was the point of the diff feature... You want to check how the tests you added to the test suite perform on the code you added to the project. Can you think of a more elegant way to do this than the solution I offered?

That is absolutely the aim of the feature. We just did not have the capability to match a change in a test class with the corresponding unit test when we first created this feature.

Right now I don't think there is a way to get around us not having that capability, except wait for #1101

[raschmitt]

@Mobrockers As someone who has also been misled by this feature in the past I think we should better document it's expected behavior in the configuration docs

Especially now after #1101 has been merged.

If you guys also feel this need I can volunteer for doing that.

[rouke-broersma]

@Mobrockers As someone who has also been misled by this feature in the past I think we should better document it's expected behavior in the configuration docs

Especially now after #1101 has been merged.

If you guys also feel this need I can volunteer for doing that.

Much appreciated :) It's too easy as the people responsible for building the feature to create documentation that works if you know exactly how the feature was built, so I think it is a very good idea for an outsider to help improve it!