dotnet watch test should only run changed tests

[ffMathy]

The issue #10366 was marked as stale prematurely. I think it still makes sense to implement.

Right now, the dotnet watch test runs all tests every time even if only one file has changed.

The test runner can already provide code coverage. It could use this coverage information to determine if some code that has been modified, should trigger re-running a specific test.

Most other test runners already do this, and it is super useful. For instance, Live Unit Testing in Visual Studio Professional, Jest does it within JavaScript, Ava seems to do it, and ReSharper's DotCover does it as well.

Right now, I have to run my whole test suite every time I change a few things. That's a huge time consumer! It forces me to use Visual Studio instead of Visual Studio Code for now, and I'd love to make the switch.

I think it is important that coverage files are still supported. That way, Visual Studio Code extensions like Coverage Gutters still works well in watch mode.

Flow

To explain this with an example, I'll describe it as a flow of steps.

1. I make a class and a test

public class SomeClass {
   public bool SomeCondition {get;set;}

   public A() {
      if(SomeCondition) {
         Console.WriteLine("A1");
      } else {
         Console.WriteLine("A2");
      }
   }
}

[TestClass]
public class SomeClassTest {
   [TestMethod]
   public void TestA1() {
      var obj = new SomeClass() { SomeCondition = true };
      obj.A();
   }

   [TestMethod]
   public void TestA2() {
      var obj = new SomeClass() { SomeCondition = false };
      obj.A();
   }
}

2. I run the tests in watch mode

The test runner now internally keeps track of the coverage. In other words, half of method A is now covered by TestA1, and half by TestA2.

3. I modify some of the code

I modify the original class to be the following (changing the Console.WriteLine("A1") to Console.WriteLine("foo"):

public class SomeClass {
   public bool SomeCondition {get;set;}

   public A() {
      if(SomeCondition) {
         Console.WriteLine("foo"); //this line was changed
      } else {
         Console.WriteLine("A2");
      }
   }
}

4. The test runner runs affected tests

Now, the test runner detects that only covered paths by the test TestA1 has been changed, so it only runs the TestA1 test.

[ghost]

We've moved this issue to the Backlog milestone. This means that it is not going to be worked on for the coming release. We will reassess the backlog following the current release and consider this item at that time. To learn more about our issue management process and to have better expectation regarding different types of issues you can read our Triage Process.

[ffMathy]

Exciting! Let me know if you have any questions along the way.

[pug-pelle-p]

+1 This would be an awesome feature! :)

[Cpcrook]

I'd definitely appreciate this functionality to replicate that of nCrunch in Visual Studio.

[ffMathy]

Meanwhile, check this CLI that I'm building that does it already: https://github.com/pruner/cli

[oatsoda]

Seems like this is potentially related to the lack of Test Impact Analysis - which .NET Framework had, but .NET Core does not. See https://github.com/microsoft/azure-pipelines-tasks/issues/12315