"Native" support for MSTest v2

[fwinkelbauer]

Backstory

MSTest.exe was the primary tool used to execute unit tests written using Microsofts own Unit Testing framework in Visual Studio. This command line tool has been replaced by vstest.console.exe in newer versions of Visual Studio.

Microsoft introduced a new version of the MSTest framework called MSTest v2 last year. The new framework offers a new set of features, e.g. the possibility to parametrize tests (meaning that a single test is executed several times using a different set of parameters).

MSText.exe cannot execute parametrized unit tests of MSTest v2, meaning that it will only run a subset of all unit tests found in a project. Example: a test project includes 80 simple unit tests and 20 parametrized unit tests. While Visual Studio executes all 100 tests, MSText.exe will only run ~80.

vstest.console.exe will not find any unit tests of a MSTest v2 project by default.

MSTest v2 is distributed via NuGet and it is used in the default template for creating test projects in Visual Studio 2017.

In order to execute all 100 tests, one has to add a parameter to the vstest.console.exe call. In Cake this would look like this:

private void MSTestV2(string pattern)
{
  VSTest(pattern, new VSTestSettings { TestAdapterPath = "." });
}

The TestAdapterPath parameter can be set to "." because the MSTest v2 dll will be copied to the output folder. Even though this is rather easy, it is far from obvious.

Question

Should Cake "know about"/"deal with" this behaviour? As of now a build script might not give the expected results and it is very hard for a developer to notice that some tests are not executed.

Possible Solutions

• A new alias, e.g. MSTestV2
• A named parameter in VSTestSettings

Side question: Shipping MSTest v2 via NuGet seems very similar to how other frameworks (e.g. NUnit) work. I have not looked into the implementation of these aliases in Cake, but could there be the possibility to "generalize" these cases? This SO post led me to the posted code (using the TestAdapterPath) above.

Open Question

@devlead raised the following question in a Slack channel: can vstest.console.exe run MSTest v1 tests if the VSTestAdapterPath parameter is set (just as in the code above)? My limited tests say yes, but I might be missing some edge cases. This would need additional testing.

Edit: I have updated this issue after some further investigation/discussion in Slack.

Thank you

[olpr-cloud]

any news on this? :-)

[fwinkelbauer]

Personally I haven't looked into this issue in more detail. I have encountered other problems using MSTest in Cake (see #1522), which is why I am favoring xUnit in my personal projects.

Previously I was using this workaround (usage can be seen here).