Stryker should abort when no tests found

[kwlin]

When no initial tests found (because of missing nuget package (MSTest.TestAdapter)), stryker should abort the run instead.

For example:

PS C:\Src\Git\Vecozo.Grouper\41-PcsGrAfleiden\Implementation\Vecozo.Grouper.PcsGrAfleiden\05-Implementation.UnitTest> dotnet stryker -tr vstest --reporters "['html', 'consoleprogressbar']" --solution-path ..\Vecozo.Gr
ouper.PcsGrAfleiden.sln --project-file="C:\Src\Git\Vecozo.Grouper\41-PcsGrAfleiden\Implementation\Vecozo.Grouper.PcsGrAfleiden\04-Implementation\04-Implementation.csproj"

   _____ _              _               _   _ ______ _______
  / ____| |            | |             | \ | |  ____|__   __|
 | (___ | |_ _ __ _   _| | _____ _ __  |  \| | |__     | |
  \___ \| __| '__| | | | |/ / _ \ '__| | . ` |  __|    | |
  ____) | |_| |  | |_| |   <  __/ |    | |\  | |____   | |
 |_____/ \__|_|   \__, |_|\_\___|_| (_)|_| \_|______|  |_|
                   __/ |
                  |___/

Version 0.9.0 (beta)

[15:06:32 INF] Using C:\Src\Git\Vecozo.Grouper\41-PcsGrAfleiden\Implementation\Vecozo.Grouper.PcsGrAfleiden\05-Implementation.UnitTest\05-Implementation.UnitTest.csproj as project file
[15:06:36 INF] Started initial build using msbuild.exe
[15:06:39 INF] Auto detected msbuild version 15.9.21.664 at: C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\15.0\Bin\MSBuild.exe
[15:07:37 INF] Initial build successful
[15:07:37 INF] Using C:\Src\Git\Vecozo.Grouper\41-PcsGrAfleiden\Implementation\Vecozo.Grouper.PcsGrAfleiden\05-Implementation.UnitTest\05-Implementation.UnitTest.csproj as project file
[15:07:42 INF] Started initial build using msbuild.exe
[15:07:45 INF] Auto detected msbuild version 15.9.21.664 at: C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\15.0\Bin\MSBuild.exe
[15:08:48 INF] Initial build successful
[15:08:53 INF] Using testrunner VsTest
[15:08:53 INF] Initial testrun started
[15:08:53 INF] Total number of tests found in initial test run: 0
[15:08:53 INF] Initial testrun successful in 453 ms
[15:08:53 INF] Using 30679 ms as testrun timeout
[15:09:04 INF] 32 mutants could not compile and got status CompileError
[15:09:04 INF] 1908 mutants ready for test
Tests progress | █--------- | 271 / 1908 | 14 % | ~10m 32s |
Killed : 0
Survived: 271
Timeout : 0

[richardwerkman]

This would mean a breaking change. Our "dotnet test" runner parses the console output for the number of tests executed. I've seen an example of a run on a franch system where the parser found 0 tests because the language was different in the console output.

If we stop stryker on 0 tests found, users on a non english system could not use stryker anymore.

I'm more fond of a less rigorous solution like logging a warning on no test cases found. For your case we should fix the vstest runner instead I think.

[simondel]

Logging a warning sounds like a good idea

[rouke-broersma]

@richardwerkman We could at the very least implement this for vstest. We could also look into using -t|--list-tests to list the found testcases. Depending on the output of this command we might be able to more accurately detect the total number of unit tests independent of language.

[nicojs]

If we stop stryker on 0 tests found, users on a non english system could not use stryker anymore.

That is a bug IMHO. We should be able to parse the lines on all languages. Maybe look for a number in the output, or something a bit stricter, but not exactly text matching. Or at least make the regex configurable, so that people can fix this themselves.

If you know for sure that 0 tests have run, then I would suggest the following:

Mutate like you normally would, but skip the running of the tests. Just directly report that all mutants survived because we couldn't find any tests. That is the most logical way of handling it, no need for a warning or abort.

[rouke-broersma]

Implemented by #506 and released since 0.12.0