DotCoverAnalyse doesnt work with multiple DotNetCoreTest calls

[ryudice]

What You Are Seeing?

DotCoverAnalyse only executes once when running DotNetCoreTest for multiple project files and only picks up the result of the first project file that was run against DotNetCorTest

What is Expected?

DotCoverAnalyse should include results from all of the project files and should support

What version of Cake are you using?

Cake.Incubator&version=3.0.0

Are you running on a 32 or 64 bit system?

64 bit

What environment are you running on? Windows? Linux? Mac?

Windows

Are you running on a CI Server? If so, which one?

No

How Did You Get This To Happen? (Steps to Reproduce)

This is the cake script:

Task("DotCover")
.IsDependentOn("Build Solution")
  .Does(()=>{
    DotCoverAnalyse(
      tool => {
        var settings = new DotNetCoreTestSettings() { NoBuild = true };
        var testProjects = GetFiles("./tests/**/*.csproj");

        foreach(var testProject in testProjects) {
            Information($"Running tests for project: {testProject}");
          tool.DotNetCoreTest(testProject.FullPath, settings);

        }
      },
      new FilePath("./test-results/result.html"),
      new DotCoverAnalyseSettings(){
        ReportType = DotCoverReportType.HTML
      }
    );
  });

Output Log

========================================
DotCover
========================================
Running tests for project: C:/Project1.csproj
Running tests for project: C:/Project2.csproj
Running tests for project: C:/Project 3.csproj
JetBrains dotCover Console Runner 2018.2.3. Build 777.0.20180912.160624
Copyright (c) 2009-2018 JetBrains s.r.o. All rights reserved.
[JetBrains dotCover] Coverage session started [10/2/2018 4:58:44 PM]
Test run for C:\Project1\bin\Debug\netcoreapp2.1\JProject1.Tests.dll(.NETCoreApp,Version=v2.1)
Microsoft (R) Test Execution Command Line Tool Version 15.8.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

Total tests: 17. Passed: 17. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 2.8562 Seconds
[JetBrains dotCover] Coverage session finished [10/2/2018 4:58:52 PM]
[JetBrains dotCover] Coverage results post-processing started [10/2/2018 4:58:52 PM]
[JetBrains dotCover] Report generation started [10/2/2018 4:58:52 PM]
[JetBrains dotCover] Report generation finished [10/2/2018 4:58:54 PM]
[JetBrains dotCover] Coverage results post-processing finished [10/2/2018 4:58:54 PM]

Task                          Duration
--------------------------------------------------
Build solution                00:00:01.5444099
DotCover                      00:00:10.4584207
--------------------------------------------------
Total:                        00:00:12.0028306

[patriksvensson]

@ryudice Does DotCover support doing this over many projects in one call?

[wwwlicious]

Ah, I have had some amount of fun with this particular one...and by fun, I mean the opposite!

The short answer is ...not really, or at least I couldn't get it to work reliably.

In the end, because of the lack of ability of dotnet test to run multiple assemblies or at least, filter out non-test assemblies from a solution file or folder argument (what vstest does), the deprecation of dotnet-xunit clr tool, the original xunit console not working with netcore ... the issues with mixed netcore and netfx test projects as well as multi-targeted test assemblies and something in the vstest tool that was problematic (but can't quite recall at the moment), I went with a cake script setup that

1. injects an xunit test logger package into each test project during cake build
2. runs dotcover analyze (using dotnet test) on each test project
3. merges the dotcover results (and the xunit test results) into single output files and generates the coverage report.

It's not as clean or as fast as things used to be with xunit console being able to run multiple test assemblies but it is functionally equivalent ... albeit the projects are currently modifed on disk following a build as I could not find a way to avoid this and send the project file source directly to the test tool (creating a temp project file as msbuild does could prob clean this part up)

...but it's proven robust with all manner of old and new/mix and match and not too much slower. I may see if I can incorporate this into cake.incubator at some point to improve the 'solution based' test running / settings for all flavours of dotnet test frameworks as well as the associated dotcover story as much of the work for things like test project and package detection in the project parsing were done for exactly this scenario.

[devlead]

Sorry wrong button 😊

@wwwlicious do you have an gist/example of your approach?

[wwwlicious]

@devlead not at the moment no, it is woven in with a bunch of other stuff that is unlikely to translate for most folks, hence it would be eaiser to look at refactoring the generic parts into cake aliases in incubator.

[hotchkj]

For what it's worth, some years later, the above still seems to be the best approach - you have to manually figure out which project files to run against dotnet test, cover each of those, then merge and run reporting separately (which is actually quite useful anyway, because a solution can contain tests that aren't actually unit tests, or only run on special hardware).

An additional wrinkle is if your solution or set of project files have different platform settings - the dotnet test runners only expect one platform (and it's very hacky within the NET SDK at that, for 32-bit you have to swap out the entire CLI...), so you will typically need to have a run for AnyCPU, a run for x64, a run for x86 etc.. We ended up writing an MSBuild-based tool to figure out project platforms.

No complaint at Cake, but I'm still astonished how absymal the unit testing support is in NET CLI.