dotnet test coverage with partially failed tests

[livarcocc]

From @saxenark on Tuesday, July 16, 2019 1:20:16 PM

Hi,

I am using .net core 2.2 in my ASP.NET core application. My command line to invoke tests is like so:

dotnet test -v n -c Release --no-build --logger:trx /p:CollectCoverage=true /p:CoverletOutputFormat=\"opencover,lcov,Cobertura\" /p:CoverletOutput= --results-directory

I have added reference to coverlet like so: dotnet add package coverlet.collector dotnet add package coverlet.msbuild

When I run the dotnet test command line (as above), there are 3 of my tests which fail and no code coverage file is geneated. I however, want the code coverage report to be generated even if the tests fail (something that Visual Studio IDE based code coverage does).

How can I achieve code coverage results even if some of my tests fail?

Thanks,

Copied from original issue: dotnet/cli#11914

[karanjitsingh]

@saxenark, I believe this is an issue with coverlet, see the thread https://github.com/tonerdo/coverlet/issues/37. Although a fix was merged so not sure what's happening here.

Additionally, coverlet is now integrated into dotnet itself, try to run your coverage via dotnet test with the coverlet integration. See https://github.com/tonerdo/coverlet/blob/master/Documentation/VSTestIntegration.md

If things still don't work for you could you provide me with a sample project with the repro?

[saxenark]

Hi, I have .net core 2.2.0 which I use in my project. What is the default friendly name of the collector?

When I run 'dotnet test --collect:"XPlat Code Coverage"'

I get the error

'Data collection : Could not find data collector 'XPlat Code Coverage'

In short, what are the step by step instructions to use the tool?

Thanks,

[saxenark]

Here is my sample project as requested. When I run the dotnet test command, there is json file in the TestResults folder. ConsoleApp2.zip

[karanjitsingh]

Hi, I have .net core 2.2.0 which I use in my project. What is the default friendly name of the collector?

When I run 'dotnet test --collect:"XPlat Code Coverage"'

I get the error

'Data collection : Could not find data collector 'XPlat Code Coverage'

In short, what are the step by step instructions to use the tool?

Thanks,

You need 2.2.3 to run with coverlet

[saxenark]

Ok. Then what is the command line to generate cobertura coverage file using dotnet test command line for .net core 2.2.0 projects?

I have used the MSBuild integration steps @ https://github.com/tonerdo/coverlet/blob/master/Documentation/MSBuildIntegration.md and with that I can generate the cobertura file format for coverage, except that it does not generate when there are partial failures in UTs.

So, please let me know what you meant by coverlet requiring 2.2.3.

[karanjitsingh]

Ok. Then what is the command line to generate cobertura coverage file using dotnet test command line for .net core 2.2.0 projects?

Coverlet integration for dotnet test is available after dotnet sdk version 2.2.300

To be able to run dotnet test --collect:"XPlat Code Coverage"

1. You need to have the latest sdk version of dotnet, see https://dotnet.microsoft.com/download/dotnet-core/2.2
2. In your project you must reference the nuget package Coverlet.Collector 1.0.1

Sample

PS E:\repros\newproj2> ls

    Directory: E:\repros\newproj2

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----        18/7/2019   4:18 PM                bin
d-----        18/7/2019   4:20 PM                obj
d-----        18/7/2019   4:20 PM                TestResults
-a----        18/7/2019   4:20 PM            506 newproj2.csproj
-a----        18/7/2019   4:18 PM            219 UnitTest1.cs

PS E:\repros\newproj2> dotnet --version
2.2.301
PS E:\repros\newproj2> dotnet test --collect:"XPlat Code Coverage"
Test run for E:\repros\newproj2\bin\Debug\netcoreapp2.2\newproj2.dll(.NETCoreApp,Version=v2.2)
Microsoft (R) Test Execution Command Line Tool Version 16.1.1
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

Attachments:
  E:\repros\newproj2\TestResults\76b5cd0a-f640-4777-9622-6d069d6ed9e7\coverage.cobertura.xml

Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 1.1153 Seconds

By default this will produce a cobertura coverage xml.

Sample csproj

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netcoreapp2.2</TargetFramework>

    <IsPackable>false</IsPackable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.0.1" />
    <PackageReference Include="MSTest.TestAdapter" Version="1.4.0" />
    <PackageReference Include="MSTest.TestFramework" Version="1.4.0" />
    <PackageReference Include="coverlet.collector" Version="1.0.1" />
  </ItemGroup>

</Project>

[karanjitsingh]

@saxenark, I'll close this issue for now. If you're still facing an issue let me know and I'll reopen

[mfaulcon]

Does this imply that code coverage cannot be calculated in the scenario referenced (partially failing tests) when the developer is using VS2017, and therefore has core 2.2.104 installed?

In my build pipeline, I have tried the same setup as @saxenark, with the same result. I have ensure that my build agents are running 2.2.300. If I switch from '/p:CollectCoverage=true' to --collect:"XPlat Code Coverage", code coverage data is being collected, but I have no control over where the output file is generated.