Solution wide coverage - Githubissues

tonerdo commented 5 years ago

Coverlet should be able to get coverage for all test projects in a specified solution file

pantosha commented 5 years ago

FYI, We use coverlet msbuild tool to run tests for all tests projects in a solution file with the next command:

dotnet test solution.sln /p:CollectCoverage=true /maxcpucount:1

The /maxcpucount:1 argument is necessary to prevent parallelizable code instrumentation. Parallelizable instrumentation lock files and block other instance of coverlet

IGx89 commented 5 years ago

That'll produce individual test reports, but won't produce a single combined test report (my impression of what this enhancement is for).

joemey commented 5 years ago

@IGx89 if you run

dotnet test solution.sln /p:CollectCoverage=true /p:Exclude="[xunit*]*" /p:CoverletOutput="../coverage.json" /p:MergeWith="../coverage.json" /maxcpucount:1

it will create a single coverage file and will report the complete coverage after each test.

some things to consider with this approach:

You have to make sure to delete the coverage.json file after the run or the next run will re-use it
the path in CoverletOutput and MergeWith are relative to the test project being run

viceice commented 5 years ago

Directory.Build.props:

<PropertyGroup>
    <CoverletOutput>./coverage/coverage.xml</CoverletOutput>
    <CoverletOutputFormat>opencover</CoverletOutputFormat>
</PropertyGroup>

Directory.Build.targets:

<Target Name="CoverRunPreparation" BeforeTargets="GenerateCoverageResult" Condition="'$(CollectCoverage)' == 'true'">
        <ItemGroup>
            <CoverletOutputPath Include="$(CoverletOutput)" />
        </ItemGroup>
        <PropertyGroup>
            <CoverletOutput Condition="'$(CoverletOutput)' != '' AND '$(TargetFrameworks)' != ''">%(CoverletOutputPath.RootDir)/%(CoverletOutputPath.Directory)/%(CoverletOutputPath.Filename).$(TargetFramework)%(CoverletOutputPath.Extension)</CoverletOutput>
            <CoverletOutput Condition="'$(CoverletOutput)' == '' AND '$(TargetFrameworks)' == ''">$(ProjectDir)coverage.xml</CoverletOutput>
            <CoverletOutput Condition="'$(CoverletOutput)' == '' AND '$(TargetFrameworks)' != ''">$(ProjectDir)coverage.$(TargetFramework).xml</CoverletOutput>
        </PropertyGroup>
</Target>

iron9light commented 5 years ago

@joemey how does MergeWith work with multiple CoverletOutputFormat?

MarcoRossignoli commented 5 years ago

@joemey how does MergeWith work with multiple CoverletOutputFormat?

@iron9light take a look at https://github.com/tonerdo/coverlet/issues/474

lkurzyniec commented 4 years ago

I have tried the solution suggested by @joemey and it works, but only for json, but it does not for opencover.

For this command dotnet test /p:CollectCoverage=true /p:CoverletOutput="../coverage.json" /p:MergeWith="../coverage.json" /maxcpucount:1 /p:CoverletOutputFormat="opencover" I received:

λ dotnet test /p:CollectCoverage=true /p:CoverletOutput="../coverage.json" /p:MergeWith="../coverage.json" /maxcpucount:1 /p:CoverletOutputFormat="opencover"
Test run for C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Api.IntegrationTests\bin\Debug\netcoreapp3.1\HappyCode.NetCoreBoilerplate.Api.IntegrationTests.dll(.NETCoreApp,Version=v3.1)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.

Test Run Successful.
Total tests: 5
     Passed: 5
 Total time: 4,8472 Seconds
Test run for C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Api.UnitTests\bin\Debug\netcoreapp3.1\HappyCode.NetCoreBoilerplate.Api.UnitTests.dll(.NETCoreApp,Version=v3.1)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.

Test Run Successful.
Total tests: 8
     Passed: 8
 Total time: 2,5702 Seconds

Calculating coverage result...
  Generating report '..\coverage.json'

+-----------------------------------+--------+--------+--------+
| Module                            | Line   | Branch | Method |
+-----------------------------------+--------+--------+--------+
| HappyCode.NetCoreBoilerplate.Api  | 13,19% | 11,86% | 19,23% |
+-----------------------------------+--------+--------+--------+
| HappyCode.NetCoreBoilerplate.Core | 0,93%  | 0%     | 3,17%  |
+-----------------------------------+--------+--------+--------+

+---------+-------+--------+--------+
|         | Line  | Branch | Method |
+---------+-------+--------+--------+
| Total   | 6,81% | 7,52%  | 7,86%  |
+---------+-------+--------+--------+
| Average | 7,06% | 5,93%  | 11,2%  |
+---------+-------+--------+--------+

Test run for C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\bin\Debug\netcoreapp3.1\HappyCode.NetCoreBoilerplate.Core.UnitTests.dll(.NETCoreApp,Version=v3.1)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.

Test Run Successful.
Total tests: 4
     Passed: 4
 Total time: 2,3510 Seconds

Calculating coverage result...
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error : Unexpected character encountered while parsing value: <. Path '', line 0, position 0. [C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonTextReader.ParseValue() [C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonTextReader.Read() [C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter) [C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent) [C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType) [C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings) [C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings) [C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Coverlet.Core.Coverage.GetCoverageResult() in C:\Users\toni\Workspace\coverlet\src\coverlet.core\Coverage.cs:line 224 [C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Coverlet.MSbuild.Tasks.CoverageResultTask.Execute() in C:\Users\toni\Workspace\coverlet\src\coverlet.msbuild.tasks\CoverageResultTask.cs:line 91 [C:\git\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]

So it counts coverage for the first project and saves it into json, but for the second project, it can not proceed.

Is it possible to count coverage and merge into one json (according to MergeWith parameter) and then convert the json results into opencover?

MarcoRossignoli commented 4 years ago

@lkurzyniec can you try to run same command but with /p:CoverletOutputFormat="opencover,json" parameter?Similar to https://github.com/tonerdo/coverlet/issues/262#issuecomment-569572522 and remember to use /maxcpucount:1

Is it possible to count coverage and merge into one json (according to MergeWith parameter) and then convert the json results into opencover?

Yes I did same sample on our "sample" section https://github.com/tonerdo/coverlet/blob/master/Documentation/Examples/MSBuild/MergeWith/HowTo.md , you can clone repo and test solution, but you cannot run command "solution wide".

MarcoRossignoli commented 4 years ago

For reference link msbuild task sync discussion https://github.com/microsoft/msbuild/issues/4898 https://github.com/microsoft/msbuild/issues/4954

lkurzyniec commented 4 years ago

Unfortunately, it doesn't work

λ dotnet test /p:CollectCoverage=true /p:CoverletOutput="../coverage.json" /p:MergeWith="../coverage.json" /maxcpucount:1 /p:CoverletOutputFormat=\"json,opencover\"
Test run for C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Api.IntegrationTests\bin\Debug\netcoreapp3.1\HappyCode.NetCoreBoilerplate.Api.IntegrationTests.dll(.NETCoreApp,Version=v3.1)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.

Test Run Successful.
Total tests: 5
     Passed: 5
 Total time: 5,0513 Seconds
Test run for C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Api.UnitTests\bin\Debug\netcoreapp3.1\HappyCode.NetCoreBoilerplate.Api.UnitTests.dll(.NETCoreApp,Version=v3.1)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.

Test Run Successful.
Total tests: 8
     Passed: 8
 Total time: 3,1337 Seconds

Calculating coverage result...
  Generating report '..\coverage.json'
  Generating report '..\coverage.json'

+-----------------------------------+--------+--------+--------+
| Module                            | Line   | Branch | Method |
+-----------------------------------+--------+--------+--------+
| HappyCode.NetCoreBoilerplate.Api  | 13,19% | 11,86% | 19,23% |
+-----------------------------------+--------+--------+--------+
| HappyCode.NetCoreBoilerplate.Core | 35,51% | 35,71% | 44,44% |
+-----------------------------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 24,81% | 19,54% | 37,07% |
+---------+--------+--------+--------+
| Average | 24,35% | 23,78% | 31,83% |
+---------+--------+--------+--------+

Test run for C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\bin\Debug\netcoreapp3.1\HappyCode.NetCoreBoilerplate.Core.UnitTests.dll(.NETCoreApp,Version=v3.1)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.

Test Run Successful.
Total tests: 4
     Passed: 4
 Total time: 2,9084 Seconds

Calculating coverage result...
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error : Unexpected character encountered while parsing value: <. Path '', line 0, position 0. [C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonTextReader.ParseValue() [C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonTextReader.Read() [C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter) [C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent) [C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType) [C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings) [C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings) [C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Coverlet.Core.Coverage.GetCoverageResult() in C:\Users\toni\Workspace\coverlet\src\coverlet.core\Coverage.cs:line 224 [C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]
C:\Users\lkurzyniec\.nuget\packages\coverlet.msbuild\2.7.0\build\coverlet.msbuild.targets(41,5): error :    at Coverlet.MSbuild.Tasks.CoverageResultTask.Execute() in C:\Users\toni\Workspace\coverlet\src\coverlet.msbuild.tasks\CoverageResultTask.cs:line 91 [C:\GIT\netcore-boilerplate\test\HappyCode.NetCoreBoilerplate.Core.UnitTests\HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj]

What's interesting here is that it generate json two times:

Calculating coverage result...
  Generating report '..\coverage.json'
  Generating report '..\coverage.json'

MarcoRossignoli commented 4 years ago

Try with

dotnet test /p:CollectCoverage=true /p:CoverletOutput="../" /p:MergeWith="../coverage.json" /maxcpucount:1 /p:CoverletOutputFormat=\"json,opencover\"

For instance you can try with our samples on repo

D:\git\coverlet\Documentation\Examples\MSBuild\MergeWith
λ dotnet test /p:CollectCoverage=true  /p:CoverletOutput=../CoverageResults/ /p:MergeWith="../CoverageResults/coverage.json" /p:CoverletOutputFormat=\"opencover,json\" -m:1

lkurzyniec commented 4 years ago

@MarcoRossignoli That works fine! In my case, it gives me solution wide results.

Summary

cmd command

dotnet test /p:CollectCoverage=true /p:CoverletOutput="../" /p:MergeWith="../coverage.json" /maxcpucount:1 /p:CoverletOutputFormat=\"json,opencover\"

gives me the same results as following two commands

dotnet test ./test/HappyCode.NetCoreBoilerplate.Core.UnitTests/HappyCode.NetCoreBoilerplate.Core.UnitTests.csproj /p:CollectCoverage=true /p:CoverletOutput=../CoverageResults/

dotnet test ./test/HappyCode.NetCoreBoilerplate.Api.UnitTests/HappyCode.NetCoreBoilerplate.Api.UnitTests.csproj /p:CollectCoverage=true /p:CoverletOutput=../CoverageResults/ /p:MergeWith="../CoverageResults/coverage.json" /p:CoverletOutputFormat="opencover"

What's more, your command dotnet test /p:CollectCoverage=true /p:CoverletOutput=../CoverageResults/ /p:MergeWith="../CoverageResults/coverage.json" /p:CoverletOutputFormat=\"opencover,json\" -m:1 from https://github.com/tonerdo/coverlet/blob/master/Documentation/Examples/MSBuild/MergeWith/HowTo.md also works :)

MarcoRossignoli commented 4 years ago

Glad to hear!

robertlarkins commented 4 years ago

Not sure if this should be a separate question, or asked here, but is it possible to do solution wide coverage using the coverlet .Net global tool? I'm using coverlet in Azure Pipelines and would like it to be system agnostic (Linux, Windows, Mac). The task I'm running looks like this:

  - script: coverlet $(buildArtifact)/tests/core/My.Project.Tests/bin/$(buildConfiguration)/netcoreapp3.1/My.Project.Tests.dll --target "dotnet" --targetargs "test $(buildArtifact)/tests/core/My.Project.Tests --no-build" --format cobertura
    displayName: Run Coverlet to get code coverage

But could it target a directory and run all tests projects under that directory? Or target a solution file?

MarcoRossignoli commented 4 years ago

I'm using coverlet in Azure Pipelines and would like it to be system agnostic (Linux, Windows, Mac)

Why don't use collectors https://github.com/tonerdo/coverlet/blob/master/Documentation/VSTestIntegration.md? Why .NET tool?

Coverlet is cross platform and works cross plat for every "drivers" you're using https://github.com/tonerdo/coverlet#quick-start but the collectors is the preferred way to consume due to integration with vstest platform and you can run command on sln(it generates one coverage file for every test project you have in solution).

robertlarkins commented 4 years ago

At this stage I was only using coverlet as part of the Azure Pipeline, so didn't think the test projects should know or depend on it. Will try installing coverlet.collector as a NuGet package.

robertlarkins commented 4 years ago

So I added coverlet.collector as a NuGet package and got it going for a single test project with this:

  - task: DotNetCoreCLI@2
    displayName: 'Run tests for $(solution) with Cobertura'
    inputs:
      command: test
      projects: '$(buildArtifact)/tests/**/*.csproj'
      arguments: -c $(buildConfiguration) --collect:"XPlat Code Coverage"

  - task: PublishCodeCoverageResults@1
    displayName: 'Publish code coverage'
    inputs:
      codeCoverageTool: cobertura
      summaryFileLocation: '$(Agent.TempDirectory)/*/coverage.cobertura.xml'

But this produces a separate coverage.cobertura.xml file per test project (each in there own guid directory). In this Azure Pipelines context how would I get the individual test results merged together so that a single code coverage publish can be done?

lkurzyniec commented 4 years ago

@robertlarkins my suggestion would be to read both this article and whole this thread. Then try to achieve solution-wide results locally. After that move your solution to Azure DevOps.

robertlarkins commented 4 years ago

@lkurzyniec Will attempt and see how I get on.

MarcoRossignoli commented 4 years ago

At the moment I've removed from documentation Merge support for collectors because we're working on better approach https://github.com/tonerdo/coverlet/pull/704 (merge works well only for sequential tests), because "as is" doesn't work(vstest generates different filenames as you can see).

Actually that parameters is "hidden" but usable, so if you want be sure to not hit https://github.com/tonerdo/coverlet/blob/master/Documentation/KnowIssues.md#1-vstest-stops-process-execution-earlydotnet-test and use merge with vstest you can use a trick of our contributor @p4p3 that with a script move generated coverage files for every test run https://github.com/tonerdo/coverlet/pull/225#issuecomment-573896446

If you don't hit issue above you can use msbuild driver and https://github.com/tonerdo/coverlet/blob/master/Documentation/Examples/MSBuild/MergeWith/HowTo.md

sungam3r commented 4 years ago

@lkurzyniec , @MarcoRossignoli Hi. I am trying to execute the above command but getting an error

PS > dotnet test /p:CollectCoverage=true /p:CoverletOutput=../.coveragedata/cover
age /p:MergeWith="../coverage.json" /p:CoverletOutputFormat=\"json,opencover\" -m:1
MSBUILD : error MSB1006: Property is not valid.
Key: opencover\

MarcoRossignoli commented 4 years ago

@sungam3r you need to translate characters on powershell take a look https://github.com/tonerdo/coverlet/blob/master/Documentation/MSBuildIntegration.md#note-for-powershell--vsts-users %2c for ,

sungam3r commented 4 years ago

Wow! Thanks. I read this section of the documentation, but for some reason I missed it.

MarcoRossignoli commented 4 years ago

No prob

aviita commented 4 years ago

Still having issues in getting merge results and allowing to run ReportGenerator from that. Following is the dotnet test command I am trying to run. Using coverlet 2.9.0 in my test projects. Getting the merge results works fine untill I add that last part regarding output format. But if I only output the default json I cannot get the ReportGenerator to produce anything valid. And if I use other formats in dotnet test, I cannot seem to be able to get results merged.

rm -Recurse CoverageResults/* ; dotnet test /p:CollectCoverage=true /p:Exclude="*UnitTests" /p:MergeWith="../../CoverageResults/coverage.json" /maxcpucount:1 /p:CoverletOutput='../../CoverageResults/coverage.json' /p:CoverletOutputFormat=\"json%2copencover\"

C:\Users\username\.nuget\packages\coverlet.msbuild\2.9.0\build\coverlet.msbuild.targets(60,5): error : Unexpected character encountered while parsing value: <. Path '', line 0, position 0. [C:\MyProject\Source\MyCompany.Domain.UnitTests\MyCompany.Domain.UnitTests.csproj] C:\Users\username\.nuget\packages\coverlet.msbuild\2.9.0\build\coverlet.msbuild.targets(60,5): error : at Newtonsoft.Json.JsonTextReader.ParseValue() [C:\MyProject\Source\MyCompany.Domain.UnitTests\MyCompany.Domain.UnitTests.csproj] C:\Users\username\.nuget\packages\coverlet.msbuild\2.9.0\build\coverlet.msbuild.targets(60,5): error : at Newtonsoft.Json.JsonReader.ReadAndMoveToContent() [C:\MyProject\Source\MyCompany.Domain.UnitTests\MyCompany.Domain.UnitTests.csproj] C:\Users\username\.nuget\packages\coverlet.msbuild\2.9.0\build\coverlet.msbuild.targets(60,5): error : at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ReadForType(JsonReader reader, JsonContract contract, Boolean hasConverter) [C:\MyProject\Source\MyCompany.Domain.UnitTests\MyCompany.Domain.UnitTests.csproj] C:\Users\username\.nuget\packages\coverlet.msbuild\2.9.0\build\coverlet.msbuild.targets(60,5): error : at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent) [C:\MyProject\Source\MyCompany.Domain.UnitTests\MyCompany.Domain.UnitTests.csproj] C:\Users\username\.nuget\packages\coverlet.msbuild\2.9.0\build\coverlet.msbuild.targets(60,5): error : at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType) [C:\MyProject\Source\MyCompany.Domain.UnitTests\MyCompany.Domain.UnitTests.csproj]

aviita commented 4 years ago

And I have tried different variations of this CoverletOutputFormat string, all resulting to error of some sort with command above:

/p:CoverletOutputFormat=\"json%2copencover\" 

/p:CoverletOutputFormat="json%2copencover"

/p:CoverletOutputFormat='json,opencover'

/p:CoverletOutputFormat='json%2copencover'

aviita commented 4 years ago

And found my problem, so I am not supposed to put the json file, but only a path to CoverletOutput param. Defining json file likely caused coverlet xml to be written to the json file:

rm -Recurse CoverageResults/* ; dotnet test /p:CollectCoverage=true /p:Exclude="*UnitTests" /p:MergeWith="../../CoverageResults/coverage.json" /maxcpucount:1 /p:CoverletOutput='../../CoverageResults/' /p:CoverletOutputFormat='\"json%2copencover\"'

MarcoRossignoli commented 4 years ago

@aviita sorry for the delay(very busy time), glad to hear you solve your problem.

kreghek commented 3 years ago

https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/Examples/MSBuild/MergeWith/HowTo.md

It was helpfull for me. I use coverall and special solutions for unit and functional tests. https://github.com/kreghek/Zilon_Roguelike/blob/362997fa7216e1ba77c18ea120580e29b4ada16e/.github/workflows/coverall.yml

mgiles commented 3 years ago

Solution-wide coverage reports seem to work well using MergeWith, but Threshold doesn't seem to work the way I'd expect with multiple projects. For example, if I do something like this:

dotnet test /p:CollectCoverage=true /p:CoverletOutput="../CoverageResults/" /p:MergeWith="../CoverageResults/coverage.json" /p:CoverletOutputFormat="\"cobertura,json\"" /p:Threshold=$MIN_CODE_COVERAGE_PERCENTAGE /p:ThresholdType=line /p:ThresholdStat=total

The solution-wide reports seem to be okay, but the threshold is checked after each test project is run, on the total coverage so far. Depending on the order test projects are run in this can cause our CI pipeline to fail even if overall coverage is actually above the threshold.

I found a workaround that involves running tests multiple times, but that's kind of hacky and adds unnecessary run time. Is there a better supported approach to accurate solution-wide thresholds? #598 seems relevant, but it was closed as a duplicate of this one so I'm asking here. Thanks in advance for any advice, and thanks for your work maintaining coverlet.

daveMueller commented 3 years ago

@mgiles As far as I know we currently don't have any other support for solution-wide thresholds. https://github.com/coverlet-coverage/coverlet/issues/1227 is a recent feature request that seems to be related. I think it would be good if you could add your thoughts there.

rcollette commented 3 years ago

FWIW, my team only need thresholds at a project level (#1227) but wish to report coverage at a solution level. Hopefully it should be obvious that test projects should not be included in threshold checks nor in reporting. Setting thresholds, inclusions/exclusions, etc at a solution level is going to make for one very big command line though.

coverlet-coverage / coverlet

Solution wide coverage #357

Summary