Add support for dotnet test --collect:"Code Coverage"

harshjain2 commented 7 years ago

Description

Add support for code coverage through dotnet test on Windows OS machine. dotnet test --collect:"Code Coverage"

ahaleiii commented 3 years ago

I am feeling a lot of the same pain points listed in this thread, and I agree they all seem to come back to Azure DevOps Pipeline integration. I think Coverlet is a perfectly viable solution for cross-platform code coverage. The pain isn't that Coverlet can't produce a binary coverage file, or even that vstest can't collect coverage on Linux; the pain is that Azure DevOps requires a binary coverage file in order to use so many of the features it's trying to develop. Unfortunately I do not see this issue resolving that.

I think @thiagocamargos's comment put it succinctly:

When people here tells others to use coverlet instead vstest coverage for linux builds, you are also telling to use coverlet for windows and ignore the vstest feature. And if that is true, then this feature may be considered useless by those who appreciate a cross-platform framework. Thus, it is better to remove it from the application to minimize development complexity.
Telling people to use coverlet is only deprioritizing this issue, keeping it open forever and misguiding people into thinking this will be done. It has been almost 3 years since this was opened and there is still no clear answer from the core team. I have been following it and being notified since the beginning and I'm wondering if 3 years of a open issue is not enough yet.

jakubch1 commented 3 years ago

Sorry for delay in providing cross platform code coverage. We are still porting a lot of quite old C++ code to .NET Standard. I believe we are done in 80%. At the beginning we will be able to support Ubuntu and Alpine distributions. Do you use other distributions?

@ganesp @ShreyasRmsft is CC related logic in Azure Devops cross plat? We can already provide you cross plat library for analyzing .coverage file.

jakubch1 commented 3 years ago

We added preview support for dynamic code coverage on Ubuntu and Alpine. To try it out you need:

Install dotnet sdk >= 5.0.200
Add latest Microsoft.CodeCoverage nuget package to your test project. For example: https://dev.azure.com/dnceng/public/_packaging?_a=package&feed=test-tools&package=Microsoft.CodeCoverage&protocolType=NuGet&version=17.0.0-preview-20210419-05&view=overview
On Alpine additionally you have to install few dependencies: apk add libxml2 libuuid libunwind

ElvenSpellmaker commented 3 years ago

Hey @jakubch1, thanks for the update. We'll give this a try thanks. ☺️

ElvenSpellmaker commented 3 years ago

Do we need to supply anything special on the build parameters?

jakubch1 commented 3 years ago

You can use: dotnet test --collect "Code Coverage"

ElvenSpellmaker commented 3 years ago

Thanks @jakubch1, I'll give this a try. Also is it possible to output both "Code Coverage" and cobertura/open cover formats?

ElvenSpellmaker commented 3 years ago

Hey @jakubch1, I tried adding --collect "Code Coverage" and the package you linked does indeed generate .coverage files now, but it overrides the nice Cobertura output in the Code Coverage tab.

I just can't see us switching until we can have a nice Code Coverage tab as well.

Before the change:

After the change:

davesmits commented 3 years ago

First of all. Awesome that coverage files now work on linux! Big win we can use the PR validation thing of Azure DevOps again.

I just tried to combine coverlet and coverage file to have best of both world but unfortunately it doesnt solve the issue. Azure Devops keeps showing the download to the coverage file :(. Is there a way to have 'both'? the coverage file so the PR validation works and nice reports?

Is there also a easy way to exclude dlls and namespaces for coverlet adding the parameter or do i need to introduce the runsettings xml again?

John0King commented 3 years ago

@pavelhorak

We also plan to support non-binary report format(s) so it will be easily possible to display code coverage results, especially on Azure DevOps.

I'm wonder that will dotnet test support any non-binary report format(s) ? or the plan is only for azure devops.

IMO, the binary format do not have any advantage than text file, ~~and also unfriendly for "diff" in source controll ~~ , and unfriendly for human read.

jakubch1 commented 3 years ago

@John0King @ElvenSpellmaker supporting non-binary format for Code Coverage is in our backlog: microsoft/vstest#2874

ElvenSpellmaker commented 3 years ago

Thanks, I'll be waiting! 🙂

sergiocyklpoint commented 3 years ago

@jakubch1 do you have any plan to release this change into sdk 3.1? Or only sdk 5 and above?

jakubch1 commented 3 years ago

Some older versions of vstest.console on Unix have issue to print correctly summary of tests if there is an attachment. So you can try including latest Microsoft.CodeCoverage package to 3.1 project and try: dotnet test --collect "Code Coverage", but you can hit this issue.

MarkusRodler commented 3 years ago

At the beginning we will be able to support Ubuntu and Alpine distributions. Do you use other distributions?

@jakubch1 If Ubuntu is supported, Debian would be also nice

jakubch1 commented 3 years ago

@ElvenSpellmaker @davesmits I had some talks with Azure DevOps team. We are still waiting for some features to be delivered in this area. Currently you can't publish .coverage file and cobertura together.

As workaround you can use coverage splitting by:

1) upgrade to: https://dev.azure.com/dnceng/public/_packaging?_a=package&feed=test-tools&package=Microsoft.CodeCoverage&protocolType=NuGet&version=17.0.0-preview-20210511-05&view=overview 2) create runsettings with:

<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
        <Configuration>
...
               <SplitCoverage>True</SplitCoverage>
...
            <CodeCoverage>
                    ....
             </CodeCoverage>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>

or from command line

dotnet test --collect "CodeCoverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.SplitCoverage="True"

You will see 2 attachments (covb and covx) for each module. On Azure DevOps this will enable such code coverage summary:

MicrosoftTeams-image (1)

PR support is also supported when uploading .covx and .covb files.

jakubch1 commented 3 years ago

@MarkusRodler we added support for more distributions in version: https://dev.azure.com/dnceng/public/_packaging?_a=package&feed=test-tools&package=Microsoft.CodeCoverage&protocolType=NuGet&version=17.0.0-preview-20210518-01&view=overview

We verified that dynamic code coverage works on:

OS	Version	Architectures	Dependencies	Lifecycle
Alpine Linux	3.11+	x64	`apk add libuuid libxml2 libunwind`	Alpine
CentOS	8+	x64	`dnf install --setopt tsflags=nodocs --refresh -y dnf-plugins-core epel-release && dnf config-manager --set-enabled powertools && dnf install --setopt tsflags=nodocs -y libunwind`	CentOS
Debian	10+	x64	`apt-get install libunwind8 libxml2`	Debian
Fedora	32+	x64		Fedora
Linux Mint	19.3+	x64	`apt-get install libunwind8`	Linux Mint
openSUSE	tumbleweed	x64	`zypper install libunwind libuuid1`	OpenSUSE
Ubuntu	18.04+	x64	`apt-get install libunwind8 libxml2`	Ubuntu

Chainsaw-does-brr commented 3 years ago

@jakubch1 , tested on ubuntu 20.04 with azure devops vs local Fine Code Coverage (Coverlet). To be honest, I'm not sure if it's related to linux support specifically (don't have a win machine to compare), but:

in FCC we got 72,2% coverage
in Azure DevOps we got only 20,01% on the same solution.

The solution itself is a monorepo with a lot of dependencies between projects. Most of the projects have their own test projects. So this probably means that when you run dotnet test my.sln, coverage counts the lines of project dependencies, but does not append the results of tests of those dependencies neither during run, nor at the backend (Azure Devops).

jakubch1 commented 3 years ago

@Chainsaw-does-brr could you please provide both coverage reports?

Chainsaw-does-brr commented 3 years ago

@jakubch1 i wish i could, but unfortunately can't.

You can simulate the setup with having both:

project A being tested by project TA
project B that depends on project A and being tested by TB.

I can provide more detail though. This is my 1st encounter with code coverage, so I'm rediscovering the wheel here, bear with me. Apparently, all the methods generate separate coverage reports per project. All of those reports have missing data on coverage from dependencies, but. FCC under the hood uses ReportGenerator to merge reports from multiple projects.

Azure devops on the other hand, has 2 methods of uploading code coverage data:

'Publish code coverage results' that can't merge reports on it's on and will take only the 1st one that matches the filter.
dotnet CLI - which, i suspect, has the same issue, but I'm not sure about this one.

So, basically, you are expected to use ReportGenerator to merge results from multiple test projects on your own if you want proper coverage data.

On a side note: I've just tried merging .coverage files with ReportGenerator and, apparently, they must be converted to .xml 1st. Found microsoft/vstest#1939 and microsoft/vstest#2424 that request adding output formats and mention that formats will be a part of this issue. Are there any updates on this by chance?

jakubch1 commented 3 years ago

@Chainsaw-does-brr Azure DevOps pipelines are able to merge .coverage files. To make it working make sure that running dotnet CLI is producing trx file for each test project (add --logger trx and also --collect "Code Coverage"). Then add step to publish all your trx files:

    - task: PublishTestResults@2
      displayName: 'Publish Test Results **/*.trx'
      inputs:
        testResultsFormat: VSTest
        testResultsFiles: '**/*.trx'
      condition: succeededOrFailed()

Each trx file should contain link to .coverage file (you need latest stable .NET sdk). Azure DevOps will merge .coverage files. You can also configure PR coverage support: https://docs.microsoft.com/en-us/azure/devops/pipelines/test/codecoverage-for-pullrequests?view=azure-devops

Chainsaw-does-brr commented 3 years ago

@jakubch1 It produces the same output as dotnet cli with publishTestResults: true. In my case it's 20.11%

While doing coverlet -> merge with ReportGenerator -> Publish Code Coverage result route yields: 52,64% (excluded test assemblies and AutoGeneratedAttribute).

ps: thanks for the tip on including coverage in pr, was wondering about this recently. :)

jakubch1 commented 3 years ago

@Chainsaw-does-brr we can't investigate it without coverage reports. Feel free to generate feedback here: https://developercommunity.visualstudio.com/search?space=8.

Keviento commented 3 years ago

@jakubch1 That's some awesome news. Do you have any idea when the NuGet will be available on public nuget.org feed ? Thanks !

jakubch1 commented 3 years ago

It's already available here: https://www.nuget.org/packages/Microsoft.CodeCoverage/16.10.0 But I recommend using this version: https://dev.azure.com/dnceng/public/_packaging?_a=package&feed=test-tools&package=Microsoft.CodeCoverage&protocolType=NuGet&version=17.0.0-preview-20210607-01 (contains fixes and support for more linux distributions). Both feeds are public

RehanSaeed commented 3 years ago

It's worth noting that Mac doesn't seem to be supported, you get this error which seems to fail the build:

Data collector 'Code Coverage' message: No code coverage data available. Code coverage is currently supported only on Windows, Ubuntu and Alpine..
[xUnit.net 00:00:12.03]     [FATAL ERROR] System.AppDomainUnloadedException

https://ci.appveyor.com/project/RehanSaeed/schema-net/builds/39525934/job/51b68p0ks7aj9hpp

If the platform is unsupported, ideally it shouldn't fail the build.

jakubch1 commented 3 years ago

@RehanSaeed I'm not sure if failure is related. Could you please run it with --diag log.txt and send us back all log* files which will be generated?

daneshka commented 3 years ago

When I run on this container mcr.microsoft.com/dotnet/sdk:5.0-alpine I get this error

$ dev/CodeCoverage/CodeCoverage.exe analyze   /output:coverage_result.xml $coverage_filename
dev/CodeCoverage/CodeCoverage.exe: line 1: MZ����@��: not found
dev/CodeCoverage/CodeCoverage.exe: line 2: $���: not found
dev/CodeCoverage/CodeCoverage.exe: line 2: �֚: not found
dev/CodeCoverage/CodeCoverage.exe: line 2: �֚: not found
dev/CodeCoverage/CodeCoverage.exe: line 2: �֚2�E�+�֚W�כ?�֚W�ӛ: not found
dev/CodeCoverage/CodeCoverage.exe: line 3: �֚W�қ4�֚W�՛1�֚i�ӛ:�֚i�қ:�֚/�қ9�֚/�כ.�֚: not found
dev/CodeCoverage/CodeCoverage.exe: line 3: �ך��֚��ߛ: not found
dev/CodeCoverage/CodeCoverage.exe: line 4: syntax error: unexpected ")"

I used the exe file in this package microsoft.codecoverage.17.0.0-preview-20210712-03.nupkg

jakubch1 commented 3 years ago

CodeCoverage.exe is windows only. Please use this: https://github.com/microsoft/vstest/issues/2424#issuecomment-870313818

ldwedari commented 3 years ago

@ElvenSpellmaker @davesmits I had some talks with Azure DevOps team. We are still waiting for some features to be delivered in this area. Currently you can't publish .coverage file and cobertura together.

As workaround you can use coverage splitting by...

@jakubch1 I'm also looking for having cobertura and .coverage published at the same time. I tried the SplitCoverage setting but it only generates a report per dll. Clicking on the links downloads the coverage file for Visual Studio. Is there a issue number that I can follow to know when this will be implemented? From what you say: "We are still waiting for some features to be delivered in this area" It seems that this will be implemented soon. Is that right?

jakubch1 commented 3 years ago

My team is responsible for just generating coverage report from test run. Publishing and displaying is part of Azure DevOps team. You can try to find or create issues under repo: https://github.com/microsoft/azure-pipelines-tasks

I found few issues related: 1) https://github.com/microsoft/azure-pipelines-tasks/issues/13632 2) https://github.com/microsoft/azure-pipelines-tasks/issues/13448 3) https://github.com/microsoft/azure-pipelines-tasks/issues/13257

divyanshumehta commented 3 years ago

Is this yet not implemented?. I ran this and I get the same warning in linux agents till now, and my Check build quality steps keeps waiting for coverage results. The tests are running but coverage is not getting collected.

jakubch1 commented 3 years ago

@divyanshumehta what is your OS, .NET version and Microsoft.NET.Test.Sdk version in csproj?

divyanshumehta commented 3 years ago

I am running on the AzureDevOps pipelines agents. I am running on linux ubuntu-20 agent .NET Core 3.1 and Microsoft.NET.Test.Sdk" Version="16.8.3". When I run the same pipeline in windows agents then it works fine.

jakubch1 commented 3 years ago

To use linux dynamic code coverage you need to upgrade to latest stable .NET 5.0 and change test projects to Microsoft.NET.Test.Sdk 16.11.0.

IvanAlekseev commented 3 years ago

@jakubch1 I am trying to run

dotnet test --collect "Code Coverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.SplitCoverage="True"

When I reference 17.0 preview version like this

    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.0.0-preview-20210607-01" />

It produces a bunch of .covx and .covb files. But when I reference 16.11 I only get single .converage file. You mentioned here that it is supported from 16.10. Am I doning something wrong or I just misunderstood?

jakubch1 commented 3 years ago

To use SplitCoverage with 16.11 you need to generate .runsettings and include:

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
        <Configuration>
           <CodeCoverage SplitCoverage="true">
           </CodeCoverage>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>

Command that you provided for splitting works only since version 17.

IvanAlekseev commented 3 years ago

Thanks @jakubch1. I still couldn make it work with .runsettings file, but managed to get azure devops run with preview package on linux and got these results

Great that it works, but it doesn't look as good as coverlet report 😞. I guess it is something for azure devops team to improve.

It needs only one dotnet test to run tests and publish coverage

      - task: DotNetCoreCLI@2
        displayName: Test Projects
        inputs:
          command: test
          projects: $(testProjects)
          publishTestResults: true
          arguments: '--no-build -c $(buildConfiguration) --logger trx --collect "Code Coverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.SplitCoverage="True"'

jakubch1 commented 3 years ago

We are working on https://github.com/microsoft/vstest/issues/2874 . When this is implemented you can get same results as with coverlet. We will be supporting cobertura format.

ElvenSpellmaker commented 3 years ago

We are working on microsoft/vstest#2874 . When this is implemented you can get same results as with coverlet. We will be supporting cobertura format.

Wooo I look forward to that!

fhnaseer commented 2 years ago

Cobertura support is added to code coverage. It is available in the following version. https://dev.azure.com/dnceng/public/_packaging?_a=package&feed=test-tools&package=Microsoft.CodeCoverage&protocolType=NuGet&version=17.1.0-preview-20211118-03

To get cobertura report, add <Format>Cobertura</Format> in the runsettings.

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0">
        <Configuration>
           <Format>Cobertura</Format>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>

Available format options:

Coverage (default binary file format) Cobertura Xml (Same as codecoverage.exe /analyze report.coverage output).

jakubch1 commented 2 years ago

In package https://dev.azure.com/dnceng/public/_packaging?_a=package&feed=test-tools&package=Microsoft.CodeCoverage&protocolType=NuGet&version=17.2.0-preview-20211214-01&view=overview we have added experimental support for macOS x64. We also removed libunwind and libuuid dependencies.

Currently we support: OS	Version	Architectures	Dependencies	Lifecycle
Alpine Linux	3.11+	x64	`apk add libxml2`	Alpine
CentOS	8+	x64		CentOS
Debian	10+	x64	`apt-get install libxml2`	Debian
Fedora	32+	x64		Fedora
Linux Mint	19.3+	x64		Linux Mint
openSUSE	tumbleweed	x64		OpenSUSE
Ubuntu	18.04+	x64	`apt-get install libxml2`	Ubuntu
macOS	10.15+	x64

gioce90 commented 2 years ago

Thanks to the preview version here indicated (now I use the 17.1.0-preview-20211130-02 version) I'm now able to generate code coverage in two different way for obtains Full coverage and the Diff coverage (for Pull Requests) on Azure DevOps. I achieved this using the new Microsoft native "Code coverage" and using two different pipelines and I want to show you how.

The problem:

If I understood well, in Azure DevOps if you want the "Code coverage" tab on the build page and you want to visualize the report directly in a friendly way (and not to download it), your pipeline have to generate files in supported formats, like cobertura.xml, as poorly described here. But in Azure DevOps if you also want a Diff coverage policy on the Pull Request, you have to generate a build with the native .coverage file (that presented in this thread). But in this way you obtain the Diff policy but you have to say goodbye to the friendly full report on the build (because the .coverage is only downloadable and is not viewable). Moreover it seems that if you generate together these different format files, only the .coverage file is taken in consideration by Azure DevOps "Code coverage" tab (https://github.com/microsoft/azure-pipelines-tasks/issues/13632#issuecomment-903107159)

Solution:

My solution involves in creating two different pipelines.

The first pipeline is for the Full Coverage. It runs for every commit on the repository. The coverage is generated with the native MS coverage but files are in the cobertura format and is easy to generate and publish a report. The yml is this:

trigger:
  batch: true
  branches:
    include:
    - feature/*

pool: 
  vmImage: 'windows-latest'

variables:
  buildConfiguration: 'Release'
  disable.coverage.autogenerate: 'true' # for report generator

steps:

# restore step
  - task: DotNetCoreCLI@2
    displayName: 'dotnet restore'
    inputs:
      command: 'restore'
      projects: '**/*.csproj'
      configuration: $(BuildConfiguration)
      feedsToUse: 'select'
      vstsFeed: '...ourFeedId...'

# test step
  # task: test with native code coverage (cobertura format) with publish coverage excluded:
  # Usefull for full coverage, but useless for PR diff coverage.
  - task: DotNetCoreCLI@2
    displayName: 'DotNetCoreCLI Test with report (cobertura format)'
    inputs:
      command: test
      projects: '**/**.Tests.csproj'
      arguments: '--no-restore --collect "Code Coverage" --logger trx --results-directory "TestResults/Coverage/" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura'
      publishTestResults: false

# report generator, publish tests and code coverage step
  - task: reportgenerator@5
    inputs:
      reports: '$(Build.SourcesDirectory)/TestResults/Coverage/**/**.cobertura.xml'
      targetdir: '$(Build.SourcesDirectory)/TestResults/Coverage/Reports'
      reporttypes: 'HtmlInline_AzurePipelines_Dark;Cobertura'
      assemblyfilters: '+My.Company.**;-My.Company.**.Tests'

  - task: PublishTestResults@2
    inputs:
      testResultsFormat: 'VSTest'
      testResultsFiles: '**/*.trx'
      mergeTestResults: true
      failTaskOnFailedTests: false

  - task: PublishCodeCoverageResults@1
    inputs:
      codeCoverageTool: 'cobertura'
      summaryFileLocation: $(Build.SourcesDirectory)/TestResults/Coverage/Reports/Cobertura.xml
      reportDirectory: $(Build.SourcesDirectory)/TestResults/Coverage/Reports
      failIfCoverageEmpty: false

This works really well. Please note that:

I can easily set Format=cobertura in the DotNetCoreCLI@2 (test) task without create a .runsettings file.
The task reportgenerator@5 have the assemblyfilters field with that input because in this way ONLY projects of my interest are included (all my solution's assemblies and don't other stuff like Moq, Humanizer and other added nuget stuff...) and without my Tests projects. I think that can be interests of many of you. This also fix the problem https://github.com/microsoft/azure-pipelines-tasks/issues/13703 .
I also talked here: https://github.com/microsoft/azure-pipelines-tasks/issues/13632#issuecomment-1023496286

The second pipeline is for the Diff Coverage on the Pull Request. Obviously I first set everything up as the documentation says. The yml is the follow:

trigger:
  batch: true
  branches:
    include:
    - develop

pool:
  vmImage: 'windows-latest'

variables:
  buildConfiguration: 'Release'
  disable.coverage.autogenerate: 'true' # for report generator

steps:

# restore step
  - task: DotNetCoreCLI@2
    displayName: 'dotnet restore'
    inputs:
      command: 'restore'
      projects: '**/*.csproj'
      configuration: $(BuildConfiguration)
      feedsToUse: 'select'
      vstsFeed: '...ourFeedId...'

# test step
  # task: test with default code coverage (.coverage format) and publish coverage included:
  # Usefull for PR diff coverage, but not for full coverage.
  - task: DotNetCoreCLI@2
    displayName: 'DotNetCoreCLI Test with report (.coverage format)'
    inputs:
      command: test
      projects: '**/**.Tests.csproj'
      arguments: '--collect "Code Coverage"'
      publishTestResults: true

Now, this works but is not good enough. Is lovely to have PR Diff coverage policy (that part works well) but I find awful to have two pipeline instead of one. There have too much in common. And is strange that the native Microsoft .coverage format is not supported by Azure DevOps for report visualization on the build. Another weird thing: in the second pipeline I still haven't found a way to properly filter the assemblies on which to run code coverage. I see that files in Test projects are also considered in my Pull Requests and, even worse, if I download the .coverage file generated I also see things like xUnit.dll, moq.dll, etc. (I also talked here)

Apart from these problems to which I hope to find a solution soon, I believe I have successfully obtained the two different code coverage. Tell me what you think or if you think there are improvements to be made.

2022/02/19 - EDIT :

Now I use the new stable version (17.1.0) of the Microsoft.NET.Test.Sdk nuget package.
I noticed that the old version of ReportGenerator get some problem with the native dotnet-coverage. With the new version (reportgenerator@5) is fine.
I noticed that with the use of a .runsettings file is possible to filter some assemblies for both diff and full code coverage. So now the field assemblyfilters: '+My.Company.**;-My.Company.**.Tests' on the reportgenerator@5 task is no needs anymore.

jakubch1 commented 2 years ago

I've created two new issues to support code coverage on Linux ARM and macOS ARM64 https://github.com/microsoft/vstest/issues/3426 https://github.com/microsoft/vstest/issues/3427.

I'm closing this issue as other cases are already covered. Please use thumbs-up on above issues so we can prioritize it.

MS-Manikandan commented 2 years ago

The second pipeline is for the Diff Coverage on the Pull Request. The yml is the follow:

trigger:
  batch: true
  branches:
    include:
    - develop

pool:
  vmImage: 'windows-latest'

variables:
  buildConfiguration: 'Release'
  disable.coverage.autogenerate: 'true' # for report generator

steps:

# restore step
  - task: DotNetCoreCLI@2
    displayName: 'dotnet restore'
    inputs:
      command: 'restore'
      projects: '**/*.csproj'
      configuration: $(BuildConfiguration)
      feedsToUse: 'select'
      vstsFeed: '...ourFeedId...'

# test step
  # task: test with default code coverage (.coverage format) and publish coverage included:
  # Usefull for PR diff coverage, but not for full coverage.
  - task: DotNetCoreCLI@2
    displayName: 'DotNetCoreCLI Test with report (.coverage format)'
    inputs:
      command: test
      projects: '**/**.Tests.csproj'
      arguments: '--collect "Code Coverage"'
      publishTestResults: true

Hi @gioce90, I have a slight clarification does the above pull request builds run all [old+newly added] tests or only the newly added tests would be ran when a PR is raised?

gioce90 commented 2 years ago

Hi @ManikandanMani , this builds and run all them (old and new tests).

simonachmueller commented 2 years ago

Hey, @nehsharmMS mentioned in her comment that Microsoft has a new preview version of the Azure DevOps for code coverage, as I understood now it's possible to display Code Coverage tab and info when just publishing .coverage file. Is that true?

gioce90 commented 2 years ago

Hey, @nehsharmMS mentioned in her comment that Microsoft has a new preview version of the Azure DevOps for code coverage, as I understood now it's possible to display Code Coverage tab and info when just publishing .coverage file. Is that true?

I don't know @simonachmueller , I'm not so sure about that....

sjd2021 commented 2 years ago

I could be missing something, but would anyone be able to explain why this works in linux for one project and not another project where the only difference is MSTest vs XUnit? The error indicates that it'll only work on Windows.

jakubch1 commented 1 year ago

For version >= 17.5.0-preview-20221021-01 dotnet test --collect "Code Coverage" is fully cross plat.

https://dev.azure.com/dnceng/public/_artifacts/feed/test-tools/NuGet/Microsoft.CodeCoverage/overview/17.5.0-preview-20221021-01

microsoft / vstest