Add support for dotnet test --collect:"Code Coverage"

[harshjain2]

Description

Add support for code coverage through dotnet test on Windows OS machine. dotnet test --collect:"Code Coverage"

[janaka]

@pvlakshm just wanted to vote for *.nix support. We (at Hudl) are moving our backend from .NET Framework on Windows to .NET Core on Docker on Linux. Having to generate coverage on Windows build agents which is not ideal. Coverlet and Altcover seem to be options but not sure how good the coverage reports are.

[zfilipov]

@pvlakshm The RFC states that support is expected to be in version 15.8 onward, expected to release Q3 2018. Is this still the case?

In the meantime, is there any way to get code coverage results into a CI/CD pipeline for .NET Core 2.0 test scenarios?

[mbenua-mparticle]

@zfilipov DotCover just released a patch as of maybe a week ago that is now correctly supporting .NET Core 2.0. You do have test on Windows, but at least you can collect coverage metrics without a ton of wrangling on CI/CD.

[ahaleiii]

@zfilipov We are using coverlet with our .NET Core 2.0 applications. It works with our VSTS CI/CD pipeline (automated).

[zfilipov]

@ahaleiii @mbenua-mparticle Thank you for your input!

@ahaleiii When you generate the code coverage JSON file, is there another task you need to add to the pipeline to upload the coverage results to TFS / VSTS?

[pvlakshm]

@janaka, noted your vote. @zfilipov, yes it is very much the case. We should be shipping a "preview" to NuGet very soon.

[ahaleiii]

@zfilipov We are generating the coverage in Cobertura format. Yes, there is another task in VSTS used to upload the results (Publish Code Coverage Results).

dotnet test task arguments for working with coverlet: --configuration $(BuildConfiguration) --no-build --filter "Category=Unit|Category=UnitTest" /p:CollectCoverage=true /p:CoverletOutputFormat=cobertura /p:CoverletOutputDirectory=test-results

[pvlakshm]

We have shipped a preview version of the Test SDK with support for code coverage for .NET Core (Windows) projects generating portable/Full PDBs.
As called out in the RFC, this will emit a .coverage file and will require VS to view the results. Enhancements to viewing the results, and support for Linux/Mac will be taken up separately.
Here is a sample that can be used to try out the above with MSTest, and xUnit, and NUnit based tests: https://github.com/pvlakshm/Samples/tree/master/NetCoreSampleForCC.

[eduherminio]

VS or VS Enterprise, @pvlakshm?

[pvlakshm]

VS Enterprise has the viewer.

[zfilipov]

@pvlakshm Thank you for the update! Will we be able to upload the code coverage results to VSTS/TFS as part of a CI/CD pipeline for viewing?

[kennethkryger]

@zfilipov I've got it running on TFS 2018 with code coverage results in the build reports.

I've added the .NET Core task to the process with the following properties:

• Version: 2.* (preivew)
• Command: test ;-)
• Path to project(s): **\*.Tests.csproj
• Arguments: --logger "trx;LogFileName=test-results.trx" --collect "Code Coverage"

After the task above I've got a Publish Test Results task with the following properties:

• Test results format: VSTest (even though my unit tests are written using XUnit!!!
• Test results files: **\*.trx
• Search folder: $(System.DefaultWorkingDirectory)
• Under "Advanced" the "Upload test results files" is checked

[Coder3333]

@kennethkryger I added those steps with TFS Online, and I get test results, but no code coverage. There is a warning in the test step, "Data collection : Could not find data collector 'Code Coverage'". Do you have something additional installed to get the code coverage?

[kennethkryger]

@Coder3333 That might be because the preview SDK isn't installed on the Hosted VSTS build agents? If you look at the logs of your builds, you should be able to see what version of the Test Execution Command Line Tool your build i using... Mine says: 15.8.0-preview-20180605-02

You might be able to install the preview SDK? See more info here (I haven't tried it!)

However, I see the same error as you do, if I don't add the version above to my Unit Test project - e.g. if I reference version 15.7.2...

[zfilipov]

@kennethkryger Thanks for the detailed steps, is this for the Microsoft preview SDK? Or are you using Coverlet for this?

[kennethkryger]

@zfilipov I'm using the Microsoft preview SDK.

[zfilipov]

@kennethkryger Thanks Kenneth. Are you able to publish your code coverage results to TFS as well?

[kennethkryger]

@zfilipov Yes I am - works perfectly! I haven't tried VSTS - only TFS (on-premise).

BTW: For the .NET Core task in my build definition I had to change the following property, in order to get my code coverage results into SonarQube:

Arguments: --logger "trx" --collect "Code Coverage" (removed ;LogFileName=test-results.trx)

[clairernovotny]

Very much looking forward to this, but I must add that we need support for embedded portable pdb's. All of our projects use those for user ease and it's critical that we get coverage for those too.

We've been relying on third party tools that have long supported this, but are eager to use the framework tools.

[ghost]

Is it possible to generate the coverage file as @kennethkryger suggested with VS Professional? My idea is to generate the report using VS Professional installed in a agent, and then publish the results to SonarQube. I don't need the Enterprise Version to see the report, is it required to be able to generate it?

[RoCore]

@mraigonsp you can install vs enterprise edition on your CI machine. the agent do not require VS licence at all. do someone know when vs 15.8 will be released?

[ghost]

Ok @RomanKernSW so we only require a license when "opening and using VS" right? so, isn't a licence necessary to use the build tools? That solution may work for us now, but I can not understand how such an essential feature as code coverage is not available in a Professional licensing.

[zfilipov]

@kennethkryger Kenneth, I was trying to follow your example using the preview Microsoft Test SDK but am getting same the error as @Coder3333 . My test executor shows: "Microsoft (R) Test Execution Command Line Tool Version 15.7.0" but I have installed the preview version of the SDK

[kennethkryger]

@zfilipov Sorry for not getting back before now... I realize that I've might have forgotten a rather important step - the .NET Core Tool Installer

This is the first task in my build definition!

I've set the properties to:

• Package to install: SDK (contains runtime)
• Version: 2.1.400-preview-008975

[JohnGalt1717]

Any update to this working properly under the sdk on Linux? Right now you can't get code coverage if you're using docker because your tests run in docker on Linux.

[henryJack]

+1 on that. Desperate for this

[ollyjshaw]

I've been following the thread for over a year. All the well meaning work arounds require windows and / or VisualStudio. As @JohnGalt1717 suggests, we're done when you can get a coverage report using docker via a linux base image.

[latop2604]

Coverlet as global tool is available on nuget : https://www.nuget.org/packages/coverlet.console/ Same for reportgenerator https://www.nuget.org/packages/dotnet-reportgenerator-globaltool/

Perfect combo

[JohnGalt1717]

That's nice and all but it should be built into the dotnet sdk since this is a basic feature of devops and required for testing to be relevant. Hence why this is an issue for the dotnet team to implement properly instead of hacking with VS.net

[Gorthog]

@kennethkryger when i try installing 2.1.400-preview-008975 on my agent i get this error:

Could not fetch download information for version 2.1.400-preview-008975. Please check if the version specified is correct. You can refer https://github.com/dotnet/core/blob/master/release-notes/releases.json

I tried installing the latest available .net core sdk version (2.1.302) but then I get the following error:

2018-07-25T15:38:37.1279471Z [command]C:\hostedtoolcache\windows\dncs\2.1.302\x64\dotnet.exe test D:\a\1\s\StringCalculator\StringCalculatorTests\StringCalculatorTests.csproj --logger trx --collect "Code Coverage" --logger trx --results-directory D:\a_temp 2018-07-25T15:38:37.3433537Z MSBUILD : error MSB1008: Only one project can be specified.

I deleted --logger "trx" from command line to dotnet test and then the error went away but no trx file was generated, even through command looks good:

[command]C:\hostedtoolcache\windows\dncs\2.1.302\x64\dotnet.exe test D:\a\1\s\StringCalculator\StringCalculatorTests\StringCalculatorTests.csproj --collect "Code Coverage" --logger trx --results-directory D:\a_temp

any help would be appreciated

--EDIT-- another piece of information:

Microsoft (R) Test Execution Command Line Tool Version 15.7.0 ... Data collection : Could not find data collector 'Code Coverage'

[li0nsar3c00l]

So aparently ths is only working with .NET Core SDK 2.400 and higher which is only in preview right now? @kennethkryger can you confirm?

[kennethkryger]

@sinapis: I can't really help with the debugging the installation process, as the ".Net Core Tool Installer" I mentioned above, simple did the trick for me... Maybe someone else reading this issue has some input on this? But my "Test Execution Command Line Tool" shows version 15.8.0-preview-20180605-02, and not 15.7.0 as yours - so the tools hasn't been installed properly, that's for sure...

@li0nsar3c00l: You are correct - currently .NET Core SDK 2.1.400 is needed, which should include version 15.8 of the Test Execution Command Line Tool... and yes, it's currently only available in preview...

[rifaterdemsahin]

any update on linux for --collect:"Code Coverage" ?

[escalonn]

For those trying to use .NET Core SDK 2.400 in VSTS - the .NET Core Tool Installer may have been able to get that version a couple weeks ago but currently it only pulls from a releases.json file which does not include any recent preview builds.

[clairernovotny]

The VSTS task was just fixed and can install the preview builds again.

[mpetito]

I've installed the preview sdk and can get the binary coverage outputs, but is there a way to convert these to xml? I'm trying to import them into sonarqube which expects the .coveragexml format.

[epoyraz]

@mpetito i had the very same problem. I added an inline PowerShell Task after the Test Task to convert the binary coverage.

# copy and rename the *.coverage to the DefaultWorkingDirectory and rename it to TestCoverage.coverage
Get-ChildItem -Path $(System.DefaultWorkingDirectory) -Filter *.coverage -Recurse -ErrorAction SilentlyContinue -Force | %{Join-Path -Path $_.Directory -ChildPath $_.Name } | Copy-Item -Destination $(System.DefaultWorkingDirectory)\TestCoverage.coverage

# convert binary to xml
CodeCoverage.exe analyze /output:$(System.DefaultWorkingDirectory)\TestCoverage.coveragexml $(System.DefaultWorkingDirectory)\TestCoverage.coverage

This is just a workaround. It works for me. I am interested in a proper solution, too :)

[mpetito]

@epoyraz Thanks! I had explored using CodeCoverage.exe but unfortunately that requires an installation of Visual Studio Enterprise edition (which i do not have). I was hoping there is a way to do this with the .NET Core sdk directly, otherwise the preview sdk generates a very nice binary coverage file that I cannot use.

[codito]

CodeCoverage.exe is available in this nuget package: https://www.nuget.org/packages/Microsoft.TestPlatform/. I think it should work without VS installed.

Look for the relative path: tools\net451\Team Tools\Dynamic Code Coverage Tools\CodeCoverage.exe in the nuget package.

[SharpNoiZy]

Please support the ExcludeFromCodeCoverage Attribute.

Except that, --logger "trx" --collect "Code Coverage" works with the new SDK.

[smadala]

Please support the ExcludeFromCodeCoverage Attribute. Except that, --logger "trx" --collect "Code Coverage" works with the new SDK.

Related: https://github.com/Microsoft/vstest/issues/1548

/cc: @pvlakshm @cltshivash @mayankbansal018

[clairernovotny]

Please do not close this issue. This issue should remain open until embedded PDB's are supported.

[henryJack]

@livarcocc - This is still an issue and should not be closed

[livarcocc]

I didn't really close this issue. github did when @smadala made a PR tagged as fixing it. @smadala can you re-activate it?

[Larswa]

@mpetito Codecoverage comes with the nuget package, and it comes with the VSTS "Visual Studio Test Platform Installer" task. In my case I end up with this on my buildserver that doesn't have VS2017 installed btw ... D:\VSTSAgent\_work\_tool\VsTest\15.8.0\x64\tools\net451\Team Tools\Dynamic Code Coverage Tools\CodeCoverage.exe

[vikasillumina]

@harshjain2 any timelines for providing this support? I am currently working on building the CI/CD for our .NET core based projects and we use docker for running the CI/CD and this feature is missing. If code coverage is natively supported with dotnet test command itself it will make it very easy to use this.

[mpetito]

@vikasillumina use coverlet for now.

I ended up using coverlet even in Windows build environments as the workarounds required above are less than ideal. The issues I ran into included:

• Output of 0 byte .coverage files without any warning or explanation.
• Binary coverage outputs that I couldn't use otherwise.
• Separate package references needed to download CodeCoverage.exe to run binary-to-xml conversion.

After spending hours trying to get this to work, I added coverlet and had my coverage imported into sonarqube in about 10 minutes.

[li0nsar3c00l]

@vikasillumina 2.1.400 is not in preview anymore, so the feature exist and can be used. I got it working in my pipeline. only downside is, that you only get the coverage report and not an output of the percentage.

[JohnGalt1717]

@li0nsar3c00l Does this work on Linux now?

[joymon]

@codito , I tried to install the Microsoft.TestPlatform nuget package to .Net standard test project. But the codecoverage.exe didn't showed up. Any ideas other than checking in the codecoverage.exe with in source