Add support for dotnet test --collect:"Code Coverage"

harshjain2 commented 7 years ago

Description

Add support for code coverage through dotnet test on Windows OS machine. dotnet test --collect:"Code Coverage"

vikasillumina commented 6 years ago

@li0nsar3c00l I actually tried the 2.1.400 version and coverage option does work, however its only supported on Windows. Please correct me if I am wrong. I am looking to use the same command natively on docker running linux. :( Thanks @mpetito for your inputs and lessons learnt on the way to use coverlet, I will give it a try.

codito commented 6 years ago

@joymon it seems to work for me, here's what I tried:

# create a new netstandard2.0 class lib
> dotnet new classlib
> dotnet add package Microsoft.TestPlatform
# see the csproj for classlib, it should have Microsoft.TestPlatform added now
> dotnet restore
> ls ~/.nuget/packages/microsoft.testplatform/15.8.0/tools/net451/Team\ Tools/Dynamic\ Code\ Coverage\ Tools/CodeCoverage.exe
# ~/.nuget is the default nuget package cache on my box. See global-packages details in https://docs.microsoft.com/en-us/nuget/consume-packages/managing-the-global-packages-and-cache-folders

Please don't check-in the exe in any case :) As an alternative, consider adding a step in your CI to download the nuget package directly. Like this in powershell script:

> Invoke-WebRequest "https://www.nuget.org/api/v2/package/Microsoft.TestPlatform" -OutFile Microsoft.TestPlatform.nupkg
> Expand-Archive Microsoft.TestPlatform.nupkg -DestinationPath ./test-pkg

vikasillumina commented 6 years ago

@mpetito Thanks once again for your input and I was able to get the Coverlet to work without any issues other than creating the Result directory to save the output file on Windows (few team members use windows for local development) On mac it worked like charm! I didn't need to do any conversion just used this command: dotnet test %csproj file% /p:CollectCoverage=true /p:CoverletOutput='./Results/' /p:CoverletOutputFormat=opencover and I got the report in xml format without any conversion needed. Please note I used 2.2.1 version of the coverlet.msbuild library https://www.nuget.org/packages/coverlet.msbuild/2.2.1 in case it made the difference.

I tried opencover and cobertura and both options worked fine for me.

StuffOfInterest commented 6 years ago

Even with the 15.8 tools there appears to still be issues with code coverage running under some scenarios. I was able to get the command line to work on my local PC. When I tried adding it to a TFS build things didn't go so well. The command that ran was:

"C:\Program Files\dotnet\dotnet.exe" test B:\agent\_work\43\s\Something.Web.Tests\Something.Web.Tests.csproj --no-build --verbosity normal --collect "Code Coverage" --logger trx --results-directory B:\agent\_work\_temp

What came out was many, many instances of errors like this:

Failed   Index_view_convention_sets_ActivityName_to_View_All_XYZ
Error Message:
 Assembly Initialization method Something.Web.Tests.AssemblyInitialization.InitalizeAppConfiguration threw exception. System.OutOfMemoryException: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.. Aborting test execution.
Stack Trace:
    at Something.Web.Tests.AssemblyInitialization.InitalizeAppConfiguration(TestContext context)

This is being run on a build server with TFS on premises. The build server has Visual Studio 2017 Enterprise version 15.8 installed and the project being tests has the Microsoft.NET.Test.Sdk version 15.8.0 installed in it. If the "--collect Code Coverage" is removed the test run works fine.

mayankbansal018 commented 5 years ago

@StuffOfInterest , we have seen similar issue, where OOM exception is thrown when Coverage is enabled. To validate that your issue is similar to one we fixed internally, can you please provide following info

Are you facing this issue consistently with Coverage on, or is it intermediate?
Are your tests netcore of full CLR based?
When you run things locally, can you please take a quick look, & check what is the approx peak memory consumption of "testhost(if fullCLR tests)" else check "child dotnet.exe(for netcore tests)" for process. Hoping your tests don't take too much time to run.

StuffOfInterest commented 5 years ago

@mayankbansal018 , I'm seeing the issue very consistently. All is happening with netcore. The memory footprint is not excessive.

First, a bit of background. I'm converting over an existing code base of libraries from .NET Framework to .NET Standard. There are currently six library projects. Each library project has a unit test project targeting .NET Core 2.1. Across the six test projects there are just over 1000 unit tests in total, which unfortunately only provide about 50% code coverage.

The .NET build step in the TFS Build service runs "dotnet test" individually for each unit test project. One of the test projects only has eight unit tests in it right now. To make the verification as easy as possible I setup a build to only run testing for this one project.

Both "dotnet build" and "dotnet restore" ran fine. When it came time for "dotnet test" to run the OOM exceptions happened for all eight tests. I had Task Manager running on the build server while the build was running. No processes went over 250 MB of memory consumption while this was happening.

The test project has version 15.8.0 of the Microsoft.NET.Test.Sdk referenced. I've installed version 2.1.402 of the .NET Core SDK on the build server. Version 15.8.0 of Visual Studio Enterprise is installed on the test server. Later this morning I plan to update Visual Studio to 15.8.4, but this will take some work being that the build server does not have direct Internet access.

mayankbansal018 commented 5 years ago

@StuffOfInterest , apologies for getting so late on this. Can you please create a separate bug for your issue, & see if you can attach a repro project to it. At the very minimum please do attach the diagnostics logs that you get when you generate Code Coverage for your projects.

StuffOfInterest commented 5 years ago

@mayankbansal018 , it looks like that may not be necessary. I updated to the recently released Microsoft.NET.Test.Sdk version 15.9.0 and updated the build server's Visual Studio to 15.8.6. Now, the tests are working!

One of those two appears to have solved the memory issue.

Thanks.

mayankbansal018 commented 5 years ago

@StuffOfInterest , great to see it got resolved. 😄

JohnGalt1717 commented 5 years ago

Still requires Visual studio and isn't a dotnet.exe function so it doesn't work in docker for instance. And using coverlet etc. is VERY slow and often fails.

This needs to be independent of Visual Studio and just work as part of dotnet.exe

henryJack commented 5 years ago

Still desperately needed for linux/Docker and is the only thing standing in the way before my corporation adopts .NET core.

carlosrpg commented 5 years ago

Not resolved to me, If enable code coverage I get the following error: (I'm using on TFS) 2018-10-16T20:56:55.6793227Z Microsoft.VisualStudio.TestPlatform.ObjectModel.TestPlatformException: Failed to initialize client proxy: could not connect to test process. 2018-10-16T20:56:55.6793768Z at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyOperationManager.SetupChannel(IEnumerable1 sources) 2018-10-16T20:56:55.6794376Z at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyExecutionManager.StartTestRun(TestRunCriteria testRunCriteria, ITestRunEventsHandler eventHandler)

I have the <DebugType>Full</DebugType> o both test and test project
I have the nuget for <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.9.0" /> on the test project
I Have Microsoft.TestPlatform -Version 15.9.0
the vstest command is: [command]"C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Extensions\TestPlatform\vstest.console.exe" MyTestDLL.dll /EnableCodeCoverage /logger:trx /framework:.NETCoreApp,Version=v2.0

It works fine locally but not on the TFS.

If I remove the /EnableCodeCoverage it runs all the tests just fine

mayankbansal018 commented 5 years ago

@JohnGalt1717 , the coverage support is not longer tied to Visual Studio, & is available with dotnet.exe. Can you please try it again with latest dotnet.exe, & make sure you also have latest version of Microsoft.Net.Test.SDK. Please let me know if you face any errors with it.

@henryJack As mentioned previously, it is part of our black log, we are exploring what all gaps we need to fill, to move our current implementation from windows only, & take it x-plat.

@carlosrpg, the error you have shared seems unrelated to code coverage. Can you please try setting env variable VSTEST_CONNECTION_TIMEOUT=200, & try again, also please enable diagnostics logs via --diag:, & share those with us.

JohnGalt1717 commented 5 years ago

@mayankbansal018 I can't find any documentation on how this is supposed to be done. We're using VSTS (Azure pipelines) and docker. I'm not sure what the output is and how to get it and have it run.

Here's my entrypoint:

ENTRYPOINT dotnet test /src/CADLearning.Api.Test/CADLearning.Api.Test.csproj -c Release --no-restore --collect "Code Coverage" --logger trx --results-directory /var/results

Which results in this in the logs:

Data collector 'Code Coverage' message: No code coverage data available. Code coverage is currently supported only on Windows..

Which is the point. This needs to work with dotnet.exe on Linux for this to work with Docker properly. I can't be running tests on windows when the code is running on a Linux docker container for obvious reasons.

This is a major issue that needs to be escalated and fixed so that this works right.

lomholdt commented 5 years ago

First of all, it seems that a requirement is the 2.1.403 SDK when running dotnet test --collect:"Code Coverage". Each test project will then have a coverage file. But What about multiple test projects? If I wan't to use eg. TFS to show an overview of coverage this is not supported for multiple files. Is it possible to create a combined code coverage file across multiple test projects in a solution?

mayankbansal018 commented 5 years ago

@JohnGalt1717 I agree that you cannot run this on Linux docker, as I mentioned previously as well, we have already started investigating, that what will it take us to move from windows only implementation, to x-plat. I don't have a clear timeline yet for this. I can share a plan once we have finalized it internally.

@lomholdt if you are running CodeCoverage via dotnet in TFS, & use publish coverage task, it would merge the multiple coverage files generated, & would show a consolidated report. However for dotnet code coverage currently does not merge the coverage files for multiple projects across the run. Can you please open a bug for the same. I don't want it to get lost in this conversation.

lomholdt commented 5 years ago

@mayankbansal018 Thanks. I have created it here.

codekoenig commented 5 years ago

if you are running CodeCoverage via dotnet in TFS, & use publish coverage task, it would merge the multiple coverage files generated, & would show a consolidated report.

@mayankbansal018 I tried to add a "Publish Code Coverage Results" task to have the generated coverage files from different projects merged, but it shows the following warning:

##[warning]Multiple file or directory matches were found. Using the first match: [Path to .coverage file]

So it doesn't seem to be able to merge multiple files. I used the following minimatch pattern in the "Summary file" field to select all generated coverage files: **\TestResults\**\*.coverage

Also I noticed that the code coverage couldn't be uploaded even for the first match as ist seems to expect another format as dotnet test did generate:

##[warning]Failed to read [path to coverage file]: Data at the root level is invalid. Line 1, position 1

Is there any guidance how to make this work?

RehanSaeed commented 5 years ago

Using dotnet test --logger "trx;LogFileName=Foo.trx" --collect:"Code Coverage" produces some crazy long file names and sub folders:

7ff221cc-88df-4f41-bbcb-07ce3dce900f
  rehan.saeed_MY_MACHINE_NAME_2018-10-24.12_25_21.coverage
rehan.saeed_MY_MACHINE_NAME_2018-10-24_12_25_18
  In
    MY_MACHINE_NAME
      rehan.saeed_MY_MACHINE_NAME_2018-10-24.12_24_56.coverage

A few questions come to mind:

Is it possible to provide a custom name, like we can for .trx files above?
I'm not very familiar with .coverage files but why are there two of them per test project and why do we have nested folders with the machine name repeated?
I know you are going for unique-ness but do we really need the username and machine name in there? If I could provide a custom name, I suppose all of that could go away.

All of this could be a lot cleaner.

mayankbansal018 commented 5 years ago

@acesiddhu Can you please help @codekoenig regarding publishing multiple coverage files via publish code coverage task

mayankbansal018 commented 5 years ago

@codekoenig @lomholdt , I just checked with team, & if you are using dotnet task to run tests in vsts pipeline, it should publish test results, as well as code coverage files, & merge them as well. "Publish Code Coverage" task however does not work with ".coverage" files, it works with "Cobertura" & other coverage outputs.

If you still face issue with dotnet task to run & publish coverage results, please run your build under diagnostics via setting build env variable "system.debug=true", & share those logs with us.

codekoenig commented 5 years ago

@mayankbansal018 Thank you, that actually seems to work now! That makes everything so much easier. Big thanks to the team for working on this!

Although when it comes to Code Coverage, the results do not seem to be correct. First I noticed coverage is reported in the TFS build, but with a very low blocks/line count that is wrong:

More interestingly, when downloading the .coverage file from the TFS build and opening in Visual Studio, more accurate numbers are shown that differ from the reported values in TFS:

When looking into the hierarchy, I can see that there are still some projects missing. I'm investigating, might be my fault, but the difference in reported coverage on TFS and what's actually inside the .coverage file seems definitely odd.

Are there known issues with this feature?

Edit: I just tried another build and the coverage was suddenly spot on correct and TFS as well as Visual Studio reported the exact same block count:

I immediately started another build and then the coverage results were suddenly completely off again. No checkins were committed in the meantime, no changes to the build were made:

Seems there might be a race condition or something happening during merging the different .coverage files ... any ideas what could cause this?

RehanSaeed commented 5 years ago

Is there an ETA on getting "Publish Code Coverage" task working with.coverage` files?

Azrrael commented 5 years ago

@mayankbansal018 I can't find any documentation on how this is supposed to be done. We're using VSTS (Azure pipelines) and docker. I'm not sure what the output is and how to get it and have it run.

Here's my entrypoint:

ENTRYPOINT dotnet test /src/CADLearning.Api.Test/CADLearning.Api.Test.csproj -c Release --no-restore --collect "Code Coverage" --logger trx --results-directory /var/results

Which results in this in the logs:

Data collector 'Code Coverage' message: No code coverage data available. Code coverage is currently supported only on Windows..

Which is the point. This needs to work with dotnet.exe on Linux for this to work with Docker properly. I can't be running tests on windows when the code is running on a Linux docker container for obvious reasons.

This is a major issue that needs to be escalated and fixed so that this works right.

We have the exact same issue. We need Code Coverage running on a dockerized Linux container. Please let us know when you have an ETA for this.

mayankbansal018 commented 5 years ago

@codekoenig , I'm not sure why the VSTS, is reporting in-consistence numbers regarding coverage results, where as VS IDE shows correct. Can you please share a coverage file from one such scenario, where number reported were incorrect. We could try to upload the file from our end, & see if it shows any discrepancy.

@RehanSaeed I'll float this though internally. It's possible I'm missing something, & there is no need to publish coverage task to upload .coverage files as well.

mayankbansal018 commented 5 years ago

@PBoraMSFT , @cltshivash , @pvlakshm do we have an estimate for CodeCoverage on x-plat?

RehanSaeed commented 5 years ago

@mayankbansal018 Azure DevOps has a task to run tests and accept coverage files but I use Cake build to run dotnet test myself which generates .coverage files. There is a built in task to accept code coverage files but it does not support the .coverage format.

tonven commented 5 years ago

Running dotnet test with --collect "Code coverage" includes in a code coverage dll libraries referenced in a test project. Does anyone has the same error?

mayankbansal018 commented 5 years ago

@Tonvengo user can configure what all dll's should be reported in coverage, please check here

codekoenig commented 5 years ago

@mayankbansal018 The problem is two-fold. One one hand, some of the generated coverage reports shows inconsistent results in VSTS and Visual Studio. At the same time, it's clear that not all projects are included in those reports as these files are noticably smaller than they should be and I can see in Visual Studio that some projects are missing from the report.

While on other occassions, I get a coverage report file that seems to be complete and contains all projects. And in this case, the results are also the same in VSTS and Visual Studio.

I have two sample files, one for each scenario I could provide you, but I don't want to share it publicly as it's a confidential project for a customer. Is there a way I can provide them to you over a private channel? Thanks!

valdasmisevicius commented 5 years ago

@codekoenig @mayankbansal018 Any progress about different numbers in VSTS and Visual Studio?

mayankbansal018 commented 5 years ago

Apologies for delay on this. I was OOF for entire dec. @codekoenig you can share the file over maban at microsoft dot com

steinmr commented 5 years ago

@codekoenig Did you resolve your issue? We have the exact same scenario, but in TFS 2018.3. For those builds that show incorrect numbers, you can go into "Tests"-tab, download all coverage files and combine them in VS. The number shown there is correct, and different from the incorrect percentage shown in "Summary"-tab. So it appears that VSTS/TFS has all the files, but in some builds it calculates the total incorrectly, and also does not merge the files when you try to download them from the "Summary"-tab. I took a look in the TFS-database, and the files coverage data can be found there, but I was unable to find the total percentage, maybe it is calculated on the fly.

codekoenig commented 5 years ago

@steinmr I'm in (e-mail) contact with @mayankbansal018 and some of his colleagues about this issue and I sent them sample files and screenshots and will provide them any logs they might ask for. So no solution yet, but investigations are ongoing and I will report back as soon as we know more.

The problem you are having definitely sounds like mine, although in my case, whenever TFS reports incorrect numbers, I can also see missing projects when I download and open the coverage file in Visual Studio. So the coverage report in Visual Studio is incorrect too, just differently incorrect, so to say. But maybe these are two unrelated issues that just happen to appear at the same time. We have a lot of projects (about 30) in our solution.

steinmr commented 5 years ago

@codekoenig Ok, thanks for keeping us posted. When you say that you download the coverage files, do you do that from the summary page (as a single file) or as attachments on the tests runs on the test page? Because in the first case, I also get inconsistent numbers and a lot of missing projects (seems to be to be a random file out of all the coverage files, but not sure). But in the latter case I see all the correct coverage files (as attachments to the test runs), identical to those generated on the build server.

codekoenig commented 5 years ago

@steinmr Ah, missed that in your post - yes I'm talking about the single/merged file from the summary page. I downloaded now the individual coverage files from the test page and you are right, if I merge them in Visual Studio manually, the results are correct. That might be an important find, I'll send the individual files to the team working on the issue too.

And, of course, looks like we have exactly the same issue now.

nigurr commented 5 years ago

@codekoenig we are acknowledging this bug in code coverage merge at Azure DevOps Service. We are running into race condition where data might be shown incorrect if many code coverage files exist.

I will raise separate tracking item which can be tracked (don't want to extend this thread with this)

Lobosque commented 5 years ago

What is the current state of code coverage? I see that there is a nuget package with a new major version released a few days ago: https://www.nuget.org/packages/Microsoft.CodeCoverage/ Does this adds support for Linux code coverage? If not, is there an ETA?

acesiddhu commented 5 years ago

@Lobosque we are actively working on enabling CC on Linux. The latest package that was shipped doesnt have that support yet. /cc: @abhishekkumawat23 @cltshivash @PBoraMSFT

ghost commented 5 years ago

@acesiddhu Is there any update in enabling CC on Linux? Does the 16.1.0 nuget have it?

acesiddhu commented 5 years ago

@accu-joeykarns as we speak the support is currently under development. We are integrating it with coverlet. Here is the documentation https://github.com/tonerdo/coverlet/pull/402 that has details on how this is going to work

RamiSh commented 5 years ago

@codekoenig we are acknowledging this bug in code coverage merge at Azure DevOps Service. We are running into race condition where data might be shown incorrect if many code coverage files exist.

I will raise separate tracking item which can be tracked (don't want to extend this thread with this)

hey -- where is the separate tracking item for this? and where are we with merging multiple code coverage files?

codekoenig commented 5 years ago

@RamiSh https://github.com/microsoft/azure-pipelines-tasks/issues/9581

gremlm commented 5 years ago

When using code coverage with vstest.console.exe it is impossible to specify location of the output .coverage file! That problem makes usage of the code coverage with vstest.console.exe useless!

steinmr commented 5 years ago

@steinmr Ah, missed that in your post - yes I'm talking about the single/merged file from the summary page. I downloaded now the individual coverage files from the test page and you are right, if I merge them in Visual Studio manually, the results are correct. That might be an important find, I'll send the individual files to the team working on the issue too.

And, of course, looks like we have exactly the same issue now.

@codekoenig I added a workaround for our common problem in microsoft/azure-pipelines-tasks#9581

giggio commented 5 years ago

I just tested the coverlet integration and it works. Is this going to be ported over to VSTest or that will be the recommended approach? If so, this issue should be closed and the docs should be updated. BTW, it works on Windows, Linux, Mac and anywhere .NET Core works.

vikasillumina commented 5 years ago

@giggio this is great news that it works on all the platform. Please can you share details of your cli command and version of dotnet core you used? I would like to try it.

gremlm commented 5 years ago

By the way, how it can be executed just in .Net, not under .net core?

Sincerely, Grigoriy Milman

On Aug 28, 2019 5:59 PM, vikasillumina notifications@github.com wrote:

@giggiohttps://github.com/giggio this is great news that it works on all the platform. Please can you share details of your cli command and version of dotnet core you used? I would like to try it.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/microsoft/vstest/issues/981?email_source=notifications&email_token=AMNNEQCEC2CQG7VMLPOGTXDQG4NOPA5CNFSM4DVX5Q6KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5M4ULQ#issuecomment-525978158, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AMNNEQFEFKGEOWCSKMX2DPLQG4NOPANCNFSM4DVX5Q6A.

giggio commented 5 years ago

@vikasillumina just follow the steps on their readme: https://github.com/tonerdo/coverlet#installation It works on .NET Core and .NET Framework, and on .NET Core they have a 'msbuild' way and a 'vstest' way. Both work, but have different capabilities (and different bugs). :)

vikasillumina commented 5 years ago

@giggio I am actually already using coverlet something like this: dotnet test $project \ /p:CollectCoverage=true \ /p:CoverletOutput='./TestResults/' \ /p:CoverletOutputFormat=cobertura \ /p:Exclude=someexclusions /p:CopyLocalLockFileAssemblies=true \ --configuration Release \ But I was curious to try any coverage cli options that can natively run with dotnet core itself like this issue asked for: "dotnet test --collect:"Code Coverage"" But thanks for your reply.

microsoft / vstest

Add support for dotnet test --collect:"Code Coverage" #981

Description