Add support for dotnet test --collect:"Code Coverage"

[harshjain2]

Description

Add support for code coverage through dotnet test on Windows OS machine. dotnet test --collect:"Code Coverage"

[arpit-nagar]

Do you have any plans for Linux ??

(edit from @codito: there are atleast 22 more votes in https://github.com/Microsoft/vstest/issues/579#issuecomment-305374623 for code coverage support in linux. Using this comment to keep track of linux support request)

[nulltoken]

Why the "having Visual Studio 2017 Enterprise installed" limitation? Is there any plan to lift it?

I believe collecting test code coverage during a CI build (with only the .NET Core SDK being deployed) would be a very valid scenario.

[ctro]

:+1: I'd very much love to have working code coverage within the confines of .NET Core.

[harshjain2]

@arpit-nagar test code coverage is VS enterprise and windows only so far, current plan is to get the same level of support in dotnet test (windows + VS enterprise). Mac/linux support isn't in our immediate backlog. /cc @pvlakshm @sbaid to consider the feedback for prioritization.

[arpit-nagar]

@harshjain2 This is a very basic requirement. And as dotnet core is multi platform this feature should also need to be available on Linux. Is there any way we can achieve code coverage on Linux for dotnet core project.

[andyjansson]

test code coverage is VS enterprise and windows only so far, current plan is to get the same level of support in dotnet test (windows + VS enterprise).

@harshjain2 What's the rationale behind requiring VS enterprise for code coverage?

[rol-dave-overall]

Downloaded VS2017 15.3 and .NET Core 2.0 and after entering (dotnet test --collect:"Code Coverage") I get an error:

The test source file "C:\Application2\tests\Console.UnitTests1\Coverage" provided was not found.

Tried changing this to be (dotnet test --collect:"CodeCoverage") i.e. without the space and then get this error:

dotnet exec needs a managed .dll or .exe extension. The application specified was 'C:\Program' Object reference not set to an instance of an object.

[jstallm]

It sounds to me like the status is that we cannot do code coverage for asp.net core 1.X using dotnet test at the command line. Is this correct?

[edumserrano]

I get the same as @rol-dave-overall. This is a show-stopper for us trying to get our solution build on Visual Studio Team Services. Disabling code coverage is not really an option.

Any available workarounds?

[RoCore]

actually have the same problem. the vso test with --collect Coverage or "Code Coverage" not working. the runsettings tried with both names

[FastNinja]

code coverage - this is the most basic requirement. what serious organisation will commit to using dot net core in non Windows environment if such a basic functionality is not there? how business can make a decision to release something in production where test coverage is not known? It can be as fast and awesome - but without this basic support it pretty much unusable outside MS/Windows worlds. The Linux community would like to embrace dot net core on board - but look it just simply can't.

[ollyjshaw]

Yes, this feature should be available for all target platforms. It should not require an installation of Visual Studio.

[DavidParks8]

Code coverage used to be a reason to upgrade to vs enterprise, but the rise of free alternatives such as OpenCover has removed the value proposition. Thus, code coverage should be added to the dotnet cli on all platforms to keep developers from using the alternative products available.

[tracyhoward]

@harshjain2 Is there a timeframe for when this will be available? Add support for dotnet test --collect:"Code Coverage" microsoft/vstest#981

[pvlakshm]

Supporting CC with the dotnet test command line is presently not scheduled in our pans leading up to Nov 2017. However, the following will be possible with .NET Core tests:

1. you can vstest.console.exe command line runner and generate CC information (the CC data collectors are supported on .NET45).
2. If you are working from within VS on tests targeting .NET Core, then you will need to add a reference to the Microsoft.CodeCoverage NuGet package to your test project, and update both the test and app-under-test projects to generate ‘Full’ debug info. Please see "Working with Code Coverage" here: https://github.com/Microsoft/vstest-docs/blob/master/docs/analyze.md (note that this is only a temporary workaround).

We are eager to get to this once we finish addressing a few more scenarios that are already underway.

[arpit-nagar]

@pvlakshm regarding point 2. Is it possible to get CC on windows build environment which have Test agent, and don't have VS?

[pvlakshm]

@arpit-nagar, we are working towards that.

[henryJack]

Code coverage is an essential part of CI. By all means, have a premium version available in a nice helpful way in VS but there should be, for example, a dotnet test --with-coverage=True --fail-under=95 type command for CI.

[ISemyanchuk]

Hi, I was able to run vstest.console.exe with code coverage for .net core 2.0 project locally, but on build agent, with the same setup( i think it is same :)) I've receiving next error : "Failed to initialize client proxy: could not connect to test process."

[GrimSmiler]

Hi!

So as far as i understand code coverage can only be generated localy by VS Enterprise users? There are no howtos for setting it up in TFS with a build agent (on premises) without installing VS enterprise on a build agent machine?

[arpit-nagar]

Do you have any timelines, By when it will be available for .net core on linux environment without VS installation ?

[henryJack]

Not promising that this was removed from the 15.5 milestone and not added to any other milestone. Could @pvlakshm at least add it to the Future milestone so it's not forgotten about?

[IvanAlekseev]

@GrimSmiler I was able to get coverage on VSTS, maybe it can help you https://github.com/Microsoft/vsts-tasks/issues/5066#issuecomment-329745668

[henryJack]

Thanks for keeping this on sprint @pvlakshm :)

[sadiamukhtar]

I am running xunit for my netcoreapp2.0 I am using the vsts task dotnet 2.* (preview) task. I am not able to get code coverage out of the task. Any update on this issue? Is this the right place to ask this question?

Thanks!

[alexzaytsev-newsroomly]

@IvanAlekseev that only works when your build machine is on Windows. We got Enterprise VS, but we use linux build agents (as we deploy and do everything on Windows). From what i can see, still out of luck with code coverage on linux.

[SleeperSmith]

Guys, is there any intention to progress this on linux?

How are we supposed to take dotnet core on linux seriously when we can't compute code coverage on it? Compile and test on windows and then run on linux? Or test on linux and ignore the test coverage altogether?

[henryJack]

It is extremely frustrating that such a basic feature is not available...

[AdrienneCohea]

Can we remove the enhancement label for this issue? Code coverage is not an enhancement. The dotnet tooling is not feature complete until code coverage works on all supported platforms.

[lsthiros]

Commenting to track/+1 this issue as it is affecting the QA of projects that I am contributing to. Serious development cannot take place with C# or any CLI system on a non-windows platform without some code coverage metric, and every search for such functionality on linux or MacOs leads to this issue.

[thisisthekap]

TD;DR: Code coverage is desperately needed.

I totally agree with @lsthiros: Code coverage is key for serious development. I also think that the dotnet core tooling on non windows platforms is simply not complete without code coverage (as @AdrienneCohea already pointed out).

[FastNinja]

Need code coverage - cannot take Dot Net Core seriously if such a basic engineering task is getting pushed between different teams and no one is interested in implementing it.

[axelheer]

Oh, calm down please.

I'm waiting very badly for dotnet test --collect coverage too, but we have still an alternative with @OpenCover and @ReportGenerator (since the "beginning"). Even the .NET Core team uses this combination to analyse their tests, as far as I know.

[thisisthekap]

@axelheer Afaik, @OpenCover is windows only. Do you know of any alternative working on non windows systems (linux, macos)?

[alexl0gan]

I am currently working on an .NET Core SDK project where the corresponding SDKs are written in Python, Java and Javascript. All these languages have support for code coverage on linux. The fact that .NET Core is lagging behind means I get roasted daily!

[voronoipotato]

https://github.com/SteveGilham/altcover

Maybe contribute to altcover. It does the same thing as opencover but is cross platform.

[lionelarmanet]

There's also this project https://github.com/lucaslorentz/minicover, seems a bit young but worth testing.

[pkaramol]

So, to sum up, there is currently no way of running tests with cov support on linux for .NET core projects? Even with mono that runs .exe files on Linux from what I understand?

[alexl0gan]

Mono has coverage built in. For example to get coverage from NUnit tests you can run

mono --debug --profile=log:coverage,covfilter=+[MyProject]MyProject packages/NUnit.ConsoleRunner.3.7.0/tools/nunit3-console.exe --noh --inprocess Tests/MyTestProject.csproj

However this is limited as it only provides coverage of the methods you call, so tests that hit all lines in one method out of twenty would give you 100% coverage.

[pkaramol]

Minicover states it is based on this: https://github.com/gaillard/SharpCover

Has anyone tried it on Linux? Hasn't been updated for a couple of years though...

[lionelarmanet]

Did not try SharpCover, but minicover was easy to setup / launch. However, I had problems to integrate it with my sonar toolchain, so I chose to make coverage w/ OpenCover on a windows node (I'm working with a distributed ci).

This article may be a good start : https://automationrhapsody.com/net-core-code-coverage-linux-minicover/.

My 2 cents.

[anjmao]

This is so sad. I can't recommend .NET Core for anyone. How can someone decided that such basic but important feature like test coverage is not supported by default. It is 2018 and we are not able to run .NET Core tests coverage on Jenkins or Travis during CI build. Amazing. Just Amazing.

[alexl0gan]

Looks like this decision has come from a product manager level, who want code coverage to be a feature in visual studio enterprise only

[voronoipotato]

Gotta say, it seems like late-capitalism is the only thing holding us back from better software. Thankfully I have confidence in my OSS comrades to fight the planned obsolescence and ultimately save Microsoft from their own undoing.

[mralnis]

Common its 2 years and still no metrics ....

[RoCore]

well right now it has prio 1 ;)

[sesispla]

Maybe coverlet is the solution we've been waiting for...

https://www.hanselman.com/blog/AutomaticUnitTestingInNETCorePlusCodeCoverageInVisualStudioCode.aspx

Thx @tonerdo for the amazing job and @shanselman for the diffusion

[shanselman]

@tonerdo

[GrimSmiler]

Used coverlet's feature to export the results in OpenCover format as part of TFS build step. Also managed to feed it to SonarQube server. Not sure about the validity of metrics, but i can assure that such pipeline is possible.

@tonerdo +1

[pvlakshm]

We are starting on enabling code coverage via dotnet test as described in this RFC.