Add support for dotnet test --collect:"Code Coverage"

harshjain2 commented 7 years ago

Description

Add support for code coverage through dotnet test on Windows OS machine. dotnet test --collect:"Code Coverage"

JesseNewman19 commented 4 years ago

Gentlemen I have the answer

Instead of: dotnet test My.ProjectName.csproj --logger trx --collect:"Code Coverage"

Use this instead: dotnet test My.ProjectName.csproj --logger trx /p:collect="Code Coverage"

If you're in Azure DevOps, do not use their checkbox for collecting test results and coverage, it will execute the first command.

adaughterson commented 4 years ago

Gentlemen I have the answer

Instead of: dotnet test My.ProjectName.csproj --logger trx --collect:"Code Coverage"

Use this instead: dotnet test My.ProjectName.csproj --logger trx /p:collect="Code Coverage"

If you're in Azure DevOps, do not use their checkbox for collecting test results and coverage, it will execute the first command.

Yea, this didn't work either. Still get this lovely message:

Data collector 'Code Coverage' message: No code coverage data available. Code coverage is currently supported only on Windows..

mtrtm commented 4 years ago

@JesseNewman19 Your command line works for me, but only outputs a .trx file. In Azure, how are you getting coverage to tie in? Do you convert to one of the two supported formats?

adaughterson commented 4 years ago

EDIT: Ugh, I retract my premature retraction. The proposed solution totally works if I don't try to generically disregard certain assemblies based on attributes using a settings file (see https://docs.microsoft.com/en-gb/visualstudio/test/customizing-code-coverage-analysis).

It seems that when I use the runtime settings, the problem rears its ugly head again.

Still a problem for me.

svejnohatomas commented 4 years ago

@nohwnd @acesiddhu Is there any update in enabling --collect="Code coverage" on Linux?

da-edwards commented 4 years ago

@nohwnd @acesiddhu Is there any update in enabling --collect="Code coverage" on Linux?

Any news on this one? We use Linux for build agents and want to make use of the code coverage status policy to display the coverage on updated code in a pull request so this is key for us.

Thanks

queen-of-code commented 4 years ago

You can use coverlet to collect coverage on Linux. It appears to be the supported way. See:

https://github.com/tonerdo/coverlet/blob/master/README.md#vstest-integration-preferred-due-to-known-issue

On Mar 12, 2020, at 08:09, da-edwards notifications@github.com wrote:

@nohwnd @acesiddhu Is there any update in enabling --collect="Code coverage" on Linux?

Any news on this one? We use Linux for build agents and want to make use of the code coverage status policy to display the coverage on updated code in a pull request so this is key for us.

Thanks

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

da-edwards commented 4 years ago

You can use coverlet to collect coverage on Linux. It appears to be the supported way. See: https://github.com/tonerdo/coverlet/blob/master/README.md#vstest-integration-preferred-due-to-known-issue … On Mar 12, 2020, at 08:09, da-edwards @.***> wrote: @nohwnd @acesiddhu Is there any update in enabling --collect="Code coverage" on Linux? Any news on this one? We use Linux for build agents and want to make use of the code coverage status policy to display the coverage on updated code in a pull request so this is key for us. Thanks — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

Yeah, we are using Coverlet to output a Cobertura format coverage report but (happy to be wrong here) not been able to integrate Coverlet supported output formats with the Azure DevOps code coverage status policy.

Was hoping to run with something like this and get a .coverage file which should then integrate with Azure DevOps pull requests:

dotnet test --no-build --results-directory:$(System.DefaultWorkingDirectory)/VSTestTestResults/Coverage/ --logger trx /p:collect="Code Coverage"

nohwnd commented 4 years ago

@da-edwards I am sorry I don't have any updates on this issue, I am working on the rest of the pri:1 issues in this repo.

danilobreda commented 4 years ago

I really does not understand how a 2017 issue with 80+ participants and a lot of "reactions" is not taken seriously and with priority... .net core is crossplatform, why the features is not? This creates loopholes. @shanselman im waiting it for 2 years now :( and all my customers would like to use it.

As said before "this feature should be available for all target platforms. It should not require an installation of Visual Studio."

vikassd2012 commented 4 years ago

I really does not understand how a 2017 issue with 80+ participants and a lot of "reactions" is not taken seriously and with priority... .net core is crossplatform, why the features is not? This creates loopholes. @shanselman im waiting it for 2 years now :( and all my customers would like to use it.

As said before "this feature should be available for all target platforms. It should not require an installation of Visual Studio."

I believe this is intentional. Microsoft even after moving towards cross platform support still wants people to use their products(Azure pipeline etc) over cross platform products, otherwise I don't see a reason why this issue which have been open for so long have not be prioritized by Microsoft.

twsl commented 4 years ago

How is this not pri:1 @nohwnd ?

poke commented 4 years ago

You do realize that this issue already is priority 1?

That being said, for those that require code coverage reports, what stops you from using Coverlet on Linux?

vikassd2012 commented 4 years ago

You do realize that this issue already is priority 1?

That being said, for those that require code coverage reports, what stops you from using Coverlet on Linux?

Is it really a priority 1? It's certainly not being treated as P1 IMHO. Regarding coverlet we are already using coverlet for last 2 years like most of us on this thread. But would have been nice to use --collect option natively with dotnet test cli.

clairernovotny commented 4 years ago

@vikassd2012

But would have been nice to use --collect option natively with dotnet test cli.

I'm genuinely curious what "native" means to you? Coverlet is a .NET Foundation project, just like .NET Core and many others. It can be invoked with dotnet test --collect:"XPlat Code Coverage", so what makes it not native?

vikassd2012 commented 4 years ago

@vikassd2012

But would have been nice to use --collect option natively with dotnet test cli.

I'm genuinely curious what "native" means to you? Coverlet is a .NET Foundation project, just like .NET Core and many others. It can be invoked with dotnet test --collect:"XPlat Code Coverage", so what makes it not native?

@clairernovotny I had tried the dotnet cli with coverage options on a window box and I found the reports to be really informative. So was hoping to take advantage of those reports in linux side. Other concern with using coverlet (no offence to coverlet team, they have done a splendid job to provide us an option for linux world in absence of this feature) is that they have broke in past after upgrades. So if we have these coverage options available within dotnet cli, we would not have this risk. Also we don't need to manage upgrading of coverlet packages along with dotnet core updates. Not sure if this helps you to understand where I am coming from. If not, we can agree to disagree. :) Also if you feel coverlet is going to be the way for dotnet cli to work with on non windows environment, we should close this issue citing to use coverlet instead and it should be included in Microsoft official documentation. So that we have the right expectation for non windows users.

poke commented 4 years ago

Other concern with using coverlet is that they have broke in past after upgrades.

You need to treat it like any other dependency: Upgrades can break stuff so you should always test things when you upgrade. But you can also always stick to older working versions.

So if we have these coverage options available within dotnet cli, we would not have this risk.

As demonstrated above, Coverlet is available within the CLI.

I assume you mean if it came bundled with the SDK. But as far as I am aware, that isn't true for the Windows coverage option either. It's part of the Test SDK.

Also we don't need to manage upgrading of coverlet packages along with dotnet core updates.

Why is this even an issue? You will have to update loads of dependencies anyway: Test SDK, MSTest, xUnit, Moq, …

You seem to assume that stuff from Microsoft is always “safe to use” and that it doesn't break but that simply isn't the case. And if it was perfect, then you will still need to understand that in order to get there they plan and test it properly before releasing updates. And that takes a lot of time and work, which is precisely what is slowing down issues like this.

Not sure if this helps you to understand where I am coming from.

Sorry, but to me this sounds like the common thinking that everything from Microsoft is magically vetted as good, and every alternative from within the community is a suboptimal solution by default. That is really a harmful behavior that needs to stop.

danilobreda commented 4 years ago

@poke

That being said, for those that require code coverage reports, what stops you from using Coverlet on Linux?

So if I have a test project (example) on windows using visualstudio, and i create a automated build (CI) on azure devops... i build on linux and windows for test, so, on windows i can run code coverage, on linux not. I will be forced to use coverlet. So my first question, why does a cross platform have a oneplatform feature?

Are you guys saying that it is better to throw away this tool and guide everyone to use coverlet? if this is the case, lets update the documentation

(Azure devops - Pipeline - oficial .net core Test Task 2.x)

poke commented 4 years ago

If you are currently working on a project that targets Windows and Linux, then yes, you should probably be using Coverlet to generate code coverage if you need that. Not everybody is required to run on Linux though. I know a lot of people, myself included, that still primarily target Windows platforms (for a multitude of reasons). So for those, they can just rely on the VSTest included mechanism here.

why does a cross platform have a oneplatform feature?

There are actually a few of those within .NET Core. WPF and WinForms come to mind but you can also create COM components that only work on Windows. Why does the platform has those? Because it is also the replacement for the .NET Framework and as such requires to replace use cases of it. Not everything automatically works cross-platform then though. Some can be with additional work, e.g. this very issue, but some others probably will never make it to other platforms.

I think it’s wrong to expect platforms that are cross-platform to have every single functionality identical on every other platform. That actually isn’t true for most platforms and languages, where you always have some OS-specific things. Sometimes, those can be abstracted so that you won’t notice it as much, but some other times not.

Are you guys saying that it is better to throw away this tool

That’s probably not the “better” option. This code coverage functionality is also built into Visual Studio and many people rely on it during development. And there are still plans to expand its functionality to other operating systems.

lets update the documentation

It might be a good idea to suggest a change to the dotnet Azure DevOps task, to include a separate option for built-in Coverlet support. So people using Coverlet could just tick a box and the task would pick up the results automatically. It probably doesn’t hurt to suggest that on the Azure Pipeline tasks repo. The only problem with that would be that Coverlet uses a different output format but they also own the PublishCodeCoverageResults task which supports Coverlet’s output.

vikassd2012 commented 4 years ago

Other concern with using coverlet is that they have broke in past after upgrades.

You need to treat it like any other dependency: Upgrades can break stuff so you should always test things when you upgrade. But you can also always stick to older working versions.

So if we have these coverage options available within dotnet cli, we would not have this risk.

As demonstrated above, Coverlet is available within the CLI.

I assume you mean if it came bundled with the SDK. But as far as I am aware, that isn't true for the Windows coverage option either. It's part of the Test SDK.

Also we don't need to manage upgrading of coverlet packages along with dotnet core updates.

Why is this even an issue? You will have to update loads of dependencies anyway: Test SDK, MSTest, xUnit, Moq, …

You seem to assume that stuff from Microsoft is always “safe to use” and that it doesn't break but that simply isn't the case. And if it was perfect, then you will still need to understand that in order to get there they plan and test it properly before releasing updates. And that takes a lot of time and work, which is precisely what is slowing down issues like this.

Not sure if this helps you to understand where I am coming from.

Sorry, but to me this sounds like the common thinking that everything from Microsoft is magically vetted as good, and every alternative from within the community is a suboptimal solution by default. That is really a harmful behavior that needs to stop.

This would have stopped 2 years ago if Microsoft had accepted coverlet as an official code coverage option for non windows environment and more importantly close this issue with that explanation. Keeping this ticket open gave false expectations.

@poke you didn't respond to my last portion of comment. It seems you only picked and chose portion of my comments according to your perception about my view on this.

"Also if you feel coverlet is going to be the way for dotnet cli to work with on non windows environment, we should close this issue citing to use coverlet instead and it should be included in Microsoft official documentation. So that we have the right expectation for non windows users."

Anyways I am done following this issue. Will leave the experts here to decide what they want to do.

danilobreda commented 4 years ago

@poke its a good idea to close this issue and indicate that the way for cross plat is coverlet? I think its the better for all new programmers thats is facing this problem... i will open some documentation issues to solve the problem raised previously.

Maybe change the CLI to say that its not supported when the command is called, use coverlet...

Thanks for explanations and your time :D

nohwnd commented 4 years ago

@twsI @danilobreda It is taken seriously, and it already is pri1. I can also explain why this is not done. 🙂 I took over this area in December when I started in MSFT, and since then I am focusing on learning the platform, getting the infrastructure and tests into shape so we can ship easily and realiably, and solving the most pressing bugs. Unfortunately this a more of a feature than a bug, and so the progress here is slower, because it requires design work and high-level and in detail understanding of the platform. I totally understand this is a pressing issue, for you, but also an issue that we want to solve correctly, not just monkey patch it and then suffer with maintaining that for the rest of eterninty 😁

There are more people in our team now, that also pick up momentum, and more will hopefully join.

da-edwards commented 4 years ago

Thanks all for various replies, @poke good shout - I'll raise a github issue for the pipeline repo to see if they can support Coverlet.

Thanks also @nohwnd - just to be clear from my perspective, I'm perfectly happy using Coverlet, we have been for over a year without a single issue, the only thing I want to be able to do is take a view at a PR what level of unit test coverage there is on changed code and currently that's a feature only available on Windows. Will keep following this issue and hopefully you and the team can ship in the near future. At the moment, I'm contemplating moving our builds back on to a Windows box to take advantage of this PR code coverage feature but I would much prefer to stick on Linux.

nohwnd commented 4 years ago

@da-edwards I think this is definitely an important scenario to cover, tagging @jakubch1 so he can keep it in mind in his current research of code coverage.

syedhassaanahmed commented 4 years ago

We need this functionality in our current project. Any update?

jakubch1 commented 4 years ago

Did you try coverlet?

syedhassaanahmed commented 4 years ago

@jakubch1 No. Should this issue be closed and indicated that coverlet is the answer, as @danilobreda also pointed out?

gremlm commented 4 years ago

Coverlet is not the answer. This feature should be compatible when executed at solution containing .Net Core and .Net Framework projects. MS buils can build such solution, all projects. dotnet - only projects related to .Net Core. Coverlet only works for .Net Core, so again it is custom feature

da-edwards commented 4 years ago

Hey @nohwnd , @jakubch1 , just wondering if there are any timelines to this one? Keen to make use of the Azure Devops pull request code coverage feature and I'd rather keep our build hosts as Linux but if it's not likely to be fixed soon, I'll move to Windows for the builds, at least in the short term.

Thanks

PureKrome commented 4 years ago

I'm still confused to why people are complaining about lack of code coverage support for XPlat when it is already here and has been for a while.

For example, this repo uses CI (on AppVeyor ... not Azure DevOps) that does linux + codecov:

dotnet test -c $env:CONFIGURATION 
            -v minimal 
            --no-build 
            --collect:"XPlat Code Coverage" 
            --settings coverlet.runsettings 
            --results-directory './CodeCoverageResults'

All the coverlet docs explain this.

In that repo, I added the following:

coverlet.runsettings so I can be specific about what to EXCLUDE from coverage. For my personal self, I don't want coverage of any tests. I just want coverage of the actual code itself.
made sure my test projects included the coverlet dependency, which is now default included to any new .net core app "xunit project" in visual studio. If you're not using VS (which is me sometimes + many others, like using VS Code) ... then just manually include the nuget package in your test csproj.

As a bonus, I also (during my CI) upload the codecoverage results to an online codecoverage website. You can swap this out and instead use a code coverage reporting tool instead, on your CI machine.

TL;DR;

there is .net core code coverage for linux and windows
get familiar the CLI .net commands. e.g. dotnet test
code coverage can work on any build system which has dotnet core SDK installed. Your own computer, AppVeyor, Azure DevOps and i'm assuming GitHub Actions.

Now, here's an awesome cat to make you smile and positive:

da-edwards commented 4 years ago

Hey @PureKrome ,

Just to be clear on the coverage, I'm very happy with using Coverlet, we've been using it for > 18 months now without issue. However, Azure DevOps doesn't support Coverlet for pull request diff code coverage policies, which we would find useful (blocking PRs with a diff code coverage < x%), as it's currently supported only on Windows. As our build infrastructure is Linux based, that poses a bit of an issue.

Thanks

PureKrome commented 4 years ago

Hi @da-edwards - ok, let me see if I understand the issue, so appologies if i'm getting all of this wrong.

If codecov < x% (eg. 85%) then 'error' so the CI fails, which means the PR shouldn't be approved.
CI system is Azure DevOps.

If this is correct, then CodeCov has the ability to error when a "threshhold" is not met. REF: CodeCov 'threshold' docs.

That said, the threshold attribute is for the 'msbuild' nuget (option microsoft/vstest#2). I prefer the codecov option microsoft/vstest#1 'vstest' nuget. Now, the msbuild nuget seems to work on linux though, but i've not done it nor can I speak for it.

Q: Could more be done? A: Yep

Q: Is it still a bit painful? A: Yep :(

Q: Warts and all, could it actually work? A: Could! I've not tried it though but it might work? Maybe post a question in the codecov issues about 'Threshold' argument and option microsoft/vstest#1 nuget. Maybe it might be a key/value setting in a runsettings file? (I use runsettings files with my codecov + option microsoft/vstest#1022

HTH.

da-edwards commented 4 years ago

Hey @PureKrome not quite, we're not (yet) interested in the overall code coverage, just on the diff, so if I was to update a method and refactor 4 lines but only 2 of those are covered with unit tests we would like to see a coverage of 50%, regardless of how many other unit tests have been added during the commit.

That's the (very specific) feature that's supported on Windows but not on Linux ... I think anyway :)

Does that make sense?

PureKrome commented 4 years ago

Yep - it does.... but .. I've never heard of that?! TIL!!!

So do you mean this?

da-edwards commented 4 years ago

That's the one ... did you manage that with Linux or is that on a Windows box?

Thanks

PureKrome commented 4 years ago

Windows :( I've not tried on linux (yet). Maybe we can move this convo over to the Codecov repo and re-ask there? (ping me into the issue you create?)

thiagocamargos commented 4 years ago

My two cents here: dotnet core and all its stack (vstest included) is supposed to be cross-platform and support linux. It is a big issue if I have to change code (and build|release instructions/package management ARE code) if I decide to change platform and everyone tells me the framework /stack is x-plat. That is misguiding.

When people here tells others to use coverlet instead vstest coverage for linux builds, you are also telling to use coverlet for windows and ignore the vstest feature. And if that is true, then this feature may be considered useless by those who appreciate a cross-platform framework. Thus, it is better to remove it from the application to minimize development complexity.

Telling people to use coverlet is only deprioritizing this issue, keeping it open forever and misguiding people into thinking this will be done. It has been almost 3 years since this was opened and there is still no clear answer from the core team. I have been following it and being notified since the beginning and I'm wondering if 3 years of a open issue is not enough yet.

@nohwnd , @jakubch1 and @AbhitejJohn Please do take this into consideration and give a clear answer. If coverlet should be the answer, please give a statement telling this, and making it clear that vstest IS NOT and WILL NOT be fully compatible with Linux. Also, a guide to cover this need (pun intended) in coverlet should be written/refered to. If Linux support is to be implement, please also make it clear.

just a quick ps: i've been reading this too.

AceHack commented 4 years ago

I would love to hear an update on this issue, it's very confusing on status and plans.

JohnGalt1717 commented 4 years ago

Sorry I haven’t had time. Tomorrow morning I’ll try and do something to make it obvious.

Really I think the real solution is this:

https://github.com/microsoft/vscode/issues/97919#issuecomment-630849160

A real standardized test framework in vs code that can be hooked into to handle all of this stuff and give us parity with vs.net

Please thumbs up if you agree.

pavelhorak commented 4 years ago

We’ve been recently reviewing existing .NET code coverage solutions and decided to enhance our dynamic code coverage solution /also referred to as VS code coverage by some/ and make it cross platform. It will continue supporting code coverage for managed code on both .NET/.NET Core frameworks and for C++ on Windows. This would provide the community with a similar dynamic instrumentation solution, where only configured modules loaded at runtime are instrumented, as VS Code Coverage scenarios employed. As a proof of concept, @jakubch1 built a new prototype based on CLR Instrumentation engine profiler and made it run on Linux and Windows. We also plan to support non-binary report format(s) so it will be easily possible to display code coverage results, especially on Azure DevOps. As of now, we encourage community users continue using Coverlet as a cross platform code coverage solution. I hope to be able to share the timeline soon.

PureKrome commented 4 years ago

@pavelhorak (and others in that team):

kendrahavens commented 4 years ago

To add, Coverlet continues to provide an excellent cross-platform code coverage solution. Bringing the built-in dynamic code coverage solution cross-platform won’t replace it since Coverlet will still be the best option for those who need static code coverage (aka ahead-of-time compilation), which is particularly useful for testing services. This effort is an attempt to satisfy the customers who have specifically asked for a dynamic (on-demand instrumentation) cross-platform code coverage solution.

da-edwards commented 4 years ago

Great news! Thanks @pavelhorak and the whole team. Really appreciate your time and efforts on this.

iyusuboff commented 4 years ago

Coming from java, even the simplest tasks are a problem in dotnot

mikeblakeuk commented 3 years ago

We are running dotnet 5.0 C# xunit AzureDevop - both windows and ubunt

We want:

unit test results in devops,
code coverage, excluding xunit dlls
code coverage, results in devops (TBC)
block the PR if the coverage is too low
see ticks/crosses in the PR

any ideas what yaml we need to run? / runsettings we need?

ElvenSpellmaker commented 3 years ago

To add, Coverlet continues to provide an excellent cross-platform code coverage solution. Bringing the built-in dynamic code coverage solution cross-platform won’t replace it since Coverlet will still be the best option for those who need static code coverage (aka ahead-of-time compilation), which is particularly useful for testing services. This effort is an attempt to satisfy the customers who have specifically asked for a dynamic (on-demand instrumentation) cross-platform code coverage solution.

We use coverlet, but Azure DevOps's built in Code Coverage checker only supports .coverage and as we build on Linux we can't get the nice code coverage features without moving the build over to Windows... =|

ElvenSpellmaker commented 3 years ago

Is there a way to convert Coverlet Cobertura files into .coverage? I know the reverse is possible.

IvanAlekseev commented 3 years ago

@ElvenSpellmaker we use this on windows machines but I guess it will work on linux as well You will not get all devops coverage features but at least you can fail build on insufficent coverage. It uses another microsoft task to achieve that

      - task: DotNetCoreCLI@2
        displayName: Test Projects
        inputs:
          command: test
          projects: $(testProjects)
          publishTestResults: true
          arguments: '--no-build -c $(buildConfiguration) --collect:"XPlat Code Coverage"'

      - script: dotnet tool install --global dotnet-reportgenerator-globaltool --ignore-failed-sources
        displayName: Install ReportGenerator

      - script: reportgenerator -reports:$(Agent.TempDirectory)/**/coverage.cobertura.xml -targetdir:$(Build.SourcesDirectory)/CodeCoverage -reporttypes:Cobertura
        displayName: Create Code coverage report

      - task: PublishCodeCoverageResults@1
        displayName: 'Publish code coverage'
        inputs:
          codeCoverageTool: Cobertura
          summaryFileLocation: '$(Build.SourcesDirectory)/CodeCoverage/Cobertura.xml'

      - task: BuildQualityChecks@8
        displayName: 'Check code coverage'
        inputs:
          checkWarnings: false
          checkCoverage: true
          coverageFailOption: build
          coverageType: branches
          baseBranchRef: refs/heads/master
          allowCoverageVariance: true
          coverageVariance: 1
          coverageDeltaType: percentage

ElvenSpellmaker commented 3 years ago

@IvanAlekseev We'd really like to use this: https://docs.microsoft.com/en-us/azure/devops/pipelines/test/codecoverage-for-pullrequests?view=azure-devops

But it seems it's not possible to trigger it on anything other than Windows as it needs the .coverage files to work. You'd think in 2020/2021 Microsoft wouldn't work on a feature for Azure DevOps that only worked with .coverage but hey ho.

Thanks for that task, I'll take a look.

IvanAlekseev commented 3 years ago

@ElvenSpellmaker I am with you on this one. It is pretty sad to see 0 progress in Azure DevOps code coverage handling for a year

microsoft / vstest

Add support for dotnet test --collect:"Code Coverage" #981

Description

TL;DR;