search

OpenCover / opencover

A code coverage tool for .NET 2 and above (WINDOWS OS only), support for 32 and 64 processes with both branch and sequence points
https://blog.many-monkeys.com
Other
1.31k stars 247 forks source link

Converted solution to VS2017 format and dotnet test, and now missing coverage #756

Closed snakefoot closed 5 years ago

snakefoot commented 7 years ago

My Framework

My Environment

I have already...

My issue is related to (check only those which apply):

Expected Behavior

Should include coverage of entire library-code.

Actual Behavior

When using filter, then it only include a single class (BaseMutexFileAppender.cs)

Without filter then it shows the correct coverage of the unit-test-library, but still only a single class of the library being tested.

Steps to reproduce the problem:

OpenCover.Console.exe -register:user -target:"C:\Program Files\dotnet\dotnet.exe" -targetargs:"test tests\NLog.UnitTests --configuration Debug --framework net461" -returntargetcode -filter:"+[NLog]* +[NLog.Extended]* -[NLog]JetBrains.Annotations.*" -excludebyattribute:*.ExcludeFromCodeCoverage* -hideskipped:All -output:coverage.xml

sawilde commented 7 years ago

You'll probably need to try that latest beta build to get coverage from anything built using dotnet this is due to changes in pdb formats that are not supported by the current release.

https://ci.appveyor.com/project/sawilde/opencover/build/4.6.780/artifacts

snakefoot commented 7 years ago

@sawilde Funny how the UnitTest-classes gets coverage, but not the actually Library-classes.

Possible for you to publish a release-candidate nuget-package, that I can grap from the NuGet-feed (easier to use on appveyor.yml) ?

sawilde commented 7 years ago

@snakefoot not yet as it isn't ready - publishing a release candidate creates an expectation

sawilde commented 7 years ago

There is a myget feed - https://www.myget.org/feed/Packages/opencover

snakefoot commented 7 years ago

@sawilde Is this a public feed ? I cannot seem to access it.

sawilde commented 7 years ago

@snakefoot I believe so but you may need a myget account perhaps.

try this:

nuget.exe install OpenCover -Version 4.6.780-rc -Source https://www.myget.org/F/opencover/api/v3/index.json
snakefoot commented 7 years ago

@sawilde Tried the BETA-build, but it is also only picking up some of the coverage:

https://codecov.io/gh/NLog/NLog/commit/615e6d61353a230968c87d5c4dc3641a7f8ca29c/build

This is the normal test-complete output:

=== TEST EXECUTION SUMMARY ===
   NLog.UnitTests  Total: 2225, Errors: 0, Failed: 0, Skipped: 6, Time: 310.009s
Committing...
Visited Classes 390 of 419 (93.08)
Visited Methods 3233 of 3694 (87.52)
Visited Points 19184 of 22782 (84.21)
Visited Branches 7360 of 9066 (81.18)

This is the output with the new solution:

Total tests: 2223. Passed: 2217. Failed: 0. Skipped: 6.
Test Run Successful.
Test execution time: 5.1928 Minutes
Committing...
Visited Classes 130 of 418 (31.1)
Visited Methods 835 of 3695 (22.6)
Visited Points 4977 of 22797 (21.83)
Visited Branches 1871 of 9932 (18.84)
snakefoot commented 7 years ago

Discovered that xUnit 2.0 xunit.console.x86.exe is still able to execute the tests, and now it the result is this:

=== TEST EXECUTION SUMMARY ===
   NLog.UnitTests  Total: 2223, Errors: 0, Failed: 0, Skipped: 6, Time: 412,726s
Committing...
Visited Classes 390 of 418 (93.3)
Visited Methods 3241 of 3695 (87.71)
Visited Points 19233 of 22821 (84.28)
Visited Branches 7379 of 9068 (81.37)
304NotModified commented 7 years ago

Have the same problem here (https://github.com/NLog/NLog.MailKit/pull/5), and that's a lot smaller repo (so easier to test)

quetzalcoatl commented 7 years ago

I've noticed @snakefoot was effectively using dotnet.exe test ... --framework net461 as the OpenCover target command. I have just succeeded in running similar thing, and this seems to be a good place to quickly write some details for other people who try to get it running on various target platforms.

My story is - I work on Win10-Home x64, VS2017 Comunity, OpenCover 4.6.519, xunit mixed 2.2.0 and 2.3.1, and I have a netcoreapp2.0 projects that I wanted to cover. I succeeded, and although that's offtopic another story I've yet to blog about, let me summarize the most important parts for comparison:

(netcoreapp2.0) In covered project csproj:

<TargetFramework>netcoreapp2.0</TargetFramework>
<DebugType>full</DebugType>

(netcoreapp2.0) In test project:

<TargetFramework>netcoreapp2.0</TargetFramework>
<!-- <DebugType>full</DebugType> - NOPE! -->

(netcoreapp2.0) OpenCover runner (single command line, multiline for readability):

OpenCover.Console.exe
    -register:user
    -target:"C:\Program Files\dotnet\dotnet.exe"
    -targetargs:"test --framework netcoreapp2.0 --logger:trx \"MyProject\MyProject.csproj\" "
    -output:"MyProject\TestResults\results-coverage.xml"
    -oldstyle
    -log:Warn

I've spent few days (really, not exagerrating) on researching different options, and that's the only combination that worked. I excluded unimportant things like coverbytest, filters, etc - there were no issues with them, these worked as expected. Similar setup worked for me for netcoreapp1.1 and netcoreapp2.0. Note that I had to enable Full PDBs only for "real projects", and enabling them for test projects did some damage, although I don't remember what exactly (sorry, that was over 2 months ago). Another thing to note is the -oldstyle switch already discussed in other issues. Warn or Info log level helps you to see "No PDBs" infos which help in diagnosing when you get empty reports. I also had to set up output and logger due to multiple projects and multiple test projects to cover in one go.

Now the fun part. I had to switch my net-core-app to net461 due to 3rd party libraries. Application works fine, tests run fine, opencover (--framework net461) returns:

(log level: all)
Cannot instrument [SNIP]\AppData\Local\Temp\[SNIP]\assembly\dl3\[SNIP]\Foobar.WebApi.exe as no PDB/MDB could be loaded

(log level: warn)
No results, this could be for a number of reasons. The most common reasons are:
    1) missing PDBs for the assemblies that match the filter please review the
    output file and refer to the Usage guide (Usage.rtf) about filters.
    2) the profiler may not be registered correctly, please refer to the Usage
    guide and the -register switch.

I similar messages when I battled with netcoreapp, resolved by using full PDBs. Now obviously it was still set, as I just switched the target framework. After another long research, I found out that ... it actually can not locate the PDBs for some reason. Adding -searchdirs parameter made it work for net461!

(net461) In covered project csproj:

<TargetFramework>net461</TargetFramework>
<DebugType>full</DebugType> <!-- dunno, I just left it enabled -->

(net461) In test project:

<TargetFramework>net461</TargetFramework>
<!-- <DebugType>full</DebugType> - dunno, I just left it disabled -->

(net461) OpenCover runner (single command line, multiline for readability, I've left only the important parts):

OpenCover.Console.exe
    -register:user
    -target:"C:\Program Files\dotnet\dotnet.exe"
    -targetargs:"test --framework net461 --logger:trx \"MyProject\MyProject.csproj\" "
    -output:"MyProject\TestResults\results-coverage.xml"
    -oldstyle
    -log:Warn
    -searchdirs:"MyProject\bin\Debug\net461"

I can only guess that either opencover does not look in \bin\$config\$platform for them, or dotnet test/vstest does not copy them properly, or xunit-runner does not copy/shadow them properly..

By the way, I noticed it still works when I remove, oldstyle. As expected, it is no longer needed as runtime is 461 and new instrumentation can be used. However, for my projects, I left the oldstyle and also left the full PDBs, maybe I'll be switching back to netcore at some point.

Anyways I'm absolutely hyped. For the 3rd time in a row (core 1.1, core 2.0, net461), I was dead-tired trying to get it working and almost gave up, and yet again it works out of the box with OpenCover 4.6.519 provided you somehow happen to guess the switches.

Unfortunatelly, I cannot share the full project/script/msbuild setup, and can't prepare a new self-contained project demos (time/effort/nda/license/etc), I'll try to do that a bit later this or next week.

sawilde commented 5 years ago

@quetzalcoatl thanks - I really must get to updating the documentation around this