Code coverage percentages incorrecly reported

[bjornbouetsmith]

Hi,

I came from opencover because open cover reported my codecoverage incorrectly, i.e. less than what reality is. Altcover reports it even lower.

I am using latest version of altcover: 8.3.839

Reports are collected by sonarcloud and published here: https://sonarcloud.io/project/overview?id=bjornbouetsmith_ROOT.Zfs

I have mstest tests that gets run during the test run and I can see that all the affected tests are successful, but there are certain methods that simply gets reported as 0 codecoverage - even though its impossible because tests would fail if methods were not run - and also when I debug the tests, I can clearly see them being entered:

Code is in a public repository: https://github.com/bjornbouetsmith/ROOT.Zfs

Very tiny repository.

I run altcover via github actions:

dotnet test --no-build Source/**/*.Tests.csproj --no-build --verbosity normal --filter "TestCategory!=Integration" /p:AltCover=true /p:AltCoverReport="/home/runner/work/ROOT.Zfs/ROOT.Zfs/Source/ROOT.Zfs.Tests/coverage.opencover.xml"

When I manually opened up the opencover coverage xml, I could see that it was the xml report that was "wrong" - and I am certain that the same is the issue with altcover

The link to the report I attach as a build artifact are: https://github.com/bjornbouetsmith/ROOT.Zfs/suites/8670659420/artifacts/390430227

Anything I am doing wrong - or is there some limitation in altcover that I am not aware of?

[bjornbouetsmith]

I have also tried running the tests locally on my developer machine (windows) - and the reported coverages from altcover is the same

[bjornbouetsmith]

If I run on windows using the "built" in coverlet collector I get the following result:

+-----------------+--------+--------+--------+
| Module          | Line   | Branch | Method |
+-----------------+--------+--------+--------+
| ROOT.Zfs.Core   | 84.55% | 82.11% | 83.16% |
+-----------------+--------+--------+--------+
| ROOT.Zfs.Public | 78.08% | 54.54% | 89.74% |
+-----------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 83.51% | 78.48% | 86.03% |
+---------+--------+--------+--------+
| Average | 81.31% | 68.32% | 86.44% |
+---------+--------+--------+--------+

Where altcover reports the following:

        ... D:\git\ROOT.Zfs\Source\ROOT.Zfs.Tests\coverage.opencover.xml.0.acv (4.258b)
         14.515 visits recorded in 00:00:00.0092945 (1.561.676 visits/sec)
         A total of 14.515 visits recorded
         Coverage statistics flushing took 0,12 seconds
         Visited Classes 62 of 78 (79.49)
         Visited Methods 173 of 243 (71.19)
         Visited Points 1203 of 1840 (65.38)
         Visited Branches 265 of 638 (41.54)
         Maximum CRAP score 76.23

         ==== Alternative Results (includes all methods including those without corresponding source) ====
         Alternative Visited Classes 79 of 97 (81.44)
         Alternative Visited Methods 228 of 343 (66.47)
         Alternative maximum CRAP score 76,23

[bjornbouetsmith]

Seems like its not easy to make codecoverage correct - using the built in coverlet collector - that also missed a method that is clearly being run otherwise tests would fail.

I wonder if its because its using linq:

if (DataSetProperties.GetAvailableProperties().FirstOrDefault() == null)
{

That line will be converted into generated code, but it seems like something prevents the coverage from being run and that causes the entire contents of the if statement to not be run.

[bjornbouetsmith]

I'm a doofus it seems - it works as expected - I had some code that was run in a different way on the server, causing code to behave slightly different (and still work) - but it caused the branch above to never be run. Sorry for the inconvenience

[SteveGilham]

I'm glad that one was resolved.

There will always be some difference between AltCover's reports and those from coverlet as we differ in presenting the data -- I follow opencover (which in turn followed NCover) in reporting columns covered within lines, and leaving separate the compiler generated methods and types; coverlet drops any column-related debug data, but tries to associate each line with a user-defined function. These differences will show up most obviously in contexts like

if (predicate) { action; }

or

ver output = sequence.Where(predicate).Select(mapping);

where predicate happens to evaluate as false; so to answer your point, LINQ (LINQ one-liners, in particular) can make a difference to the output.