search

microsoft / codecoverage

MIT License
84 stars 11 forks source link

[cobertura] Missing branch coverage when all branches are covered #104

Open viceice opened 8 months ago

viceice commented 8 months ago

I have two test projects which are running on seperate ci nodes because of different requirements. So i get two coverage files which i like to import in jenkins reports.

The code:

else if (ambiguous && founds == 1)

Project a, which fully covers that line gives:

<line number="61" hits="1" branch="True" />

Project b doesn't cover that line (which is expected) and gives:

<line number="61" hits="0" branch="True" condition-coverage="0% (0/4)">
  <conditions>
    <condition number="0" type="jump" coverage="0%" />
    <condition number="1" type="jump" coverage="0%" />
  </conditions>
</line>

Both reports should be identical beside coverage values.

uhafner commented 8 months ago

It is also worth to mention that in the same report other lines are correctly marked, e.g.:

<line number="84" hits="1" branch="True" condition-coverage="100% (2/2)">
     <conditions>
          <condition number="0" type="jump" coverage="100%" />
     </conditions>
</line>
jakubch1 commented 8 months ago

@viceice could you please upgrade all your test projects to use latest Microsoft.NET.Test.Sdk and Microsoft.CodeCoverage. What commands you use to generate reports?

viceice commented 8 months ago

@viceice could you please upgrade all your test projects to use latest Microsoft.NET.Test.Sdk and Microsoft.CodeCoverage.

@jakubch1 I'm using the latest non-preview (17.9.0) of Microsoft.NET.Test.Sdk which has a direct dependency to Microsoft.CodeCoverage v17.9.0.

I'm using lockfiles and central versioning, so versions are verified. Should i try the latest preview?

What commands you use to generate reports?

I'm using a runsettings file:

<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
  <RunConfiguration>
    <TargetPlatform>x64</TargetPlatform>
  </RunConfiguration>

  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0">
        <Configuration>
          <Format>Cobertura</Format>
          <CodeCoverage>
            <ModulePaths>
              <!-- Disable native instrumentation, causes SharePoint debug issues -->
              <EnableStaticNativeInstrumentation>False</EnableStaticNativeInstrumentation>
              <EnableDynamicNativeInstrumentation>False</EnableDynamicNativeInstrumentation>
              <Include>
                <ModulePath>Proj(\.[.a-zA-Z]+)?\.dll$</ModulePath>
              </Include>
              <Exclude>
                <ModulePath>Proj(\..+)?\.Test\.dll$</ModulePath>
              </Exclude>
            </ModulePaths>
            <Attributes>
              <Exclude>
                <Attribute>^System\.CodeDom\.Compiler\.GeneratedCodeAttribute$</Attribute>
                <Attribute>^System\.Diagnostics\.CodeAnalysis\.ExcludeFromCodeCoverageAttribute$</Attribute>
              </Exclude>
            </Attributes>
          </CodeCoverage>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>

  <NUnit>
    <TestOutputXml>nunit</TestOutputXml>
    <OutputXmlFolderMode>RelativeToResultDirectory</OutputXmlFolderMode>
    <NewOutputXmlFileForEachRun>true</NewOutputXmlFileForEachRun>
  </NUnit>
</RunSettings>

which is referenced in repo root Directory.Build.props like this:

  <RunSettingsFilePath>$(MSBuildThisFileDirectory)/proj.runsettings</RunSettingsFilePath>
jakubch1 commented 8 months ago

@viceice thanks for info. How you execute tests? dotnet test on solution level? Could you please post output of those operations?

viceice commented 8 months ago

I'm running two comands on CI.

First running cheap tests in ./test folder on a general jenkins node inside a docker windows 2022 container. There is a sln and some test projects.

dotnet test test --no-build
...
Passed!  - Failed:     0, Passed:   680, Skipped:    14, Total:   694, Duration: 17 s - Proj.Test.dll (net8.0)

Attachments:
  ~\proj\test\Proj.XY.Test\TestResults\c64ddcc7-72a7-45a8-aae4-32e5cafbb395\user_PCNAME_2024-02-29.08_36_39.cobertura.xml

This gives the first coverage file.

The second command runs on a native SharePoint server for CI (jenkins node) purposes. So we only run SharePoint tests here. There is a sln and one test projects.

dotnet test sp/test --no-build
...
Passed!  - Failed:     0, Passed:   680, Skipped:    14, Total:   694, Duration: 17 s - Proj.SP.Test.dll (net48)

Attachments:
  ~\proj\sp\test\Proj.SP.Test\TestResults\5f49f3d0-341e-4fe5-83ac-e6e2bd5f991e\user_PCNAME_2024-02-29.08_37_31.cobertura.xml
jakubch1 commented 8 months ago

@viceice thanks for info. What version of .NET SDK you use?

@fhnaseer I suspect that that dotnet test test --no-build is executing several tests projects and doing at the end attachments merging. Could you please check which version of merging is used here? I remember some old merging was dropping conditions.

viceice commented 8 months ago

my local system

> dotnet --info                                                                                                                        ─╯
.NET SDK:
 Version:           8.0.200
 Commit:            438cab6a9d
 Workload version:  8.0.200-manifests.91505830

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.22631
 OS Platform: Windows
 RID:         win-x64
 Base Path:   C:\Program Files\dotnet\sdk\8.0.200\

.NET workloads installed:
There are no installed workloads to display.

Host:
  Version:      8.0.2
  Architecture: x64
  Commit:       1381d5ebd2

.NET SDKs installed:
  6.0.419 [C:\Program Files\dotnet\sdk]
  7.0.406 [C:\Program Files\dotnet\sdk]
  8.0.102 [C:\Program Files\dotnet\sdk]
  8.0.200 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.10 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.13 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.27 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.16 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.10 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.15 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.27 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.16 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 6.0.3 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.5 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.10 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.13 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.15 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.27 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 7.0.2 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 7.0.16 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 8.0.2 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found:
  x86   [C:\Program Files (x86)\dotnet]
    registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

Can't check CI right now, but can reproduce locally with above versions.

I've also Visual Studio 17.9.1 on my local machine and the corresponding build tools on CI.

jakubch1 commented 8 months ago

@viceice thanks for reporting this. We will work on fixing it when using dotnet test. As workaround you could try to switch to use dotnet-coverage tool during test run:

dotnet-coverage collect -s coverage.config "dotnet test test --no-build"

You would need also to remove runsettings file from csproj and convert it to coverage.config (main tag should be <Configuration>).

viceice commented 8 months ago

my current workaround is to use dotnet coverage merge to merge both reports, which can then imported to jenkins.

viceice commented 8 months ago

is there any other benefit to use dotnet coverage over build-in ?

jakubch1 commented 8 months ago

By using dotnet-coverage you get better performance and always latest features.

Build-in tool is running coverage for each test project separately and then merges results. While dotnet-coverage is using 1 collection for all test projects and merging is not needed. It means that if same project is referenced in many test projects build-in needs to instrument it several times. External tool instruments just once. Also external tool targets .net6 and is more optimized.

viceice commented 8 months ago

ok, will give it a try when our infra is back running 🙈

neverless this bug should be fixed anyways 🙃

viceice commented 8 months ago

OK, dotnet-coverage works partly better. I now have totally missing lines in one report. they are condittional net8.0 only and can't be coverred by net48 of cause:

#if NETFRAMEWORK
// some full framework code
#else
// some dotnet core code
#endif

So the lines for the non-full framework are missing in the sp report.

fhnaseer commented 8 months ago

@viceice Can you please tell version of dotnet-coverage?

viceice commented 8 months ago

@viceice Can you please tell version of dotnet-coverage?

still same 17.9.0 as before

it's 17.10.3

fhnaseer commented 8 months ago

I am looking at the Jenkins ticket. With the latest dotnet-coverage merge it is working as expected, right?

jakubch1 commented 8 months ago

@fhnaseer original issue (missing conditions) is not when merging is done by dotnet-coverage. I think case here is when there is attachments processing (merging) done by TP/.NET SDK.

@viceice can you please list all versions of Microsoft.NET.Test.Sdk for all projects?

viceice commented 8 months ago

it's same as already mentioned above. I'm using central versioning and lockfiles. I also verified before.

fhnaseer commented 8 months ago

I'm running two comands on CI.

First running cheap tests in ./test folder on a general jenkins node inside a docker windows 2022 container. There is a sln and some test projects.

dotnet test test --no-build
...
Passed!  - Failed:     0, Passed:   680, Skipped:    14, Total:   694, Duration: 17 s - Proj.Test.dll (net8.0)

Attachments:
  ~\proj\test\Proj.XY.Test\TestResults\c64ddcc7-72a7-45a8-aae4-32e5cafbb395\user_PCNAME_2024-02-29.08_36_39.cobertura.xml

This gives the first coverage file.

The second command runs on a native SharePoint server for CI (jenkins node) purposes. So we only run SharePoint tests here. There is a sln and one test projects.

dotnet test sp/test --no-build
...
Passed!  - Failed:     0, Passed:   680, Skipped:    14, Total:   694, Duration: 17 s - Proj.SP.Test.dll (net48)

Attachments:
  ~\proj\sp\test\Proj.SP.Test\TestResults\5f49f3d0-341e-4fe5-83ac-e6e2bd5f991e\user_PCNAME_2024-02-29.08_37_31.cobertura.xml

@viceice Can you please run test on individual test projects (rather on solution level) and share coverage files generated per test project?

viceice commented 8 months ago

I can only provide parts of it. hopefully tomorrow.

I think the lines already mentioned are the interesting ones.

jakubch1 commented 8 months ago

@viceice we believe what is happening is your case:

  1. When you run dotnet test test --no-build you execute tests for several projects and our internal logic is generating coverage report for each project separately (with conditions, you don't even see those reports on output)
  2. Our logic is invoking merging of all reports from point 1. As logic is not able to correctly merge conditions from all reports => conditions are dropped. This issue is explained here: https://github.com/microsoft/codecoverage/issues/11#issuecomment-1457935172 Unfortunately because of not full information is available in cobertura report, conditions have to be dropped in certain scenarios.

To make sure that this is your scenario we need to get information about line 61 for all projects from point 1 (separately) and if this is the scenario we will not be able to fix it. The only way is to use dotnet-coverage tool which has a little bit different architecture and is able to merge all conditions as it contains additional information during merge.

viceice commented 8 months ago

Report a

          <lines>
            <line number="15" hits="1" branch="False" />
            <line number="21" hits="1" branch="False" />
            <line number="23" hits="0" branch="False" />
            <line number="25" hits="0" branch="False" />
            <line number="26" hits="1" branch="False" />
            <line number="28" hits="1" branch="False" />
            <line number="29" hits="1" branch="False" />
            <line number="30" hits="1" branch="False" />
            <line number="31" hits="1" branch="False" />
            <line number="32" hits="1" branch="False" />
            <line number="33" hits="1" branch="False" />
            <line number="34" hits="1" branch="False" />
            <line number="37" hits="1" branch="False" />
            <line number="38" hits="1" branch="True" />
            <line number="39" hits="0" branch="False" />
            <line number="40" hits="0" branch="False" />
            <line number="41" hits="0" branch="False" />
            <line number="44" hits="1" branch="False" />
            <line number="45" hits="1" branch="True" condition-coverage="100% (4/4)">
              <conditions>
                <condition number="0" type="jump" coverage="100%" />
                <condition number="1" type="jump" coverage="100%" />
              </conditions>
            </line>
            <line number="46" hits="1" branch="False" />
            <line number="47" hits="1" branch="False" />
            <line number="48" hits="1" branch="False" />
            <line number="50" hits="1" branch="True" condition-coverage="100% (2/2)">
              <conditions>
                <condition number="0" type="jump" coverage="100%" />
              </conditions>
            </line>
            <line number="51" hits="1" branch="False" />
            <line number="52" hits="1" branch="False" />
            <line number="54" hits="1" branch="False" />
            <line number="55" hits="1" branch="True" />
            <line number="56" hits="1" branch="False" />
            <line number="58" hits="0" branch="False" />
            <line number="59" hits="0" branch="False" />
            <line number="61" hits="1" branch="True" />
            <line number="62" hits="1" branch="False" />
            <line number="63" hits="1" branch="False" />
            <line number="64" hits="1" branch="False" />
            <line number="67" hits="0" branch="False" />
            <line number="68" hits="0" branch="False" />
            <line number="69" hits="1" branch="False" />
            <line number="71" hits="1" branch="False" />
            <line number="73" hits="1" branch="False" />
          </lines>

Report b

          <lines>
            <line number="15" hits="0" branch="False" />
            <line number="21" hits="0" branch="False" />
            <line number="23" hits="0" branch="False" />
            <line number="25" hits="0" branch="False" />
            <line number="26" hits="0" branch="False" />
            <line number="28" hits="0" branch="False" />
            <line number="29" hits="0" branch="False" />
            <line number="30" hits="0" branch="False" />
            <line number="31" hits="0" branch="False" />
            <line number="32" hits="0" branch="False" />
            <line number="33" hits="0" branch="False" />
            <line number="34" hits="0" branch="False" />
            <line number="37" hits="0" branch="False" />
            <line number="38" hits="0" branch="True" condition-coverage="0% (0/2)">
              <conditions>
                <condition number="0" type="jump" coverage="0%" />
              </conditions>
            </line>
            <line number="39" hits="0" branch="False" />
            <line number="40" hits="0" branch="False" />
            <line number="41" hits="0" branch="False" />
            <line number="44" hits="0" branch="False" />
            <line number="45" hits="0" branch="True" condition-coverage="0% (0/4)">
              <conditions>
                <condition number="0" type="jump" coverage="0%" />
                <condition number="1" type="jump" coverage="0%" />
              </conditions>
            </line>
            <line number="46" hits="0" branch="False" />
            <line number="47" hits="0" branch="False" />
            <line number="48" hits="0" branch="False" />
            <line number="50" hits="0" branch="True" condition-coverage="0% (0/2)">
              <conditions>
                <condition number="0" type="jump" coverage="0%" />
              </conditions>
            </line>
            <line number="51" hits="0" branch="False" />
            <line number="52" hits="0" branch="False" />
            <line number="54" hits="0" branch="False" />
            <line number="55" hits="0" branch="True" condition-coverage="0% (0/2)">
              <conditions>
                <condition number="0" type="jump" coverage="0%" />
              </conditions>
            </line>
            <line number="56" hits="0" branch="False" />
            <line number="58" hits="0" branch="False" />
            <line number="59" hits="0" branch="False" />
            <line number="61" hits="0" branch="True" condition-coverage="0% (0/4)">
              <conditions>
                <condition number="0" type="jump" coverage="0%" />
                <condition number="1" type="jump" coverage="0%" />
              </conditions>
            </line>
            <line number="62" hits="0" branch="False" />
            <line number="63" hits="0" branch="False" />
            <line number="64" hits="0" branch="False" />
            <line number="67" hits="0" branch="False" />
            <line number="68" hits="0" branch="False" />
            <line number="69" hits="0" branch="False" />
            <line number="71" hits="0" branch="False" />
            <line number="73" hits="0" branch="False" />
          </lines>

Report c

          <lines>
            <line number="15" hits="0" branch="False" />
            <line number="21" hits="0" branch="False" />
            <line number="23" hits="0" branch="False" />
            <line number="25" hits="0" branch="False" />
            <line number="26" hits="0" branch="False" />
            <line number="28" hits="0" branch="False" />
            <line number="29" hits="0" branch="False" />
            <line number="30" hits="0" branch="False" />
            <line number="31" hits="0" branch="False" />
            <line number="32" hits="0" branch="False" />
            <line number="33" hits="0" branch="False" />
            <line number="34" hits="0" branch="False" />
            <line number="37" hits="0" branch="False" />
            <line number="38" hits="0" branch="True" condition-coverage="0% (0/2)">
              <conditions>
                <condition number="0" type="jump" coverage="0%" />
              </conditions>
            </line>
            <line number="39" hits="0" branch="False" />
            <line number="40" hits="0" branch="False" />
            <line number="41" hits="0" branch="False" />
            <line number="44" hits="0" branch="False" />
            <line number="45" hits="0" branch="True" condition-coverage="0% (0/4)">
              <conditions>
                <condition number="0" type="jump" coverage="0%" />
                <condition number="1" type="jump" coverage="0%" />
              </conditions>
            </line>
            <line number="46" hits="0" branch="False" />
            <line number="47" hits="0" branch="False" />
            <line number="48" hits="0" branch="False" />
            <line number="50" hits="0" branch="True" condition-coverage="0% (0/2)">
              <conditions>
                <condition number="0" type="jump" coverage="0%" />
              </conditions>
            </line>
            <line number="51" hits="0" branch="False" />
            <line number="52" hits="0" branch="False" />
            <line number="54" hits="0" branch="False" />
            <line number="55" hits="0" branch="True" condition-coverage="0% (0/2)">
              <conditions>
                <condition number="0" type="jump" coverage="0%" />
              </conditions>
            </line>
            <line number="56" hits="0" branch="False" />
            <line number="58" hits="0" branch="False" />
            <line number="59" hits="0" branch="False" />
            <line number="61" hits="0" branch="True" condition-coverage="0% (0/4)">
              <conditions>
                <condition number="0" type="jump" coverage="0%" />
                <condition number="1" type="jump" coverage="0%" />
              </conditions>
            </line>
            <line number="62" hits="0" branch="False" />
            <line number="63" hits="0" branch="False" />
            <line number="64" hits="0" branch="False" />
            <line number="67" hits="0" branch="False" />
            <line number="68" hits="0" branch="False" />
            <line number="69" hits="0" branch="False" />
            <line number="71" hits="0" branch="False" />
            <line number="73" hits="0" branch="False" />
          </lines>

Report d

          <lines>
            <line number="15" hits="0" branch="False" />
            <line number="21" hits="0" branch="False" />
            <line number="23" hits="0" branch="False" />
            <line number="25" hits="0" branch="False" />
            <line number="26" hits="0" branch="False" />
            <line number="28" hits="0" branch="False" />
            <line number="29" hits="0" branch="False" />
            <line number="30" hits="0" branch="False" />
            <line number="31" hits="0" branch="False" />
            <line number="32" hits="0" branch="False" />
            <line number="33" hits="0" branch="False" />
            <line number="34" hits="0" branch="False" />
            <line number="37" hits="0" branch="False" />
            <line number="38" hits="0" branch="True" condition-coverage="0% (0/2)">
              <conditions>
                <condition number="0" type="jump" coverage="0%" />
              </conditions>
            </line>
            <line number="39" hits="0" branch="False" />
            <line number="40" hits="0" branch="False" />
            <line number="41" hits="0" branch="False" />
            <line number="44" hits="0" branch="False" />
            <line number="45" hits="0" branch="True" condition-coverage="0% (0/4)">
              <conditions>
                <condition number="0" type="jump" coverage="0%" />
                <condition number="1" type="jump" coverage="0%" />
              </conditions>
            </line>
            <line number="46" hits="0" branch="False" />
            <line number="47" hits="0" branch="False" />
            <line number="48" hits="0" branch="False" />
            <line number="50" hits="0" branch="True" condition-coverage="0% (0/2)">
              <conditions>
                <condition number="0" type="jump" coverage="0%" />
              </conditions>
            </line>
            <line number="51" hits="0" branch="False" />
            <line number="52" hits="0" branch="False" />
            <line number="54" hits="0" branch="False" />
            <line number="55" hits="0" branch="True" condition-coverage="0% (0/2)">
              <conditions>
                <condition number="0" type="jump" coverage="0%" />
              </conditions>
            </line>
            <line number="56" hits="0" branch="False" />
            <line number="58" hits="0" branch="False" />
            <line number="59" hits="0" branch="False" />
            <line number="61" hits="0" branch="True" condition-coverage="0% (0/4)">
              <conditions>
                <condition number="0" type="jump" coverage="0%" />
                <condition number="1" type="jump" coverage="0%" />
              </conditions>
            </line>
            <line number="62" hits="0" branch="False" />
            <line number="63" hits="0" branch="False" />
            <line number="64" hits="0" branch="False" />
            <line number="67" hits="0" branch="False" />
            <line number="68" hits="0" branch="False" />
            <line number="69" hits="0" branch="False" />
            <line number="71" hits="0" branch="False" />
            <line number="73" hits="0" branch="False" />
          </lines>
viceice commented 8 months ago

Only project a fully covers the line 61 (see code in issue description)

jakubch1 commented 8 months ago

Is project a single test project or solution? Could you please provide diagnostic logs from project a? https://github.com/microsoft/vstest/blob/main/docs/troubleshooting.md#collect-logs-and-crash-dump

viceice commented 8 months ago

Is project a single test project or solution? Could you please provide diagnostic logs from project a?

Yes, there are 4 test projects, but they depend on each other (tree)

https://github.com/microsoft/vstest/blob/main/docs/troubleshooting.md#collect-logs-and-crash-dump Will try todo when i've some time. Will there be our private source code inside those dumps?

jakubch1 commented 8 months ago

You can skip dumps. Enough for us is: --diag:log.txt

viceice commented 8 months ago

wait, that project build's for net8.0 and net48. so probably another coverage result merge happens?

now running for both frameworks seperately.

viceice commented 8 months ago

Project a with net8.0

          <lines>
            <line number="15" hits="1" branch="False" />
            <line number="21" hits="1" branch="False" />
            <line number="23" hits="0" branch="False" />
            <line number="25" hits="0" branch="False" />
            <line number="26" hits="1" branch="False" />
            <line number="28" hits="1" branch="False" />
            <line number="29" hits="1" branch="False" />
            <line number="30" hits="1" branch="False" />
            <line number="31" hits="1" branch="False" />
            <line number="32" hits="1" branch="False" />
            <line number="33" hits="1" branch="False" />
            <line number="34" hits="1" branch="False" />
            <line number="37" hits="1" branch="False" />
            <line number="38" hits="1" branch="True" condition-coverage="50% (1/2)">
              <conditions>
                <condition number="0" type="jump" coverage="50%" />
              </conditions>
            </line>
            <line number="39" hits="0" branch="False" />
            <line number="40" hits="0" branch="False" />
            <line number="41" hits="0" branch="False" />
            <line number="44" hits="1" branch="False" />
            <line number="45" hits="1" branch="True" condition-coverage="100% (4/4)">
              <conditions>
                <condition number="0" type="jump" coverage="100%" />
                <condition number="1" type="jump" coverage="100%" />
              </conditions>
            </line>
            <line number="46" hits="1" branch="False" />
            <line number="47" hits="1" branch="False" />
            <line number="48" hits="1" branch="False" />
            <line number="50" hits="1" branch="True" condition-coverage="100% (2/2)">
              <conditions>
                <condition number="0" type="jump" coverage="100%" />
              </conditions>
            </line>
            <line number="51" hits="1" branch="False" />
            <line number="52" hits="1" branch="False" />
            <line number="54" hits="1" branch="False" />
            <line number="55" hits="1" branch="True" condition-coverage="50% (1/2)">
              <conditions>
                <condition number="0" type="jump" coverage="50%" />
              </conditions>
            </line>
            <line number="56" hits="1" branch="False" />
            <line number="58" hits="0" branch="False" />
            <line number="59" hits="0" branch="False" />
            <line number="61" hits="1" branch="True" condition-coverage="50% (2/4)">
              <conditions>
                <condition number="0" type="jump" coverage="50%" />
                <condition number="1" type="jump" coverage="50%" />
              </conditions>
            </line>
            <line number="62" hits="1" branch="False" />
            <line number="63" hits="1" branch="False" />
            <line number="64" hits="1" branch="False" />
            <line number="67" hits="0" branch="False" />
            <line number="68" hits="0" branch="False" />
            <line number="69" hits="1" branch="False" />
            <line number="71" hits="1" branch="False" />
            <line number="73" hits="1" branch="False" />
          </lines>

Project a with net48

          <lines>
            <line number="15" hits="1" branch="False" />
            <line number="21" hits="1" branch="False" />
            <line number="23" hits="0" branch="False" />
            <line number="25" hits="0" branch="False" />
            <line number="26" hits="1" branch="False" />
            <line number="28" hits="1" branch="False" />
            <line number="29" hits="1" branch="False" />
            <line number="30" hits="1" branch="False" />
            <line number="31" hits="1" branch="False" />
            <line number="32" hits="1" branch="False" />
            <line number="33" hits="1" branch="False" />
            <line number="34" hits="1" branch="False" />
            <line number="37" hits="1" branch="False" />
            <line number="38" hits="1" branch="True" condition-coverage="50% (1/2)">
              <conditions>
                <condition number="0" type="jump" coverage="50%" />
              </conditions>
            </line>
            <line number="39" hits="0" branch="False" />
            <line number="40" hits="0" branch="False" />
            <line number="41" hits="0" branch="False" />
            <line number="44" hits="1" branch="False" />
            <line number="45" hits="1" branch="True" condition-coverage="100% (4/4)">
              <conditions>
                <condition number="0" type="jump" coverage="100%" />
                <condition number="1" type="jump" coverage="100%" />
              </conditions>
            </line>
            <line number="46" hits="1" branch="False" />
            <line number="47" hits="1" branch="False" />
            <line number="48" hits="1" branch="False" />
            <line number="50" hits="1" branch="True" condition-coverage="100% (2/2)">
              <conditions>
                <condition number="0" type="jump" coverage="100%" />
              </conditions>
            </line>
            <line number="51" hits="1" branch="False" />
            <line number="52" hits="1" branch="False" />
            <line number="54" hits="1" branch="False" />
            <line number="55" hits="1" branch="True" condition-coverage="50% (1/2)">
              <conditions>
                <condition number="0" type="jump" coverage="50%" />
              </conditions>
            </line>
            <line number="56" hits="1" branch="False" />
            <line number="58" hits="0" branch="False" />
            <line number="59" hits="0" branch="False" />
            <line number="61" hits="1" branch="True" condition-coverage="50% (2/4)">
              <conditions>
                <condition number="0" type="jump" coverage="50%" />
                <condition number="1" type="jump" coverage="50%" />
              </conditions>
            </line>
            <line number="62" hits="1" branch="False" />
            <line number="63" hits="1" branch="False" />
            <line number="64" hits="1" branch="False" />
            <line number="67" hits="0" branch="False" />
            <line number="68" hits="0" branch="False" />
            <line number="69" hits="1" branch="False" />
            <line number="71" hits="1" branch="False" />
            <line number="73" hits="1" branch="False" />
          </lines>
viceice commented 8 months ago

So it seems it's again the wrong merge when running both framework tests once 🤔

jakubch1 commented 8 months ago

Unfortunately this can't be fixed because:

            <line number="61" hits="1" branch="True" condition-coverage="50% (2/4)">
              <conditions>
                <condition number="0" type="jump" coverage="50%" />
                <condition number="1" type="jump" coverage="50%" />
              </conditions>
            </line>

means line has 2 conditions and for each one case is covered. We are not able to calculate correct merge result (we don't know which part of condition was really covered) so we decide to remove conditions (similar to ReportGenerator). It is limitation of cobertura format.

50% + 50% could be merged to 50% or 100%.

The only workaround is to use dotnet-coverage collect to execute all your tests. In this scenario we keep internally additional data (which exactly part of condition was covered) and we are able to perform full merge.

viceice commented 8 months ago

Thanks, i already use dotnet-coverage collect now on ci runs and the normal collector inside visual studio to see coverage there (Fine Code Coverage plugin).

uhafner commented 8 months ago

Unfortunately this can't be fixed because:

            <line number="61" hits="1" branch="True" condition-coverage="50% (2/4)">
              <conditions>
                <condition number="0" type="jump" coverage="50%" />
                <condition number="1" type="jump" coverage="50%" />
              </conditions>
            </line>

means line has 2 conditions and for each one case is covered. We are not able to calculate correct merge result (we don't know which part of condition was really covered) so we decide to remove conditions (similar to ReportGenerator). It is limitation of cobertura format.

50% + 50% could be merged to 50% or 100%.

The only workaround is to use dotnet-coverage collect to execute all your tests. In this scenario we keep internally additional data (which exactly part of condition was covered) and we are able to perform full merge.

I understand that this is a limitation of the cobertura report format (or more precisely this is a problem of all reports, the actual coverage can only be computed with the execution data).

Despite of this architectural problem I think it would be better if you change the output in such cases. Your current output is brach=true without a condition-coverage. That means that the condition coverage will be automatically set to 100% (see DTD). I think it would be better to set in this case also the condition-coverage="100% (4/4) explicitly. Then other tools can report at least the correct number of branches (if covered or not is undefined, so the <conditions> can be skipped. (Another option would be to use the minimum coverage, so 50%)

baumheld commented 7 months ago 
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0">
        <Configuration>
          <Format>Cobertura</Format>
          <CodeCoverage>
            <ModulePaths>
              <!-- Disable native instrumentation, causes SharePoint debug issues -->
              <EnableStaticNativeInstrumentation>False</EnableStaticNativeInstrumentation>
              <EnableDynamicNativeInstrumentation>False</EnableDynamicNativeInstrumentation>
            </ModulePaths>
            [...]
          </CodeCoverage>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>

@viceice Just a heads up. Are you sure <EnableStaticNativeInstrumentation> and <EnableDynamicNativeInstrumentation> should be under the node <ModulePaths>? Shouldn't it be one level up, under <CodeCoverage> according to these docs?

viceice commented 7 months ago

no, was a typo and already moved to correct place