Support of Visual Studio Code Coverage reports

[guwirth]

Hi,

Would be nice if plugin would also support Visual Studio Code Coverage reports.

Regards

[wenns]

Hi Guenter,

can you provide some background to what would we gain. Does it have advantages over the channels / formats we already have. What formats does VS provide, which tools would one use etc.

[guwirth]

Hi,

Use Case: Support Visual Studio coverage files with C++ plugin

Steps to do: 1) Generate a Visual Studio 2012 coverage file with: C:\Program Files (x86)\Microsoft Visual Studio 11.0\Team Tools\Dynamic Code Coverage Tools\CodeCoverage.exe" collect /output:out.coverage (...)

Where (...) is performed like (could be also another test runner) mstest.exe /testmetadata:.\some.vsmdi /testlist:03_COMP_smoke/clientAndServer/pass /runconfig:.\localtestrun2010.testrunconfig /resultsfile:testResults.trx

The CodeCoverage.exe is run as master of the unit test runner.

2) Generate a native xml file from the coverage file: C:\Program Files (x86)\Microsoft Visual Studio 11.0\Team Tools\Dynamic Code Coverage Tools\CodeCoverage.exe" analyze /output:mstest_coverage.xml out.coverage

3) The mstest_coverage.xml file now has to be processed for SonarQube.

Thinking awhile over this I think it would be sufficient to add also a possibility for a XSL transformation for coverage data:

sonar.cxx.coverage.xsltURL= …

Regards

[jmecosta]

i might be interested in this feature, we are paying bullseye licences to get coverage and we also pay visual studio licenses. So might be that we scrap one, so there is definitely the need to visual studio users.

On Wed, Dec 4, 2013 at 12:02 PM, Günter Wirth notifications@github.comwrote:

Hi,

Use Case: Support Visual Studio coverage files with C++ plugin

Steps to do: 1) Generate a Visual Studio 2012 coverage file with: C:\Program Files (x86)\Microsoft Visual Studio 11.0\Team Tools\Dynamic Code Coverage Tools\CodeCoverage.exe" collect /output:out.coverage (...)

Where (...) is performed like (could be also another test runner) mstest.exe /testmetadata:.\some.vsmdi /testlist:03_COMP_smoke/clientAndServer/pass /runconfig:.\localtestrun2010.testrunconfig /resultsfile:testResults.trx

The CodeCoverage.exe is run as master of the unit test runner.

2) Generate a native xml file from the coverage file: C:\Program Files (x86)\Microsoft Visual Studio 11.0\Team Tools\Dynamic Code Coverage Tools\CodeCoverage.exe" analyze /output:mstest_coverage.xml out.coverage

3) The mstest_coverage.xml file now has to be processed for SonarQube.

Thinking awhile over this I think it would be sufficient to add also a possibility for a XSL transformation for coverage data:

sonar.cxx.coverage.xsltURL= …

Regards

— Reply to this email directly or view it on GitHubhttps://github.com/wenns/sonar-cxx/issues/79#issuecomment-29792458 .

[wenns]

@guwirth Thanks for summarizing this up. I could add another point: the gcov-chain on Linux (at least for the releases we tried so far) does not produce high-quality results. The branch counts are often wrong, there are branches listed on places where there are'nt any (at least in source code) etc. Apropos: how does the MS toolchain feels like? Does it produce acceptable results?

The other tool-chain we support right now is bullseye. For me, its not ideal either: it doesnt support line coverage and doesnt map good to the Sonar model.

As for the 'sonar.cxx.coverage.xsltURL' thing: I dont like it. My feeling (and experience, partly) is that it doesnt give us much. I think, transformation should be done outside of the plugin (using Python, Perl, ..., and whatever approach you like, including XSLT, if you're a fan of).

[Bertk]

Support for MS-test results are also interesting for me and we use bullseye until now which could then be discontinued. By the way, the C# plug-in V2.2. (Feb. 2014) will support MS-Test.

SONARDOTNT-357 - Make it possible to reuse MSTest reports SONARDOTNT-358 - Make it possible to reuse MS Coverage reports

[wenns]

To draw a conclusion: that would make sense, apparently. Im not interested in this, personally, so if you guys are, promote the issue to 'enhancement' and give it a shot.

The implementation choices from my point of view are: either write something external to convert to cobertura format or add another parser to the coverage sensor. Converting to bullseye format doesnt make sense as the format maps badly to the sonar model. BTW: does MS coverage support both line and branch coverage? cannot quite understand what I find via google.

[guwirth]

... how does the MS toolchain feels like?

After configuring tool in the right way results are not bad but also sometimes suprising: See http://blogs.msdn.com/b/jsocha/archive/2011/08/17/interpreting-c-code-coverage-results.aspx

... bullseye ... doesnt support line coverage

Visual Studio is supporting:

Block-based statement coverage For the purposes of the tools, a block is defined as a sequence of instructions that have a single entry point and a single exit point. Exit points include branch instructions, a function call, a return instruction, or, for managed code, a throw instruction.

Line-based coverage For line-based coverage, the tools identify all of the blocks that make up a line and then use this information to determine the level of coverage for the line. If all of the blocks that make up the line are covered, then the tools report that the line is covered. If no blocks in the line are covered, then the tools report that the line is not covered. If some, but not all, of the blocks in the line are covered, then the tools report that the line is partially covered.

See also http://msdn.microsoft.com/en-us/library/dd537628.aspx

As for the 'sonar.cxx.coverage.xsltURL' thing: I dont like it.

I think plugin should be consistent. Transformation should be available for all or none XML reports.

[wenns]

On 12/05/2013 10:42 PM, Günter Wirth wrote:

    ... how does the MS toolchain feels like?

After configuring tool in the right way results are not bad but also sometimes suprising: See http://blogs.msdn.com/b/jsocha/archive/2011/08/17/interpreting-c-code-coverage-results.aspx

Thanks for the link, Guenter. I expected something like this: there are no simple things in C++ land ;) But its all good as long we can get it compressed/converted to the Sonar model.

    ... bullseye ... doesnt support line coverage

Visual Studio is supporting:

Block-based statement coverage For the purposes of the tools, a block is defined as a sequence of instructions that have a single entry point and a single exit point. Exit points include branch instructions, a function call, a return instruction, or, for managed code, a throw instruction.

Line-based coverage For line-based coverage, the tools identify all of the blocks that make up a line and then use this information to determine the level of coverage for the line. If all of the blocks that make up the line are covered, then the tools report that the line is covered. If no blocks in the line are covered, then the tools report that the line is not covered. If some, but not all, of the blocks in the line are covered, then the tools report that the line is partially covered.

See also http://msdn.microsoft.com/en-us/library/dd537628.aspx

    As for the 'sonar.cxx.coverage.xsltURL' thing: I dont like it.

I think plugin should be consistent. Transformation should be available for all or none XML reports.

So think I. And as I strive for the second alternative (Transformation should be available for ... none XML reports) in the mid-term, Id like to not build more stuff on it.

— Reply to this email directly or view it on GitHub https://github.com/wenns/sonar-cxx/issues/79#issuecomment-29941178.

[guwirth]

One of the biggest advantages of the community plugin is the openness. So if there is an easy way in Java or Sonar to support transformation I would vote for it. As more out of the box is supported as easier the plugin is to use.

[wenns]

Conceptually true, but in this case: the is an "easy way to support transformation in Java or Sonar" namely: $ | conversion | sonar_runner Just put the conversion step logically between the 'report creator' and Sonar. I'd say it's as easy as the current Cxx-solution and even more flexible, because you are not constrained to use XSLT. The first appoach also looks better from the engineering POV because of looser coupling.

[guwirth]

Would be helpful to add a 'Design Decisions' page to Wiki to document such things.

[guwirth]

The other tool-chain we support right now is bullseye. For me, its not ideal either: it doesnt support line coverage and doesnt map good to the Sonar model.

@wenns What exactly is the problem with Bullseye?

BullseyeCoverage Measurement Technique: http://www.bullseye.com/measurementTechnique.html Why do you think line coverage is missing (http://www.bullseye.com/coverage.html#basic_statement)? Isn't Condition/Decision Coverage the better choice?

[jmecosta]

Bullseye provides function coverage + condition/decision coverage only.

Line coverage is not available.

Conceptually the sonar model is completely different than this, in sonar they count lines hits + branch coverage. Branch coverage in sonar sucks because it does not give you which condition are covered only the absolute number.

In bullseye this is available and its really helpull to create new test cases to produce more coverage.

So what we done was to approximate the 2 models, the condition/decision is well mapped to sonar model although we loose the more information. And the function coverage is mapped to line coverage, which is not visually correct I sonar since you will see lines inside functions without any coverage information On Feb 22, 2014 5:35 PM, "Günter Wirth" notifications@github.com wrote:

The other tool-chain we support right now is bullseye. For me, its not ideal either: it doesnt support line coverage and doesnt map good to the Sonar model.

@wenns https://github.com/wenns What exactly is the problem with Bullseye?

BullseyeCoverage Measurement Technique: http://www.bullseye.com/measurementTechnique.html Why do you think line coverage is missing ( http://www.bullseye.com/coverage.html#basic_statement)? Isn't Condition/Decision Coverage the better choice?

Reply to this email directly or view it on GitHubhttps://github.com/wenns/sonar-cxx/issues/79#issuecomment-35805367 .

[jmecosta]

Another think I forgot, when using bullseye sensor line coverage is actually function coverage in the sonar ui. This is a bit confusing if you use the sonar ui since it calls it line coverage On Feb 22, 2014 5:50 PM, "Jorge Costa" jmecosta@gmail.com wrote:

Bullseye provides function coverage + condition/decision coverage only.

Line coverage is not available.

Conceptually the sonar model is completely different than this, in sonar they count lines hits + branch coverage. Branch coverage in sonar sucks because it does not give you which condition are covered only the absolute number.

In bullseye this is available and its really helpull to create new test cases to produce more coverage.

So what we done was to approximate the 2 models, the condition/decision is well mapped to sonar model although we loose the more information. And the function coverage is mapped to line coverage, which is not visually correct I sonar since you will see lines inside functions without any coverage information On Feb 22, 2014 5:35 PM, "Günter Wirth" notifications@github.com wrote:

The other tool-chain we support right now is bullseye. For me, its not ideal either: it doesnt support line coverage and doesnt map good to the Sonar model.

@wenns https://github.com/wenns What exactly is the problem with Bullseye?

BullseyeCoverage Measurement Technique: http://www.bullseye.com/measurementTechnique.html Why do you think line coverage is missing ( http://www.bullseye.com/coverage.html#basic_statement)? Isn't Condition/Decision Coverage the better choice?

Reply to this email directly or view it on GitHubhttps://github.com/wenns/sonar-cxx/issues/79#issuecomment-35805367 .

[guwirth]

Like to add Visual Studio coverage support. See below an example for coverage result calling with myFunc(0, 1):

#include "stdafx.h"

void myFunc(int a, int b)
{
    int x = 0;
    int y = 0;

    if( a ) {
        x = 1;
        y = 1;
    }

    if( b )
    {
        x = 2;
        y = 2;
    }

    if( a &&
        b
        ) {
            x = 3;
            y = 3;
    }
}

Resulting Visual Studio XML coverage file:

<?xml version="1.0" encoding="UTF-8" ?>
<results>
  <modules>
    <module name="unittest1.dll" path="unittest1.dll" id="93AA4138F7AFB24C91DC9614B700B83C07000000" block_coverage="72.73" line_coverage="68.75" blocks_covered="8" blocks_not_covered="3" lines_covered="11" lines_partially_covered="1" lines_not_covered="4">
      <functions>
        <function id="77200" name="myFunc" type_name="" block_coverage="62.50" line_coverage="61.54" blocks_covered="5" blocks_not_covered="3" lines_covered="8" lines_partially_covered="1" lines_not_covered="4">
          <ranges>
            <range source_id="0" covered="yes" start_line="4" start_column="0" end_line="4" end_column="0" />
            <range source_id="0" covered="yes" start_line="5" start_column="0" end_line="5" end_column="0" />
            <range source_id="0" covered="yes" start_line="6" start_column="0" end_line="6" end_column="0" />
            <range source_id="0" covered="yes" start_line="8" start_column="0" end_line="8" end_column="0" />
            <range source_id="0" covered="no" start_line="9" start_column="0" end_line="9" end_column="0" />
            <range source_id="0" covered="no" start_line="10" start_column="0" end_line="10" end_column="0" />
            <range source_id="0" covered="yes" start_line="13" start_column="0" end_line="13" end_column="0" />
            <range source_id="0" covered="yes" start_line="15" start_column="0" end_line="15" end_column="0" />
            <range source_id="0" covered="yes" start_line="16" start_column="0" end_line="16" end_column="0" />
            <range source_id="0" covered="partial" start_line="21" start_column="0" end_line="21" end_column="0" />
            <range source_id="0" covered="no" start_line="22" start_column="0" end_line="22" end_column="0" />
            <range source_id="0" covered="no" start_line="23" start_column="0" end_line="23" end_column="0" />
            <range source_id="0" covered="yes" start_line="25" start_column="0" end_line="25" end_column="0" />
          </ranges>
        </function>
      </functions>
      <source_files>
        <source_file id="0" path="x:\coveragetest\source.cpp" checksum_type="MD5" checksum="8A4EC27BE0FE2BCE77401F836581836D">
        </source_file>
      </source_files>
    </module>
  </modules>
</results>

Adding line coverage is no problem. Information for condition coverage is not available, only block coverage on function level. My idea is to add block coverage as condition coverage for the myFunc line: line 3, 5+3 blocks, 5 blocks covered

Any other ideas?

[wenns]

What a pity... I hoped that we can get both, line and branch coverage. As for other ideas: I think, missing branch coverage is actually better that one thats only partually correct and potentially misleading. Things like "if you use MS coverage, have in mind that branch coverage is actually

, in case of bullseye do not pay attention to , bla" are _very_ annoying, IMO. If the core allows it (not sure about > that), I would just go with a fully present (and hopefully correct) line > coverage and skip branch coverage altogether. 2014-04-24 8:19 GMT+02:00 Günter Wirth notifications@github.com: > Like to add Visual Studio coverage support. See below an example for > coverage result calling with _myFunc(0, 1)_: > > #include "stdafx.h" > void myFunc(int a, int b){ > int x = 0; > int y = 0; > > ``` > if( a ) { > x = 1; > y = 1; > } > > if( b ) > { > x = 2; > y = 2; > } > > if( a && > b > ) { > x = 3; > y = 3; > }} > ``` > > Resulting Visual Studio XML coverage file: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Adding line coverage is no problem. Information for condition coverage is > not available, only block coverage on function level. My idea is to add > block coverage as condition coverage for the myFunc line: > line 3, 5+3 blocks, 5 blocks covered > > Any other ideas? > > — > Reply to this email directly or view it on GitHubhttps://github.com/wenns/sonar-cxx/issues/79#issuecomment-41246121 > .

[guwirth]

What a pity... I hoped that we can get both, line and branch coverage.

You get both. SonarQube is calculating the branch coverage in the file header by accumulating the condition coverage. With the API you can set only setHits for line coverage and setConditions for condition coverage. By setting the condition coverage to the first line of the function SonarQube shows an accurate branch coverage.

Things like "if you use MS coverage, have in mind that branch coverage is actually , in case of bullseye do not pay attention to <this and that, bla" are very annoying, IMO.

Because the tools are measuring different coverage types the result is different. I think it is more important that the native usage of the coverage tools and the results in SonarQube are the same. The coverage tools itself can't be compared with one another.

[guwirth]

Docu: https://github.com/wenns/sonar-cxx/wiki/Get-code-coverage-metrics