search

jenkinsci / code-coverage-api-plugin

Deprecated Jenkins Code Coverage Plugin
https://plugins.jenkins.io/code-coverage-api/
MIT License
111 stars 77 forks source link

Add additional code coverage types of VectorCAST reports #514

Open TimSVector opened 1 year ago

TimSVector commented 1 year ago

What feature do you want to see added?

I'd like to see additional code coverage types added:

  1. Modified Condition/Decision Coverage (MC/DC)
  2. Function Coverage
  3. Function Call Coverage

These could be added to any of the support formats or a new format supported

Upstream changes

No response

uhafner commented 1 year ago

The Jenkins plugin just visualizes the results of existing tools. It does not itself record the coverage data, this needs to be done by the coverage tools in the build pipeline. Do you have a tool that reports that type of coverage?

Jenkins already shows the method coverage. What exactly do you want to see in a Function Coverage?

TimSVector commented 1 year ago

Great suggestion - method code will actually work for Function Coverage,

We have VectorCAST which we have scripts to produce coverage for Statement, Branch, MC/DC, Function and Function Call coverage in modified Emma format. These code coverage formats are needed for ISO26262 and DO-178 programs (C/C++/Ada)

This currently implemented in the VectorCAST Coverage Plugin, but moving to a standard plugin instead is of interest.

TimSVector commented 1 year ago

Hi Ullrich,

Sorry for the delay - I thought I had responded previously.

My company's tool, VectorCAST, is a test tool used for testing C/C++/Ada code. We produce the following coverage metrics based on standard like DO-178C, ISO 26262, etc.

The specific metrics we provide for these standards:

Statement - Instruction/Line (not sure the difference) Branch - Branch Condition Coverage - coverage for each of the subconditions in a branch MC/DC Equivalence Pairs - https://en.wikipedia.org/wiki/Modified_condition/decision_coverage Function Coverage - Was a function execute Function Call Coverage - Was a function call in a function covered

These aren't special VectorCAST coverage types, but derived from the standards we support (see below):

@.***

We have developed the VectorCAST Code Coverage (VCC) Plugin (https://plugins.jenkins.io/vectorcast-coverage/) based off the old EMMA coverage plugin and modified it to serve our needs but your plugin has many new features that I don't want to reimplement.

I'd be happy to make a fork off of your repo and use that as VCC v2, but I'd rather have it available to anyone using C/C++/Ada.

Let me know.

Tim Schneider

From: Ullrich Hafner @.> Sent: Wednesday, November 9, 2022 6:35 AM To: jenkinsci/code-coverage-api-plugin @.> Cc: Schneider, Tim @.>; Author @.> Subject: Re: [jenkinsci/code-coverage-api-plugin] Add additional code coverage types (Issue #514)

The Jenkins plugin just visualizes the results of existing tools. It does not itself record the coverage data, this needs to be done by the coverage tools in the build pipeline. Do you have a tool that reports that type of coverage?

Jenkins already shows the method coverage. What exactly do you want to see in a Function Coverage?

- Reply to this email directly, view it on GitHubhttps://github.com/jenkinsci/code-coverage-api-plugin/issues/514#issuecomment-1308617687, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AENYLQYTQMOZ3DRN7U33QNTWHOD6PANCNFSM6AAAAAAR22SIM4. You are receiving this because you authored the thread.Message ID: @.***>

uhafner commented 1 year ago

From an architectural standpoint, it would make sense to create a new parser for your reports in https://github.com/jenkinsci/coverage-model. Then we simply add a new enum in the code coverage plugin so that users can select the new type. Then everything else should work out of the box. Maybe we need to adapt the model a little bit since CC, MC/DC is not used up to now.

TimSVector commented 8 months ago

Hi Ullrich - Happy New Year.

Its been in a back burner for a while, but over our holiday shutdown, I was able to investigate this a bit more. I found that expanded Cobertura format gave me the coverage I was interested in. I was able to copy and modify the coverage-model\src\main\java\edu\hm\hafner\coverage\parser\CoberturaParser.java to VectorCASTParser.java and add in the additional coverages to the Cobertura format. I have two questions:

  1. Would you rather me expand the existing CoberturaParser.java to include the additional coverages or keep the formats separate parser files?
  2. How do you build/integrate the coverage-model (coverage-model-0.37.0-SNAPSHOT.jar) into the coverage-plugin for testing? I looked and couldn't find documentation on building the coverage-plugin with an experimental coverage-model.jar

Additions for VectorCAST coverage would be attributes on , like this:

    <class name="database" filename="CurrentRelease/database/src/database.c" line-rate="1.0" mcdcpair-coverage="0.5" functioncall-coverage="1.0" complexity="5">
       <line number="57" hits="1" branch="true" condition-coverage="100.0% (2/2)" mcdcpair-coverage="100.0% (1/1)">

I will implement your idea for method for function coverage.

Thanks for the assistance.

Tim

From: Schneider, Tim @.> Sent: Thursday, July 20, 2023 10:37 AM To: jenkinsci/code-coverage-api-plugin @.>; jenkinsci/code-coverage-api-plugin @.>; Opitz, Derek @.> Cc: Author @.***> Subject: RE: [jenkinsci/code-coverage-api-plugin] Add additional code coverage types (Issue #514)

Hi Ullrich,

Sorry for the delay - I thought I had responded previously.

My company's tool, VectorCAST, is a test tool used for testing C/C++/Ada code. We produce the following coverage metrics based on standard like DO-178C, ISO 26262, etc.

The specific metrics we provide for these standards:

Statement - Instruction/Line (not sure the difference) Branch - Branch Condition Coverage - coverage for each of the subconditions in a branch MC/DC Equivalence Pairs - https://en.wikipedia.org/wiki/Modified_condition/decision_coverage Function Coverage - Was a function execute Function Call Coverage - Was a function call in a function covered

These aren't special VectorCAST coverage types, but derived from the standards we support (see below):

@.***

We have developed the VectorCAST Code Coverage (VCC) Plugin (https://plugins.jenkins.io/vectorcast-coverage/) based off the old EMMA coverage plugin and modified it to serve our needs but your plugin has many new features that I don't want to reimplement.

I'd be happy to make a fork off of your repo and use that as VCC v2, but I'd rather have it available to anyone using C/C++/Ada.

Let me know.

Tim Schneider

From: Ullrich Hafner @.**@.>> Sent: Wednesday, November 9, 2022 6:35 AM To: jenkinsci/code-coverage-api-plugin @.**@.>> Cc: Schneider, Tim @.**@.>>; Author @.**@.>> Subject: Re: [jenkinsci/code-coverage-api-plugin] Add additional code coverage types (Issue #514)

The Jenkins plugin just visualizes the results of existing tools. It does not itself record the coverage data, this needs to be done by the coverage tools in the build pipeline. Do you have a tool that reports that type of coverage?

Jenkins already shows the method coverage. What exactly do you want to see in a Function Coverage?

- Reply to this email directly, view it on GitHubhttps://github.com/jenkinsci/code-coverage-api-plugin/issues/514#issuecomment-1308617687, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AENYLQYTQMOZ3DRN7U33QNTWHOD6PANCNFSM6AAAAAAR22SIM4. You are receiving this because you authored the thread.Message ID: @.**@.>>

uhafner commented 7 months ago
  1. Would you rather me expand the existing CoberturaParser.java to include the additional coverages or keep the formats separate parser files?

I think a new parser would make sense. If somehow possible it would make sense to add some extension methods in the existing Cobertura parser so that you can extend the existing Cobertura parser without too much duplication. Do you think this is possible? This will simplify the maintenance.

  1. How do you build/integrate the coverage-model (coverage-model-0.37.0-SNAPSHOT.jar) into the coverage-plugin for testing? I looked and couldn't find documentation on building the coverage-plugin with an experimental coverage-model.jar

You need to call mvn install for the coverage-model locally and then refer to this version in https://github.com/jenkinsci/coverage-plugin/blob/8dd9436cfe582cd4dc244f367a2ed3f7099191b9/plugin/pom.xml#L28

Once you created a PR you will get an official snapshot deployed to our CD repository.