Add Unit Test Code Coverage as a build step for the `common/artifact-manager`

[Michaelpalacce]

Description

We want to add a quality gate for code coverage for the build process of the common/artifact-manager. Increasing the code coverage would be great for future refactoring and overall code quality.

We need to investigate tools that can do this for mvn/java.

[sugeeshC]

JaCoCo is widely used code coverage tool that is compatible with Java 17.

It appears that JaCoCo is already inclueded in the artifact-manager module of the project, as specified in the common/artifact-manager/pom.xml. Currently, the configuration generates an XML report during the build process, but it seems that this report is not being utilized to enforce any coverage requirements.

To enhance the value of these reports, we could configure JaCoCo to enforce minimum coverage thresholds and fail the build if these thresholds are not met. This will ensure that the codebase maintains a certain level of test coverage. We can exclude or include the packages or classes as we want, as shown in follows.

Also generating the HTML report would be useful to identify the state of each class. Generated reports can be found on '\target\site\jacoco\' folder.

Suggested pox.xml change (included and excluded packages should change accordingly)

<execution>
    <id>prepare-agent</id>
    <goals>
        <goal>prepare-agent</goal>
    </goals>
</execution>
<execution>
    <id>report</id>
    <goals>
        <goal>report</goal>
    </goals>
    <configuration>
        <formats>
            <format>XML</format>
            <format>HTML</format>
        </formats>
    </configuration>
</execution>
<execution>
    <id>jacoco-check</id>
    <goals>
        <goal>check</goal>
    </goals>
    <configuration>
        <rules>
            <rule>
                <element>PACKAGE</element>
                <limits>
                    <limit>
                        <counter>LINE</counter>
                        <value>COVEREDRATIO</value>
                        <minimum>0.6</minimum>
                    </limit>
                </limits>
                <includes>
                    <include>com.vmware.pscoe.iac.artifact.store.vrang</include>
                </includes>
                <excludes>
                    <exclude>com.vmware.pscoe.iac.artifact.store.vrang.VrangContentSourceUtils</exclude>
                    <exclude>com.vmware.pscoe.iac.artifact.store.vrang.VraNgContentSourceStore</exclude>
                </excludes>
            </rule>
        </rules>
    </configuration>
</execution>

[sugeeshC]

Openclover is another widely used code coverage tool that is compatible with Java 17. But the support for Java 17 still in experimental stage. Updates are less frequent compared to JaCoCo. Have maven support, but configurations more complex than JaCoCo. Reports uses more newer UI than JaCoCo, but details are mostly the same. Can't see a reason for choose over JaCoCo.

[sugeeshC]

Futher more about JaCoCo

As mentioned before, JaCoCo report can be generated in the 'mvn verify' phase in the maven life cycle.(XML,CSV,HTML[Readable]) This generated XML report can be used to integrate with Github Actions to visualize the coverage metrics directly on the Github pull requests. This integration provides an easy-to-read UI elements on the Github PR and can show followings. 1.) overall coverage 2.) coverage on the changed files 3.) coverage percentage for each changed file

You can see the my testing PR in forked repo here.

Generated report from 'mvn verify' contains coverage data for all the classes. Currently exploring on possibility to get only the changed files coverage percentages in locally.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[VenelinBakalov]

@sugeeshC thank you for all the detailed research! this looks great. I will shortly cover this on a roadmap meeting but I do not see anything against accepting this issue and using your suggested approach with JaCoCo for this. It is also great that we can see the results directly in a PR as shown in your test branch. By any chance did you manage to find a configuration to get only the changed files coverage percentages? @Michaelpalacce if we manage to accept and merge this, I guess the next step would be to open a new issue for raising the threshold of the already existing code.

[sugeeshC]

@VenelinBakalov Yes I had looked into it. To show the changed files code coverage in the PR, we have two options.

1. Use of 'madrapps/jacoco-reports@v1.6.1' action ( this action isn't a github verified action ), You can see this action integrated PR in here.

   With it, we can change the code-coverage threshold with following configurations

   • min-coverage-overall: 40
   • min-coverage-changed-files: 60

2. Write a custom code to identify change files and check there code coverage from Jacoco XML report and print the necessary statistics. on the PR. And integrate this custom code with a workflow for triggering. ( I implemented a custom code in here, and you can see it is integrated with this workflow and the result can be seen at in this PR.)

   If we are going with this, would need to modify the custom written code a bit to failed the workflow run, based on the provided min-coverage configuration value.

But if we go with this direction, the problem is in locally identifying the change files coverage and the threshold. When it comes to changed files, we need to define (target branch and source branch to compare). We can run the 2.) method custom code locally and get the output threshold values, but would need to provide (target branch) as a input to it. Due to this problem, had a chat with @Michaelpalacce and checked on the more simpler way of doing this without thinking about the 'changed files'.

More simpler way is as initially mentioned, adding the threshold to maven, covering all files in artifact-manager. So if developer build the project locally then he face an error if threshold doesn't satisfied. He can access the HTML generate report since it would be generated locally. (\target\site\jacoco) Then it comes to the problem, if somebody doesn't build the project locally, but change some files on artifact-manager and push to the remote repo directly. Then the build will be automatically failed in the pipeline (mvn verify will fail) mentioning the code coverage threshold doesn't satisfied. For that user, I am looking for a way to make the generated HTML coverage report accessible. (one way is saving it as workflow run -> Artifacts, so in the workflow run the build will fail due to the code-coverage threshold and user can download the artifact and check the each file coverage)

[VenelinBakalov]

Maybe an alternative would be to just kick start with very low percentage and try to slowly rise it over time

[VenelinBakalov]

As agreed on Roadmap meeting on 12.09.2024 the current approach would be to enable JaCoCo using the current coverage is minimum threshold and create sequential tasks for raising the threshold by certain percentage.

[sugeeshC]

Jacoco Maven - Configuration parameters

1.) Rule Element - Specifies the target for the rule.

• BUNDLE - Target the entire code bundle or package
• PACKAGE - applies to a specific package
• CLASS - Limits are applied to a class
• SOURCEFILE - Targets the source file
• METHOD - Specific to methods within the code

2.) Limit Counter - Defines the type of coverage to track.

• INSTRUCTION - Counts instructions in the code
• LINE - Tracks individual lines of code
• BRANCH - Monitors branches
• COMPLEXITY - Calculates the cyclomatic complexity
• METHOD - Monitors the number of methods
• CLASS - Tracks the number if classes

3.) Limit Value - Defines the threshold for the metric being measured

• TOTALCOUNT - Total number of elements

• COVEREDCOUNT - Number of elements covered by tests

• MISSEDCOUNT - Number of elements not covered

• COVEREDRATIO - Ratio of covered elements to the total

• MISSEDRATIO - Ratio of missed element to the total

• With [Rule Element - BUNDLE][Limit Counter - INSTRUCTION][Limit Value - COVEREDRATIO] configurations, current project artifact-manager code, maximum threshold should be 0.25.

[VenelinBakalov]

Follow up issue: https://github.com/vmware/build-tools-for-vmware-aria/issues/447