Number "lines to cover" Clover vs Jacoco

[mgonzcast]

Our team has switched from Jacoco to Clover to ingest coverage reports into Sonarqube 8.9.2 Community Edition.

I have performed scans over two different projects with exactly the same code. One project I am using Clover and the project I am using Jacoco.

I have read in some forums that there might be a discrepancy on how the stastistics are calculated but I find weird that the overall "Lines to cover" under Jacoco are around 1k and under Clover are around 250 lines.

Checking the Measures tab under Sonarqube and running through the different packages I find that same java package reports 292 lines to cover in Jacoco based project while it reports only 62 lines to cover under the Clover project.

If I click on the top right "hamburguer" menu the "File measures" option. the number of lines, lines of code and comment lines are exactly the same but on the Coverage column is where the discrepancy shows.

Any idea? @Tony-Proum ?

[Tony-Proum]

Hi, I would try to use the clover site (e.g: mvn ~clover:site~ clover:clover) to see if the difference is also present on the clover report. The plugin here is just used to upload the clover report to sonarqube, the calculation have to be made before with clover itself. Could you check if the report is different when generated locally ?

I know that with maven, it exists some commands to aggregate reports when the project contains several modules etc, maybe does this issue related to the configuration of the clover reporting ?

(Sorry for the late response, I just receive the email notification 😮)

[mgonzcast]

No worries, I have just added your name to the conversation so the notification is triggered.

The maven reports are generated like this:

mvn clean clover:setup verify clover:aggregate clover:clover

Should I remove the aggregate flag? I can´t find in the html report for the particular .java file I was checking (randomly picked) and I can´t see where those "Lines to cover" are shown.

[Tony-Proum]

I think that this may be the point : if the file seems to be randomly picked by clover, it may not appear in the report uploaded in sonarqube. The repo here is responsible of the last point, but as far as I can remember, the configuration may be a bit tricky when the project directory tree is not conventional. Maybe the project you are instrumenting is needing more configuration ? For specific questions on clover, you may have more competent help here : https://github.com/openclover/clover

Also, I can try to help debugging even if there is a long time since I used clover. Is your project open source ? or maybe is your code publicly available ?

[mgonzcast]

@Tony-Proum A few things:

• the project is neither open source nor publicly available.
• When I say randomly picked, I mean I just chose myself randomly to compare with Jacoco any java file. Apparently both Jacoco and Clover and scanning the coverage of all java files perfectly.
• The question here is for a given java file, Jacoco and Clover are reporting different numbers, I am copying here:

I attach screenshots of what I mean. Is the right side where it says Coverage where number differs, uncovered lines and lines to cover

Clover: https://imgur.com/a/o6hnKDN

Jacoco: https://imgur.com/a/MAkNj68

Also I can´t find in the pdf or the html clover report any reference of those numbers

[Tony-Proum]

the site should be generated in your target/clover/site directory (if you're using maven) also, I think that you should open an issue or talk with the clover team directly https://github.com/openclover/clover AFAIK, the percentage calculated and uploaded by the plugin is extracted from the xml files generated by this project itself : https://github.com/openclover/clover

[mgonzcast]

From the sonarqube forum, they say it means "Executable Lines"

[mgonzcast]

the site should be generated in your target/clover/site directory (if you're using maven) also, I think that you should open an issue or talk with the clover team directly https://github.com/openclover/clover AFAIK, the percentage calculated and uploaded by the plugin is extracted from the xml files generated by this project itself : https://github.com/openclover/clover

Not sure if you have read my last answer @Tony-Proum

[Tony-Proum]

I just seen your last comment, could you try to generate the clover site in order to see what's in here ? In my test cases the generated site and the coverage uploaded by the plugin seems equivalent so I cannot reproduce your issue.

In order to generate the site you can use this sample configuration :

 <build>
    <pluginManagement>
        ... ( Your plugins here ) ...
        <plugin>
            <groupId>org.openclover</groupId>
            <artifactId>clover-maven-plugin</artifactId>
            <version>4.3.1</version>
            <configuration>
              <generateHtml>true</generateHtml>
              <generateXml>true</generateXml>
            </configuration>
        </plugin>
      </plugins>
    </pluginManagement>
  </build>```

then run 
´´´
./mvnw clean clover:setup test clover:aggregate clover:clover  
´´´
then, in the `target/site/clover` directory you should find the generated html site

I would like to be sure that the report generated is not errored 

[mgonzcast]

Thanks @Tony-Proum ! you pointed me to the right direction as you made me to review the whole thing.

I was using maven but my colleagues added this in the pom.xml:

<reportDescriptor>clover-report.xml</reportDescriptor>

I didn´t notice that part until today. Under the clover-report.xml you could find:

`

    </clover-report>`

HTML and PDF reports were also added.

So now, I have commented out the clover-report.xml file and ingested the target\site\clover\clover.xml file. The reports generated before with the clover-report were an aggregate and they had many missing parts.

Now the numbers are pretty similar:

Uncovered Lines 55 Lines to Cover 260 Uncovered Conditions 26 Conditions to Cover 78

while in Jacoco were:

Uncovered Lines 67 Lines to Cover 292 Uncovered Conditions 28 Conditions to Cover 83

Before with Jacoco I was using also doing mutationt tests with:

mvn clean install org.pitest:pitest-maven:mutationCoverage -DtimeoutConstant=10000 -DoutputFormats=XML

How do I combine clover and mutation tests now? Now with clover I am getting a bunch of bug reports like:

Alive Mutant: A method call has been removed without being detected by a test. Mutation: removed call to com_atlassian_clover/CoverageRecorder::inc

which seem related to clover

I am using now this:

mvn clean clover:setup verify clover:aggregate clover:clover org.pitest:pitest-maven:mutationCoverage -DtimeoutConstant=10000 -DoutputFormats=XML

[Tony-Proum]

Oh happy to know that I may have helped 😁

I've done a search, and it seems like openclover does not support mutation coverage : https://openclover.org/doc/manual/latest/general--comparison-of-code-coverage-tools.html (so does jacoco) maybe should you exclude the mutation testing thing from the mvn clover command (exclude from the verify phase or use a more specific phase)

Or if you want to had this coverage too, maybe can you open a feature request here : https://github.com/openclover/clover

[mgonzcast]

Oh happy to know that I may have helped 😁

I've done a search, and it seems like openclover does not support mutation coverage : https://openclover.org/doc/manual/latest/general--comparison-of-code-coverage-tools.html (so does jacoco) maybe should you exclude the mutation testing thing from the mvn clover command (exclude from the verify phase or use a more specific phase)

Or if you want to had this coverage too, maybe can you open a feature request here : https://github.com/openclover/clover

Thanks for pointing that out. How do I exclude the mutation thing? Sorry, I am pretty new to all this. Thanks for being so helpful

[Tony-Proum]

You have to run the test phase without running the mutation tests.

in this command: mvn clean clover:setup **verify** clover:aggregate clover:clover the verify phase may run the mutation tests. So clover will try to find the associated coverage. One way to avoid those tests to be run is to use the test phase instead of the verify one (the test phase is supposed to run only the unit tests).

mvn clean clover:setup **test** clover:aggregate clover:clover

Warning : running it this way will not check the coverage of the integration tests.

OR

you can try to configure you plugin in a different way in order to be able to run all your tests but the mutation ones using a specific maven command. Then use this commande instead of the verify one :

mvn clean clover:setup **your phase or goals here** clover:aggregate clover:clover

you can check this blog post as it explain the configuration and debugging part of the phase / goal configuration : https://www.baeldung.com/maven-goals-phases

[mgonzcast]

Thanks @Tony-Proum !

If I run

mvn clean clover:setup test clover:aggregate clover:clover org.pitest:pitest-maven:mutationCoverage -DtimeoutConstant=10000 -DoutputFormats=XML

the sonar project appears empty. Maybe am I missing something?

Thanks again!

[Tony-Proum]

@mgonzcast I think that it miss the sonar:sonar part of the maven command (this part is responsible of uploading the sonar/clover report to sonarQube)

[mgonzcast]

Sorry, I use the sonar scanner instead of calling it from maven. I know it´s strange but I run the sonar scanner manually, this is not part of an automated thing but more like a Code audit that I run manually. Now It seems it is ingesting fine the information.

This is exactly the steps that I do:

cd .\myProject

mvn clean install org.pitest:pitest-maven:mutationCoverage -DtimeoutConstant=10000 -DoutputFormats=XML

dependency-check\bin\dependency-check.bat" --out . -f="XML" -f="HTML" --scan .

cd ..

sonar-scanner-4.6.0.2311-windows\bin\sonar-scanner.bat" -D"sonar.projectKey=Gestor2CNegro-clover" -D"sonar.sources=." -D"sonar.host.url=http://localhost:9000" -D"sonar.login=111111111111111111111111111111" -D"sonar.dependencyCheck.htmlReportPath=.\Gestor2CNegro\dependency-check-report.html" -D"sonar.dependencyCheck.xmlReportPath=.\myProject\dependency-check-report.xml" -D"sonar.clover.reportPath=.\myProject\target\site\clover\clover.xml" -D"dc5.mutationAnalysis.pitest.sensor.reports.directory=.\myProject\target\pit-reports\202202011234"

[Tony-Proum]

😄 oh ok, nothing wrong to run it this way, happy to see that it now works as expected !