Pitest info do not show up in Sonarqube

[mosser]

Hi,

I use Pitest on a bunch of legacy Java project. As I cannot change their POM file, I configure all the properties using CLI, but it should not matters. The point is to obtain in an automated way associated reports in Sonar, more specifically for tests.

First, I compile to code using the JaCoCo agent to compute coverage metrics

mvn org.jacoco:jacoco-maven-plugin:prepare-agent clean package

Then, I asks Pitest to generate an XML report for mutation testing analysis:

mvn -DoutputFormats=XML -Dthreads=4 org.pitest:pitest-maven:mutationCoverage

It creates a file in target/pit-reports/YYYYMMDD____/mutations.xml. If invoked with the HML output format, it generates the expected report (with the killed/survived ratio and so on).

Then, I use the sonar-pitest plugin to upload it to sonarqube (Version 7.0 (build 36138)), started using Docker (image: sonarqube:alpine, latest version). I installed the plugin directly from the marketplace inside sonar.

mvn -Dsonar.host.url=http://localhost:9000 sonar:sonar

The Sonarqube adminsitration panel, w.r.t Pitest, use the default value for sonar.pitest.reportsDirectory (target/pit-reports) and sonar.pitest.mode(reuseReport).

Unfortunately, when running the last command, there is no mention of Pitest in the execution log (the [INFO] level displays information about Javasquid, surefire, JaCoCo, bu nothing about pitest).

When going to the project Measures tab, I did not find any information about mutation testing in any of the available measurements.

Am I missing something, or is it a bug with sonarqube 7.

Thanks.

[bradflood]

@mosser thank you for the feedback. A few quick comments... First, I haven't tested the plugin with SQ 7 yet, so that may be an issue. Second, There is a bug when creating measurements that I haven't fixed yet, so you won't see Measurements even on SQ 6 LTS. Having said that, with the configuration as you indicated, you should see "issues" in SQ since you are seeing the surviving mutations in the HTML report.
The fact that you are not seeing anything in the SQ logs probably also points to a problem with SQ 7 compatibility. I'm pretty swamped right now. Will try to give it a look soon though.

[marijnvermeulenadobe]

Hi @bradflood and @mosser,

Since I've been pursuing this topic, I've tested the functionalities of the plugin in almost every latest version of SonarQube. In short:

• Measures are displayed until SQ 6.6 + plugin v0.8 ; disappeared in v6.7 LTS + plugin v0.9
• Issues are reported as 'code smells' until SQ 6.6; as from SQ 6.7 they are reported as bugs

I'm personally interested if this change of behaviour is caused by changes in SonarQube or version 0.9 of the sonar-pitest plugin? Is there a possibility that I can help with that?

Cheers, Marijn

[bradflood]

@marijnvermeulen-adobe Yes, I would welcome the help. Much appreciated.

Regarding measures, the capability was temporarily removed from plugin 0.9 because it caused an analysis failure. This would be a good place to start.
You could remove the comments starting here and track the resulting analysis error back.

Regarding issues, I did not intentionally change their categorization. If you could provide a specific example, perhaps I could dig further to find what changed.

[rhdietrich]

We run sonarqube Version 7.1 (build 11001). After downgrading pitest plugin from 0.9 to 0.8 mutation analysis shows up in SQ again (with results)

[bradflood]

@rhdietrich @marijnvermeulen-adobe @mosser I have reintroduced measures Note that I also updated to pitest 1.4.0. A few things to note:

• pitest 1.4.0 requires Java 8
• a few of the pitest mutator names have changed, and I have updated sonar-pitest to recognize these new names. If you are using an older version of pitest, this may result in some mutations being unrecognized in the report.

@VinodAnandan would you be able to release 0.9.1 from master?

[VinodAnandan]

Hi @bradflood ,

Sorry for the late reply. Sure, I will do that in a couple of days, hope that's fine? If it's urgent please let me know.

Thank you,

Vinod

[artsok]

Hi, @VinodAnandan

Maybe useful: SonarQube 6.5 + sonar-pitest-plugin 0.8 = work SonarQube 6.7.4 + sonar-pitest-plugin 0.8 = work SonarQube 6.7.4 + sonar-pitest-plugin 0.9 = didn't work

[marijnvermeulenadobe]

Hi, I've built version 0.9.1-SNAPSHOT and installed it manually; it works as a charm so good job there!! Can't wait until it it generally available from the SonarQube Plugins marketplace.

Cheers,

[artsok]

@marijnvermeulen-adobe in what version of SQ?

[bradflood]

@VinodAnandan could you please release 0.9.1?

@marijnvermeulen-adobe - we are going to stop working on this plugin at some point, and allow the mutation-analysis-plugin to be the preferred tool for processing pitest output for SonarQube.

There are a few gaps (kotlin support being the primary one), so we'll keep this going until such time that the user community is fully behind the mutation-analysis-plugin.

cc: @gmuecke, the primary developer of the mutation-analysis-plugin.

[costalfy]

Hi all, I build the master branche (1.0-SNAPSHOT) and i deploy the plugin on my sonar version 7.1 (build 11001). I have also installed Mutation Analysis plugin v1.3.

Now, i have 2 sections in the "Measures" tab but without any result after my sonar analysis

For your information, i build my projet with mvn clean install and my ${projet.parent.build.directory}/pit-reports contains the XML with the PIT test results

Could you help me to configure sonar to parse the XML PIT test results?

Best Regards

[gmuecke]

Did you enable any mutation related rule in your active quality profile? see also https://github.com/devcon5io/mutation-analysis-plugin/wiki/Installation#quality-profile

[costalfy]

Yes, all mutation-operator Tags are activated and nothing happen :-(

[gmuecke]

Did you check if the quality profile with the activated pitest / mutation analysis rule (looks like its named "copy of Sonarqube way" or similar) is either set as default or explicitly assigned to your analysis project?

If that is the case and still no result show up, could you run the analysis in debug mode (mvn -X) and provide the log for the mutation analysis somehow?

Btw. if you switch to the mutation-analysis-plugin, you don't need the pitest plugin anymore (with version 1.3 there is now also the kotlin support, so the mutation-analysis-plugin's features are now a complete superset of the pitest plugin. Using both plugins won't cause conflicts, only redundant measures and issues.

[costalfy]

Hi @gmuecke this is the maven debug log file of my Jenkins CI debug-sonar-pitest.zip My project have been set with the "copy of Sonarqube way" quality profile in Sonar 7.1 Nothing in the Mutation Analysis

[gmuecke]

From the logfile I read, that you configured to have the pitest-reports written to the parent 's build directory (line 16904), but only the backend module contains java code & tests & actual mutations.

So the problem that both plugins have here, that they do not find the pit report for backend because its in parent, which in return isn't analyzed, because parent has no Java file and analysis is skipped.

Both plugins have the option to specify a report folder, though I'm not sure it works with ../ (parent folder), but it might be worth a try.

Otherwise I strongly recommend to leave the mutation report in the module where the mutation took place (leave it the default folder).

In addition to this, I opened a new issue https://github.com/devcon5io/mutation-analysis-plugin/issues/23 to explore the feasibility of extracting the report directory from the maven configuration. But no promises made that this will actually been implemented.

[costalfy]

Yes, it works this is my parent pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
  ...
    <build>
        <plugins>
            ...
            <plugin>
                <groupId>org.pitest</groupId>
                <artifactId>pitest-maven</artifactId>
                <version>${pitest-maven.version}</version>
                <configuration>
                    <targetClasses>be.*</targetClasses>
                    <excludedMethods>
                        <excludedMethod>equals</excludedMethod>
                        <excludedMethod>hashCode</excludedMethod>
                        <excludedMethod>toString</excludedMethod>
                        <excludedMethod>builder</excludedMethod>
                        <excludedMethod>build</excludedMethod>
                        <excludedMethod>canEqual</excludedMethod>
                    </excludedMethods>
                    <excludedTestClasses>
                        <excludedTestClass>be.myproject.ApiDocumentation</excludedTestClass>
                    </excludedTestClasses>
                    <timeoutConstant>8000</timeoutConstant>
                    <outputFormats>XML,HTML</outputFormats>
                    <threads>8</threads>
                    <withHistory>true</withHistory>
                    <!--reportsDirectory>${project.parent.build.directory}/pit-reports</reportsDirectory-->
                </configuration>
                <executions>
                    <execution>
                        <id>pitest</id>
                        <goals>
                            <goal>mutationCoverage</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
           ...
        </plugins>
    </build>
    <reporting>
        <plugins>
            <plugin>
                <groupId>org.pitest</groupId>
                <artifactId>pitest-maven</artifactId>
                <version>${pitest-maven.version}</version>
                <reportSets>
                    <reportSet>
                        <reports>
                            <report>report</report>
                        </reports>
                    </reportSet>
                </reportSets>
            </plugin>
        </plugins>
    </reporting>
    <properties>
        ...
        <!-- sonar.pitest.reportsDirector>${project.build.directory}/pit-reports</sonar.pitest.reportsDirector -->
        ...
    </properties>
...
</project>

Thank you very much for your help

[gmuecke]

Cool :) Glad I could help.

Btw. try out the experimental features for some more metrics, maybe they provide some additional insights to you

[mgonzcast]

Hi! Not sure If this issue is the best place or maybe I should open a separate issue by itself.

I have several Projects under a folder. Since I am running sonarqube manually and not automatically just for scanning a project from time to time I use something like this (changed info for anonimity):

Before I run Jacoco manually and Maven with:

mvn org.pitest:pitest-maven:mutationCoverage -DtimeoutConstant=9000 -DoutputFormats=XML

cd \projects\

"C.\route_to_sonar\sonar-scanner-4.6.0.2311-windows\bin\sonar-scanner.bat" -D"sonar.projectKey=MyProject" -D"sonar.sources=." -D"sonar.host.url=http://localhost:9000" -D"sonar.login=fffffffffffffffffffffffffffffffffffffffffffffffffff" -D"sonar.dependencyCheck.htmlReportPath=.\MyProject\dependency-check-report.html" -D"sonar.dependencyCheck.xmlReportPath=.\MyProject\dependency-check-report.xml" -D"sonar.coverage.jacoco.xmlReportPaths=.\MyProject\jacoco.xml" -D"dc5.mutationAnalysis.pitest.sensor.reports.directory=.\MyProject\target\pit-reports\202110180914"

Why do I have to override dc5.mutationAnalysis.pitest.sensor.reports.directory with the exact number of report? Couldn´t I able to use .\MyProject\target\pit-reports\ instead?

Shouldn´t it take the latest? Or am I assuming something wrong?