Descartes failing to pass test which should get passed, false non green suit detection on spoon.

[Rohitesh-Kumar-Jain]

Characteristics

• Issue Type: [bug, feature, test report] Bug
• Reproducibility: [always, sometimes, random] always
• Severity: [feature, minor, major, crash, block] BUILD FAILURE
• Tool/Service/Component: [name, version] descartes, 1.3.1,
• Execution Environment: [platform, OS, etc] macOS, IntelliJIDEA
• Reporter: [name, mail] Rohitesh Kumar Jain, thekumarjain@gmail.com

Description

Descartes is detecting false non-green suit on Spoon.

After getting an error on running Descartes on all the methods and test, I tried running it using subsets of the test, and pin pointed the subset which is causing the build to fail.

The subset failing the test is : spoon.test.logging

I am getting the following error:

[INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 11:19 min [INFO] Finished at: 2021-04-24T18:36:29+05:30 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.pitest:pitest-maven:1.6.4:mutationCoverage (default-cli) on project spoon-core: Execution default-cli of goal org.pitest:pitest-maven:1.6.4:mutationCoverage failed: 6 tests did not pass without mutation when calculating line coverage. Mutation testing requires a green suite. [ERROR] See http://pitest.org for more details. [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException

Steps to reproduce

Get spoon from here.

Add this plugin to the pom.xml file.

     <plugin>
            <groupId>org.pitest</groupId>
            <artifactId>pitest-maven</artifactId>
            <version>1.6.4</version>
            <configuration>
                <mutationEngine>descartes</mutationEngine>
                <targetClasses>
                    <param>spoon.*</param>
                </targetClasses>
                <targetTests>
                    <param>spoon.test.logging.*</param>
                </targetTests>
            </configuration>
            <dependencies>
                <dependency>
                    <groupId>eu.stamp-project</groupId>
                    <artifactId>descartes</artifactId>
                    <version>1.3.1</version>
                </dependency>
            </dependencies>
     </plugin>

run mvn clean package org.pitest:pitest-maven:mutationCoverage

Other files and URLs

This existing issue is relevant (sorry if I am duplicating the issue, I feel that I have pinpointed the exact subset which is failing the Build that's why raising it again 😅)

[oscarlvp]

Thank you for reporting the issue @Rohitesh-Kumar-Jain however, this seems to be a problem related to PIT and not Descartes. The issue happens when PIT instruments the code and executes the tests to compute code coverage. At this point no mutation has been performed. I have seen similar problems reported in PIT's repository. As a workaround you can useskipFailingTests, a boolean option provided by PIT to ignore the failing test cases. If you do this, then you should also check if ignoring those tests does not introduce false positives.

[Rohitesh-Kumar-Jain]

Hi,

As a workaround you can useskipFailingTests

Thanks this worked, I was able to run Descartes on Spoon after this : )

I think this issue should be closed now?