search

hcoles / pitest

State of the art mutation testing system for the JVM
Apache License 2.0
1.67k stars 357 forks source link

pitest crashed on usage wildcard in targetClasses/targetTests #368

Open romani opened 7 years ago

romani commented 7 years ago

It is NOT a general problem, there is a workaround to define all classes/tests one by one, without wild card. It is reproduced only on certain set of Tests/Classes.

steps to reproduce:

$ git clone git@github.com:checkstyle/checkstyle.git
$ # apply diff from bellow
$ mvn -Ppitest-checks-annotation,no-validations clean verify org.pitest:pitest-maven:mutationCoverage -Dverbose=true

output:

[INFO] --- pitest-maven:1.2.2:mutationCoverage (default-cli) @ checkstyle ---
[INFO] Found plugin : Default csv report plugin
[INFO] Found plugin : Default xml report plugin
[INFO] Found plugin : Default html report plugin
[INFO] Found plugin : Default limit mutations plugin
[INFO] Found plugin : Kotlin junk mutations filter
[INFO] Found plugin : Static initializer code detector plugin
[INFO] Found plugin : Static initializer filter plugin
[INFO] Found plugin : Excluded annotations plugin
[INFO] Found plugin : Inlined finally block filter plugin
[INFO] Found plugin : Try with resources filter
[INFO] Found plugin : Logging calls filter
[INFO] Found plugin : Infinite for loop filter
[INFO] Found plugin : Long running iterator loop filter
[INFO] Found shared classpath plugin : Default mutation engine
[INFO] Adding org.pitest:pitest to SUT classpath
[INFO] Mutating from /home/rivanov/java/github/checkstyle/checkstyle/target/classes
6:04:24 AM PIT >> INFO : ---------------------------------------------------------------------------
6:04:24 AM PIT >> INFO : Enabled (+) and disabled (-) features.
6:04:24 AM PIT >> INFO : -----------------------------------------
6:04:24 AM PIT >> INFO : +FINFINC        Filters mutations to increments that may cause infinite loops
6:04:24 AM PIT >> INFO : +FFBLOCK        Filters mutations in code duplicated by finally block inlining
6:04:24 AM PIT >> INFO : +FSTATI         Filters mutations in static initializers and code called only from them
6:04:24 AM PIT >> INFO : +FTRYWR         Filters mutations in code generated for try with resources statements
6:04:24 AM PIT >> INFO : +FSTATINIT      Filters mutations in static initializers and code called only from them
6:04:24 AM PIT >> INFO : +FLOGCALL       Filters mutations in code that makes calls to logging frameworks
6:04:24 AM PIT >> INFO : +FINFIT         Filters mutations that may cause infinite loops by removing calls to iterator.next
6:04:24 AM PIT >> INFO : +FANN           Filters mutations in classes and methods with matching annotations of class or runtime retention
6:04:24 AM PIT >> INFO : ---------------------------------------------------------------------------
6:04:24 AM PIT >> FINE : Running report with ReportOptions [targetClasses=[^com\.puppycrawl\.tools\.checkstyle\.checks\.annotation\..*$], excludedMethods=[], excludedClasses=[], codePaths=[/home/rivanov/java/github/checkstyle/checkstyle/target/classes], reportDir=/home/rivanov/java/github/checkstyle/checkstyle/target/pit-reports, historyInputLocation=null, historyOutputLocation=null, sourceDirs=[/home/rivanov/java/github/checkstyle/checkstyle/src/main/java, /home/rivanov/java/github/checkstyle/checkstyle/target/generated-sources/antlr, /home/rivanov/java/github/checkstyle/checkstyle/target/generated-sources/annotations, /home/rivanov/java/github/checkstyle/checkstyle/src/test/java, /home/rivanov/java/github/checkstyle/checkstyle/src/test/resources, /home/rivanov/java/github/checkstyle/checkstyle/src/it/java, /home/rivanov/java/github/checkstyle/checkstyle/src/it/resources, /home/rivanov/java/github/checkstyle/checkstyle/target/generated-test-sources/test-annotations], classPathElements=[/home/rivanov/java/github/checkstyle/checkstyle/target/test-classes, /home/rivanov/java/github/checkstyle/checkstyle/target/classes, /home/rivanov/.m2/repository/antlr/antlr/2.7.7/antlr-2.7.7.jar, /home/rivanov/.m2/repository/org/antlr/antlr4-runtime/4.7/antlr4-runtime-4.7.jar, /home/rivanov/.m2/repository/commons-beanutils/commons-beanutils/1.9.3/commons-beanutils-1.9.3.jar, /home/rivanov/.m2/repository/commons-logging/commons-logging/1.2/commons-logging-1.2.jar, /home/rivanov/.m2/repository/commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.jar, /home/rivanov/.m2/repository/commons-cli/commons-cli/1.4/commons-cli-1.4.jar, /home/rivanov/.m2/repository/com/google/guava/guava/22.0/guava-22.0.jar, /home/rivanov/.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar, /home/rivanov/.m2/repository/com/google/errorprone/error_prone_annotations/2.0.18/error_prone_annotations-2.0.18.jar, /home/rivanov/.m2/repository/com/google/j2objc/j2objc-annotations/1.1/j2objc-annotations-1.1.jar, /home/rivanov/.m2/repository/org/codehaus/mojo/animal-sniffer-annotations/1.14/animal-sniffer-annotations-1.14.jar, /home/rivanov/.m2/repository/org/apache/ant/ant/1.10.1/ant-1.10.1.jar, /home/rivanov/.m2/repository/org/apache/ant/ant-launcher/1.10.1/ant-launcher-1.10.1.jar, /home/rivanov/.m2/repository/junit/junit/4.12/junit-4.12.jar, /home/rivanov/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar, /home/rivanov/.m2/repository/com/github/stefanbirkner/system-rules/1.16.1/system-rules-1.16.1.jar, /home/rivanov/.m2/repository/nl/jqno/equalsverifier/equalsverifier/2.3/equalsverifier-2.3.jar, /home/rivanov/.m2/repository/org/powermock/powermock-api-mockito/1.7.0/powermock-api-mockito-1.7.0.jar, /home/rivanov/.m2/repository/org/powermock/powermock-api-mockito-common/1.7.0/powermock-api-mockito-common-1.7.0.jar, /home/rivanov/.m2/repository/org/powermock/powermock-api-support/1.7.0/powermock-api-support-1.7.0.jar, /home/rivanov/.m2/repository/org/mockito/mockito-core/1.10.19/mockito-core-1.10.19.jar, /home/rivanov/.m2/repository/org/objenesis/objenesis/2.1/objenesis-2.1.jar, /home/rivanov/.m2/repository/org/powermock/powermock-module-junit4/1.7.0/powermock-module-junit4-1.7.0.jar, /home/rivanov/.m2/repository/org/powermock/powermock-module-junit4-common/1.7.0/powermock-module-junit4-common-1.7.0.jar, /home/rivanov/.m2/repository/org/powermock/powermock-reflect/1.7.0/powermock-reflect-1.7.0.jar, /home/rivanov/.m2/repository/org/powermock/powermock-core/1.7.0/powermock-core-1.7.0.jar, /home/rivanov/.m2/repository/org/javassist/javassist/3.21.0-GA/javassist-3.21.0-GA.jar, /home/rivanov/.m2/repository/commons-io/commons-io/2.5/commons-io-2.5.jar, /home/rivanov/.m2/repository/org/eclipse/jgit/org.eclipse.jgit/4.8.0.201706111038-r/org.eclipse.jgit-4.8.0.201706111038-r.jar, /home/rivanov/.m2/repository/com/jcraft/jsch/0.1.54/jsch-0.1.54.jar, /home/rivanov/.m2/repository/com/googlecode/javaewah/JavaEWAH/1.1.6/JavaEWAH-1.1.6.jar, /home/rivanov/.m2/repository/org/apache/httpcomponents/httpclient/4.3.6/httpclient-4.3.6.jar, /home/rivanov/.m2/repository/org/apache/httpcomponents/httpcore/4.3.3/httpcore-4.3.3.jar, /home/rivanov/.m2/repository/commons-codec/commons-codec/1.6/commons-codec-1.6.jar, /home/rivanov/.m2/repository/org/slf4j/slf4j-api/1.7.2/slf4j-api-1.7.2.jar, /home/rivanov/.m2/repository/org/slf4j/slf4j-simple/1.7.25/slf4j-simple-1.7.25.jar, /home/rivanov/.m2/repository/org/jacoco/org.jacoco.agent/0.7.9/org.jacoco.agent-0.7.9-runtime.jar, /opt/jvm/jdk1.8.0_131/jre/../lib/tools.jar, /home/rivanov/.m2/repository/org/pitest/pitest/1.2.2/pitest-1.2.2.jar], mutators=[], dependencyAnalysisMaxDistance=-1, mutateStaticInitializers=false, jvmArgs=[], numberOfThreads=4, timeoutFactor=10.0, timeoutConstant=50000, targetTests=[^com\.puppycrawl\.tools\.checkstyle\.checks\.annotation\..*$], loggingClasses=[], maxMutationsPerClass=-1, verbose=true, failWhenNoMutations=true, outputs=[HTML], groupConfig=TestGroupConfig [excludedGroups=[], includedGroups=[]], mutationUnitSize=0, shouldCreateTimestampedReports=true, detectInlinedCode=true, exportLineCoverage=false, mutationThreshold=0, coverageThreshold=0, mutationEngine=gregor, javaExecutable=null, includeLaunchClasspath=true, properties={}, maxSurvivors=0, excludedRunners=[], features=[]]
6:04:24 AM PIT >> FINE : System class path is /opt/maven/apache-maven-3.3.9//boot/plexus-classworlds-2.5.2.jar
6:04:24 AM PIT >> FINE : Maximum available memory is 3534 mb
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 20.429 s
[INFO] Finished at: 2017-07-07T06:04:24-07:00
[INFO] Final Memory: 92M/1558M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.pitest:pitest-maven:1.2.2:mutationCoverage (default-cli) on project checkstyle: Execution default-cli of goal org.pitest:pitest-maven:1.2.2:mutationCoverage failed: 65535
[ERROR] 
[ERROR] Please copy and paste the information and the complete stacktrace below when reporting an issue
[ERROR] VM : Java HotSpot(TM) 64-Bit Server VM
[ERROR] Vendor : Oracle Corporation
[ERROR] Version : 25.131-b11
[ERROR] Uptime : 21327
[ERROR] Input ->
[ERROR] 1 : -Dclassworlds.conf=/opt/maven/apache-maven-3.3.9//bin/m2.conf
[ERROR] 2 : -Dmaven.home=/opt/maven/apache-maven-3.3.9/
[ERROR] 3 : -Dmaven.multiModuleProjectDirectory=/home/rivanov/java/github/checkstyle/checkstyle
[ERROR] BootClassPathSupported : true
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.

workaround was to define all classes/tests one by one, diff is simple:

diff --git a/pom.xml b/pom.xml
index 851010c..ebcba2a 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1576,20 +1576,10 @@
             <version>${pitest.plugin.version}</version>
             <configuration>
               <targetClasses>
-                <param>com.puppycrawl.tools.checkstyle.checks.annotation.AnnotationLocationCheck</param>
-                <param>com.puppycrawl.tools.checkstyle.checks.annotation.AnnotationUseStyleCheck</param>
-                <param>com.puppycrawl.tools.checkstyle.checks.annotation.MissingDeprecatedCheck</param>
-                <param>com.puppycrawl.tools.checkstyle.checks.annotation.MissingOverrideCheck</param>
-                <param>com.puppycrawl.tools.checkstyle.checks.annotation.PackageAnnotationCheck</param>
-                <param>com.puppycrawl.tools.checkstyle.checks.annotation.SuppressWarningsCheck</param>
+                <param>com.puppycrawl.tools.checkstyle.checks.annotation.*</param>
               </targetClasses>
               <targetTests>
-                <param>com.puppycrawl.tools.checkstyle.checks.annotation.AnnotationLocationCheckTest</param>
-                <param>com.puppycrawl.tools.checkstyle.checks.annotation.AnnotationUseStyleCheckTest</param>
-                <param>com.puppycrawl.tools.checkstyle.checks.annotation.MissingDeprecatedCheckTest</param>
-                <param>com.puppycrawl.tools.checkstyle.checks.annotation.MissingOverrideCheckTest</param>
-                <param>com.puppycrawl.tools.checkstyle.checks.annotation.PackageAnnotationCheckTest</param>
-                <param>com.puppycrawl.tools.checkstyle.checks.annotation.SuppressWarningsCheckTest</param>
+                <param>com.puppycrawl.tools.checkstyle.checks.annotation.*</param>
               </targetTests>
               <mutationThreshold>99</mutationThreshold>
               <timeoutFactor>${pitest.plugin.timeout.factor}</timeoutFactor>
rnveach commented 5 years ago

This specific issue wasn't related to the wildcard directly but the wildcard was bringing in an input file that pitest couldn't handle. See https://github.com/hcoles/pitest/issues/302#issuecomment-444738552 on the exact file and the full stack trace we received.