hcoles / pitest

State of the art mutation testing system for the JVM
Apache License 2.0
1.7k stars 358 forks source link

PIT fails to execute any tests; rooted in ClassNotFound: org.junit.runner.manipulation.Filter #970

Open kphannan opened 2 years ago

kphannan commented 2 years ago

Included are versions of Java, Gradle and the verbose output from PiTest.

This was a clean build with JUnit 5.8.2 based unit tests.

If needed I can put the entire project into GitHub for access to the entire codebase. It is just prototype code being used to demonstrate some design and coding, nothing proprietary. the project is relatively small 33 total source files including unit tests.

-------- Begin console output ------------

➜ basics git:(feature/improve-strategy-tests) ✗ gradle -v executing gradlew instead of gradle


Gradle 7.3.3

Build time: 2021-12-22 12:37:54 UTC Revision: 6f556c80f945dc54b50e0be633da6c62dbe8dc71

Kotlin: 1.5.31 Groovy: 3.0.9 Ant: Apache Ant(TM) version 1.10.11 compiled on July 10 2021 JVM: 17 (Eclipse Adoptium 17+35) OS: Mac OS X 11.1 x86_64

➜ basics git:(feature/improve-strategy-tests) ✗ java -version openjdk version "17" 2021-09-14 OpenJDK Runtime Environment Temurin-17+35 (build 17+35) OpenJDK 64-Bit Server VM Temurin-17+35 (build 17+35, mixed mode, sharing) ➜ basics git:(feature/improve-strategy-tests) ✗ gradle check jacocoTestReport pitest executing gradlew instead of gradle

Task :lib:checkstyleMain [ant:checkstyle] [WARN] /Users/kevin/projects/playground/basics/lib/src/main/java/basics/counter/Strategy.java:38: At-clause should have a non-empty description. [NonEmptyAtclauseDescription] Checkstyle rule violations were found. See the report at: file:///Users/kevin/projects/playground/basics/lib/build/reports/checkstyle/main.html Checkstyle files with violations: 1 Checkstyle violations by severity: [warning:1]

Task :lib:checkstyleTest [ant:checkstyle] [WARN] /Users/kevin/projects/playground/basics/lib/src/test/java/basics/counter/StrategyTest.java:10: First sentence of Javadoc is missing an ending period. [SummaryJavadoc] Checkstyle rule violations were found. See the report at: file:///Users/kevin/projects/playground/basics/lib/build/reports/checkstyle/test.html Checkstyle files with violations: 1 Checkstyle violations by severity: [warning:1]

Task :lib:pmdMain 1 PMD rule violations were found. See the report at: file:///Users/kevin/projects/playground/basics/lib/build/reports/pmd/main.html

Task :lib:pitest FAILED 10:57:10 PM PIT >> INFO : --------------------------------------------------------------------------- 10:57:10 PM PIT >> INFO : Enabled (+) and disabled (-) features. 10:57:10 PM PIT >> INFO : ----------------------------------------- 10:57:10 PM PIT >> INFO : +FANN Filters mutations in classes and methods with matching annotations of class or runtime retention 10:57:10 PM PIT >> INFO : [annotation] Annotation to avoid (full package name not required) 10:57:10 PM PIT >> INFO : +FENUM Filters mutations in enum constructors 10:57:10 PM PIT >> INFO : +FFBLOCK Filters mutations in code duplicated by finally block inlining 10:57:10 PM PIT >> INFO : +FFEACH Filters mutations in compiler generated code that implements for each loops 10:57:10 PM PIT >> INFO : +FFLOOP Filters any mutations to increments in for loops as they may cause timeouts 10:57:10 PM PIT >> INFO : +FINFINC Filters mutations to increments that may cause infinite loops 10:57:10 PM PIT >> INFO : +FINFIT Filters mutations that may cause infinite loops by removing calls to iterator.next 10:57:10 PM PIT >> INFO : +FINULL Filters mutations in compiler generated code that checks for null by calling getClass 10:57:10 PM PIT >> INFO : +FKOTLIN Filters out junk mutations in bytecode created by compiler for kotlin language features 10:57:10 PM PIT >> INFO : +FLOGCALL Filters mutations in code that makes calls to logging frameworks 10:57:10 PM PIT >> INFO : +FMRNULL Filters mutations in compiler generated code that inserts Objects.requireNonNull for method references 10:57:10 PM PIT >> INFO : +FRECORD Filters mutations in compiler generated record code 10:57:10 PM PIT >> INFO : +FRETEQUIV Filters return vals mutants with bytecode equivalent to the unmutated class 10:57:10 PM PIT >> INFO : +FSEQUIVEQUALS Filters equivalent mutations that affect only performance in short cutting equals methods 10:57:10 PM PIT >> INFO : +FSTATI Filters mutations in static initializers and code called only from them 10:57:10 PM PIT >> INFO : +FTRYWR Filters mutations in code generated for try with resources statements 10:57:10 PM PIT >> INFO : -CLASSLIMIT Limits the maximum number of mutations per class 10:57:10 PM PIT >> INFO : [limit] Integer value for maximum mutations to create per class 10:57:10 PM PIT >> INFO : -EXPORT Exports mutants bytecode and other details to disk 10:57:10 PM PIT >> INFO : --------------------------------------------------------------------------- 10:57:11 PM PIT >> FINE : Running report with ReportOptions [targetClasses=[basics.], excludedMethods=[], excludedClasses=[], excludedTestClasses=[], codePaths=[/Users/kevin/projects/playground/basics/lib/build/classes/java/main], reportDir=/Users/kevin/projects/playground/basics/lib/build/reports/pitest, historyInputLocation=null, historyOutputLocation=null, sourceDirs=[/Users/kevin/projects/playground/basics/lib/src/main/resources, /Users/kevin/projects/playground/basics/lib/src/main/java], classPathElements=[/Users/kevin/.gradle/caches/modules-2/files-2.1/org.pitest/pitest/1.7.3/4cc79d87a737e29b551159d96dbb0ec2e2e00f83/pitest-1.7.3.jar, /Users/kevin/projects/playground/basics/lib/build/classes/java/test, /Users/kevin/projects/playground/basics/lib/build/resources/test, /Users/kevin/projects/playground/basics/lib/build/classes/java/main, /Users/kevin/projects/playground/basics/lib/build/resources/main, /Users/kevin/.gradle/caches/modules-2/files-2.1/org.apache.commons/commons-lang3/3.12.0/c6842c86792ff03b9f1d1fe2aab8dc23aa6c6f0e/commons-lang3-3.12.0.jar, /Users/kevin/.gradle/caches/modules-2/files-2.1/org.hibernate.validator/hibernate-validator/7.0.2.Final/62275c153f967395a0bb3e7a6373ff071ec0f103/hibernate-validator-7.0.2.Final.jar, /Users/kevin/.gradle/caches/modules-2/files-2.1/org.junit.jupiter/junit-jupiter-engine/5.8.2/c598b4328d2f397194d11df3b1648d68d7d990e3/junit-jupiter-engine-5.8.2.jar, /Users/kevin/.gradle/caches/modules-2/files-2.1/org.junit.platform/junit-platform-engine/1.8.2/b737de09f19864bd136805c84df7999a142fec29/junit-platform-engine-1.8.2.jar, /Users/kevin/.gradle/caches/modules-2/files-2.1/org.junit.platform/junit-platform-commons/1.8.2/32c8b8617c1342376fd5af2053da6410d8866861/junit-platform-commons-1.8.2.jar, /Users/kevin/.gradle/caches/modules-2/files-2.1/org.junit.jupiter/junit-jupiter-params/5.8.2/ddeafe92fc263f895bfb73ffeca7fd56e23c2cce/junit-jupiter-params-5.8.2.jar, /Users/kevin/.gradle/caches/modules-2/files-2.1/org.junit.jupiter/junit-jupiter/5.8.2/5a817b1e63f1217e5c586090c45e681281f097ad/junit-jupiter-5.8.2.jar, /Users/kevin/.gradle/caches/modules-2/files-2.1/org.junit.jupiter/junit-jupiter-api/5.8.2/4c21029217adf07e4c0d0c5e192b6bf610c94bdc/junit-jupiter-api-5.8.2.jar, /Users/kevin/.gradle/caches/modules-2/files-2.1/org.glassfish/jakarta.el/4.0.2/d67e18e7b3500dc91fe941ed3a1807074eaac750/jakarta.el-4.0.2.jar, /Users/kevin/.gradle/caches/modules-2/files-2.1/jakarta.validation/jakarta.validation-api/3.0.0/8c8eecc40da64037d7731356511c568d466f8480/jakarta.validation-api-3.0.0.jar, /Users/kevin/.gradle/caches/modules-2/files-2.1/org.jboss.logging/jboss-logging/3.4.1.Final/40fd4d696c55793e996d1ff3c475833f836c2498/jboss-logging-3.4.1.Final.jar, /Users/kevin/.gradle/caches/modules-2/files-2.1/com.fasterxml/classmate/1.5.1/3fe0bed568c62df5e89f4f174c101eab25345b6c/classmate-1.5.1.jar, /Users/kevin/.gradle/caches/modules-2/files-2.1/org.opentest4j/opentest4j/1.2.0/28c11eb91f9b6d8e200631d46e20a7f407f2a046/opentest4j-1.2.0.jar, /Users/kevin/.gradle/caches/modules-2/files-2.1/jakarta.el/jakarta.el-api/4.0.0/ba9834bd2dba5014072bacbafecffaec1673b0f9/jakarta.el-api-4.0.0.jar], mutators=[], features=[], dependencyAnalysisMaxDistance=-1, jvmArgs=[-Djava.awt.headless=true], numberOfThreads=4, timeoutFactor=1.25, timeoutConstant=4000, targetTests=[^basics..$], loggingClasses=[], verbosity=VERBOSE, failWhenNoMutations=true, outputs=[XML, HTML], groupConfig=TestGroupConfig [excludedGroups=[], includedGroups=[]], fullMutationMatrix=false, mutationUnitSize=0, shouldCreateTimestampedReports=false, detectInlinedCode=false, exportLineCoverage=false, mutationThreshold=0, testStrengthThreshold=0, coverageThreshold=0, mutationEngine=gregor, javaExecutable=null, includeLaunchClasspath=false, properties={}, maxSurvivors=-1, excludedRunners=[], includedTestMethods=[], testPlugin=, useClasspathJar=false, skipFailingTests=false] 10:57:11 PM PIT >> FINE : System class path is /Users/kevin/.gradle/caches/modules-2/files-2.1/org.pitest/pitest-command-line/1.7.3/6b38162935953025ec76b3fa42c1e86a3035930/pitest-command-line-1.7.3.jar:/Users/kevin/.gradle/caches/modules-2/files-2.1/org.pitest/pitest-entry/1.7.3/4ac3ac202b16e4b1aac7af79d6e6fe6261bc9fa3/pitest-entry-1.7.3.jar:/Users/kevin/.gradle/caches/modules-2/files-2.1/org.pitest/pitest/1.7.3/4cc79d87a737e29b551159d96dbb0ec2e2e00f83/pitest-1.7.3.jar 10:57:11 PM PIT >> FINE : Maximum available memory is 4096 mb 10:57:11 PM PIT >> FINE : Incremental analysis set 135 mutations to a status of NOT_STARTED 10:57:11 PM PIT >> INFO : Incremental analysis reduced number of mutations by 0 10:57:11 PM PIT >> INFO : Created 12 mutation test units in pre scan 10:57:11 PM PIT >> FINE : MINION : Installing PIT agent 10:57:11 PM PIT >> INFO : Sending 29 test classes to minion 10:57:11 PM PIT >> INFO : Sent tests to minion 10:57:11 PM PIT >> INFO : MINION : 10:57:11 PM PIT >> FINE : Expecting 29 tests classes from parent 10:57:11 PM PIT >> INFO : MINION : 10:57:11 PM PIT >> FINE : Tests classes received 10:57:11 PM PIT >> FINE : MINION : java.lang.NoClassDefFoundError: org/junit/runner/manipulation/Filter 10:57:11 PM PIT >> FINE : MINION : at org.pitest.junit.JUnitCompatibleConfiguration.testUnitFinder(JUnitCompatibleConfiguration.java:54) 10:57:11 PM PIT >> FINE : MINION : at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) 10:57:11 PM PIT >> FINE : MINION : at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) 10:57:11 PM PIT >> FINE : MINION : at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) 10:57:11 PM PIT >> FINE : MINION : at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) 10:57:11 PM PIT >> FINE : MINION : at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) 10:57:11 PM PIT >> FINE : MINION : at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) 10:57:11 PM PIT >> FINE : MINION : at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) 10:57:11 PM PIT >> FINE : MINION : at org.pitest.mutationtest.config.PrioritisingTestConfiguration.makeFinder(PrioritisingTestConfiguration.java:61) 10:57:11 PM PIT >> FINE : MINION : at org.pitest.mutationtest.config.PrioritisingTestConfiguration.(PrioritisingTestConfiguration.java:20) 10:57:11 PM PIT >> FINE : MINION : at org.pitest.mutationtest.config.MinionSettings.getTestFrameworkPlugin(MinionSettings.java:38) 10:57:11 PM PIT >> FINE : MINION : at org.pitest.coverage.execute.CoverageMinion.createTestPlugin(CoverageMinion.java:168) 10:57:11 PM PIT >> FINE : MINION : at org.pitest.coverage.execute.CoverageMinion.getTestsFromParent(CoverageMinion.java:132) 10:57:11 PM PIT >> FINE : MINION : at org.pitest.coverage.execute.CoverageMinion.main(CoverageMinion.java:82) 10:57:11 PM PIT >> FINE : MINION : Caused by: java.lang.ClassNotFoundException: org.junit.runner.manipulation.Filter 10:57:11 PM PIT >> FINE : MINION : at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) 10:57:11 PM PIT >> FINE : MINION : at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) 10:57:11 PM PIT >> FINE : MINION : at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520) 10:57:11 PM PIT >> FINE : MINION : ... 14 more 10:57:11 PM PIT >> INFO : MINION : 10:57:11 PM PIT >> SEVERE : Error calculating coverage. Process will exit. 10:57:11 PM PIT >> INFO : MINION : java.lang.NoClassDefFoundError: org/junit/runner/manipulation/Filter 10:57:11 PM PIT >> INFO : MINION : at org.pitest.junit.JUnitCompatibleConfiguration.testUnitFinder(JUnitCompatibleConfiguration.java:54) 10:57:11 PM PIT >> INFO : MINION : at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) 10:57:11 PM PIT >> INFO : MINION : at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) 10:57:11 PM PIT >> INFO : MINION : at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) 10:57:11 PM PIT >> INFO : MINION : at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) 10:57:11 PM PIT >> INFO : MINION : at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) 10:57:11 PM PIT >> INFO : MINION : at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) 10:57:11 PM PIT >> INFO : MINION : at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) 10:57:11 PM PIT >> INFO : MINION : at org.pitest.mutationtest.config.PrioritisingTestConfiguration.makeFinder(PrioritisingTestConfiguration.java:61) 10:57:11 PM PIT >> INFO : MINION : at org.pitest.mutationtest.config.PrioritisingTestConfiguration.(PrioritisingTestConfiguration.java:20) 10:57:11 PM PIT >> INFO : MINION : at org.pitest.mutationtest.config.MinionSettings.getTestFrameworkPlugin(MinionSettings.java:38) 10:57:11 PM PIT >> INFO : MINION : at org.pitest.coverage.execute.CoverageMinion.createTestPlugin(CoverageMinion.java:168) 10:57:11 PM PIT >> INFO : MINION : at org.pitest.coverage.execute.CoverageMinion.getTestsFromParent(CoverageMinion.java:132) 10:57:11 PM PIT >> INFO : MINION : at org.pitest.coverage.execute.CoverageMinion.main(CoverageMinion.java:82) 10:57:11 PM PIT >> INFO : MINION : Caused by: java.lang.ClassNotFoundException: org.junit.runner.manipulation.Filter 10:57:11 PM PIT >> INFO : MINION : at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) 10:57:11 PM PIT >> INFO : MINION : at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) 10:57:11 PM PIT >> INFO : MINION : at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520) 10:57:11 PM PIT >> INFO : MINION : ... 14 more 10:57:11 PM PIT >> SEVERE : Coverage generator Minion exited abnormally due to UNKNOWN_ERROR Exception in thread "main" org.pitest.util.PitError: Coverage generation minion exited abnormally!

Please copy and paste the information and the complete stacktrace below when reporting an issue VM : OpenJDK 64-Bit Server VM Vendor : Eclipse Adoptium Version : 17+35 Uptime : 973 Input -> 1 : -Dfile.encoding=UTF-8 2 : -Duser.country=US 3 : -Duser.language=en 4 : -Duser.variant BootClassPathSupported : false

Please copy and paste the information and the complete stacktrace below when reporting an issue VM : OpenJDK 64-Bit Server VM Vendor : Eclipse Adoptium Version : 17+35 Uptime : 974 Input -> 1 : -Dfile.encoding=UTF-8 2 : -Duser.country=US 3 : -Duser.language=en 4 : -Duser.variant BootClassPathSupported : false

    at org.pitest.util.Unchecked.translateCheckedException(Unchecked.java:20)
    at org.pitest.coverage.execute.DefaultCoverageGenerator.calculateCoverage(DefaultCoverageGenerator.java:106)
    at org.pitest.coverage.execute.DefaultCoverageGenerator.calculateCoverage(DefaultCoverageGenerator.java:52)
    at org.pitest.mutationtest.tooling.MutationCoverage.runAnalysis(MutationCoverage.java:149)
    at org.pitest.mutationtest.tooling.MutationCoverage.runReport(MutationCoverage.java:139)
    at org.pitest.mutationtest.tooling.EntryPoint.execute(EntryPoint.java:123)
    at org.pitest.mutationtest.tooling.EntryPoint.execute(EntryPoint.java:54)
    at org.pitest.mutationtest.commandline.MutationCoverageReport.runReport(MutationCoverageReport.java:98)
    at org.pitest.mutationtest.commandline.MutationCoverageReport.main(MutationCoverageReport.java:45)

Caused by: org.pitest.util.PitError: Coverage generation minion exited abnormally!

Please copy and paste the information and the complete stacktrace below when reporting an issue VM : OpenJDK 64-Bit Server VM Vendor : Eclipse Adoptium Version : 17+35 Uptime : 973 Input -> 1 : -Dfile.encoding=UTF-8 2 : -Duser.country=US 3 : -Duser.language=en 4 : -Duser.variant BootClassPathSupported : false

    at org.pitest.coverage.execute.DefaultCoverageGenerator.gatherCoverageData(DefaultCoverageGenerator.java:148)
    at org.pitest.coverage.execute.DefaultCoverageGenerator.calculateCoverage(DefaultCoverageGenerator.java:90)
    ... 7 more

FAILURE: Build failed with an exception.

BUILD FAILED in 15s 11 actionable tasks: 11 executed ➜ basics git:(feature/improve-strategy-tests) ✗

kphannan commented 2 years ago

After doing some digging through internet searches I stumbled on a post noting there is a dependency needed.

testRuntimeOnly 'org.pitest:pitest-junit5-plugin:0.15'

After adding the dependency the mutation tests ran. This could be an issue with the PiTest Gradle plugin instead of PiTest itself.

szpak commented 2 years ago

To use PIT with JUnit 5 you need pitest-junit5-plugin as a dependency or you can ask the Gradle plugin to do it for you, as mentioned in the FAQ:

plugins {
    id 'java'
    id 'info.solidsoft.pitest' version '1.7.0'
}

pitest {
    //adds dependency to org.pitest:pitest-junit5-plugin and sets "testPlugin" to "junit5"
    junit5PluginVersion = '0.15'    //or 0.14 for Junit Jupiter 5.7 (JUnit Platform 1.7)
    // ...
}
szpak commented 2 years ago

Nevertheless, you should get some nicer error and a suggestion about adding some PIT plugin for TestNG or JUnit 5 instead of:

java.lang.NoClassDefFoundError: org/junit/runner/manipulation/Filter

@hcoles Might it be some regression in 1.7.x? I was improving the error message in https://github.com/hcoles/pitest/pull/745, so probably it was displayed in the past.