unable to use latest versions of pitest, tesng and jmockit together

estekhin commented 10 years ago

Related to https://github.com/jmockit/jmockit1/issues/24 which describes problems when using gradle+testng+pitest+jmockit.

When using maven pitest plugin and a very simple project (a pom.xml, a "main" class and a test class) simply adding jmockit dependency breaks the build even if the jmockit is not used in tests.

The pom xml is:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>sandbox</groupId>
    <artifactId>sandbox-pitest-jmockit</artifactId>
    <version>1.0-SNAPSHOT</version>

    <dependencies>
        <dependency>
            <groupId>org.jmockit</groupId>
            <artifactId>jmockit</artifactId>
            <version>1.13</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.testng</groupId>
            <artifactId>testng</artifactId>
            <version>6.8.8</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.pitest</groupId>
                <artifactId>pitest-maven</artifactId>
                <version>1.1.1</version>
            </plugin>
        </plugins>
    </build>
</project>

mvn clean org.pitest:pitest-maven:mutationCoverage results in a slave failure which seems to be due to Caused by: java.lang.ClassNotFoundException: org.junit.runner.manipulation.Filter (where junit came from?).

Removing the jmockit dependency from the pom above will allow the pitest to run successfully.

StefanPenndorf commented 10 years ago

The problem is that jmockit 1.13 includes a bundles org.junit.runner.Runner class. Pitest checks for this class to determine whether JUnit is used as test framework. It finds Runner and thinks JUnit is on the class path but because Runner is the only JUnit class available it fails when loading JUnits Filter class.

I think the best solution would be removing the bundled org.junit.runner.Runner class from jmockit distro: If JUnit folks update this class some day it will not be clear whether the JUnit version or the jmockit version will be picked by class loader. That might cause other troubles in the future.

estekhin commented 10 years ago

JMockit+JUnit explicitly requires that jmockit jar is before junit jar in the classpath. Now I see that this is due to that bundled org.junit.runner.Runner which is modified to include static initializer code that tries to initialize JMockit.

estekhin commented 10 years ago

I am not sure that this should be fixed on pitest side, so I will close this issue.

estekhin commented 9 years ago

@KyleRogers can you review this once more?

I am looking at https://github.com/hcoles/pitest/blob/master/pitest/src/main/java/org/pitest/mutationtest/config/ConfigurationFactory.java and thinking about these variants:

it is possible to use some other class to detect JUnit vs TestNG?
it is possible to allow external configuration of JUnit/TestNG? For example, it could be an option which, when specified, will skip classRepository.fetchClass(..).hasSome() checks and force-add the requested config. And when it is not specified, the current autodetection behavior will simply continue to work.

StefanPenndorf commented 9 years ago

Is it possible to use some other class? Yes, I think every class or interface that exists since JUnit 4.0 will do the job (e.g. org.junit.runner.JUnitCore). But I don't know if that will break someone else.....
Is it possible to overwrite autoconfiguration with explicit configuration of the desired framework? That is possible although we have to find someone who implements it ;-).

What do you think @hcoles ?

hcoles commented 9 years ago

Switching to lookup org.junit.Test would be simple and might actually provide a better behaviour when only junit 3 was one the classpath.

Explicitly configuring would also make sense - happy to accept a PR.

hcoles commented 9 years ago

I've switched the class - however I don't have a test in place for TestNG + JMockit so no idea if this fixes the issue.

StefanPenndorf commented 9 years ago

You can add the jmockit dependency to the basic testng maven integration test (maven-verification module). If the test fails this does not fix the issue.

Am 27.01.2015 um 22:21 schrieb Henry Coles:

I've switched the class - however I don't have a test in place for TestNG + JMockit so no idea if this fixes the issue.

— Reply to this email directly or view it on GitHub https://github.com/hcoles/pitest/issues/148#issuecomment-71730421.

hcoles commented 9 years ago

I've added a new test that is basically C&P of the existing testng case with the added dependencies.

Passes with 1.1.4 snapshot, fails with 1.1.3.

Feels like a rather shallow test as it doesn't actually use jmockit, but I'm a little short on OSS time just now.

StefanPenndorf commented 9 years ago

@estekhin PIT 1.1.4 has been released, can you check if it works for you?

estekhin commented 9 years ago

I will try it with the real project first thing on Monday.

estekhin commented 9 years ago

A simple Maven+TestNG+JMockit project fails with pitest-1.1.3, runs with pitest-1.1.4,

A pretty complex Gradle+TestNG+JMockit project with pitest-1.1.4 at the very least starts to run, but now it fails due to "Exception in thread "main" org.pitest.help.PitHelpError: All tests did not pass without mutation when calculating line coverage. Mutation testing requires a green suite.", but the test suite passes when invoked simply for testing, without pitest.

More then a half of the suite passes before this "java.exe' finished with non-zero exit value 1" happens and causes the test task to fail and skip all the remaining tests.

Removing only JMockit-based tests does not help. Removing both JMockit-based tests AND JMockit dependency allows pitest to proceed.

Just in case, the exact log output is:

10:05:19.164 [ERROR] [system.err] Exception in thread "main" org.pitest.help.PitHelpError: All tests did not pass without mutation when calculating line coverage. Mutation testing requires a green suite.
See http://pitest.org for more details.
10:05:19.165 [ERROR] [system.err]   at org.pitest.coverage.execute.DefaultCoverageGenerator.verifyBuildSuitableForMutationTesting(DefaultCoverageGenerator.java:107)
10:05:19.165 [ERROR] [system.err]   at org.pitest.coverage.execute.DefaultCoverageGenerator.calculateCoverage(DefaultCoverageGenerator.java:92)
10:05:19.166 [ERROR] [system.err]   at org.pitest.coverage.execute.DefaultCoverageGenerator.calculateCoverage(DefaultCoverageGenerator.java:49)
10:05:19.166 [ERROR] [system.err]   at org.pitest.mutationtest.tooling.MutationCoverage.runReport(MutationCoverage.java:112)
10:05:19.166 [ERROR] [system.err]   at org.pitest.mutationtest.tooling.EntryPoint.execute(EntryPoint.java:101)
10:05:19.166 [ERROR] [system.err]   at org.pitest.mutationtest.tooling.EntryPoint.execute(EntryPoint.java:43)
10:05:19.166 [ERROR] [system.err]   at org.pitest.mutationtest.commandline.MutationCoverageReport.runReport(MutationCoverageReport.java:72)
10:05:19.166 [ERROR] [system.err]   at org.pitest.mutationtest.commandline.MutationCoverageReport.main(MutationCoverageReport.java:43)
10:05:19.267 [ERROR] [org.gradle.BuildExceptionReporter] 
10:05:19.268 [ERROR] [org.gradle.BuildExceptionReporter] FAILURE: Build failed with an exception.
10:05:19.268 [ERROR] [org.gradle.BuildExceptionReporter] 
10:05:19.268 [ERROR] [org.gradle.BuildExceptionReporter] * What went wrong:
10:05:19.269 [ERROR] [org.gradle.BuildExceptionReporter] Execution failed for task ':chronos-client:pitest'.
10:05:19.269 [ERROR] [org.gradle.BuildExceptionReporter] > Process 'command 'C:\Program Files\Java\jdk1.8.0_31\bin\java.exe'' finished with non-zero exit value 1
10:05:19.269 [ERROR] [org.gradle.BuildExceptionReporter] 
10:05:19.269 [ERROR] [org.gradle.BuildExceptionReporter] * Exception is:
10:05:19.271 [ERROR] [org.gradle.BuildExceptionReporter] org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':chronos-client:pitest'.
10:05:19.271 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:69)
10:05:19.271 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:46)
10:05:19.271 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.api.internal.tasks.execution.PostExecutionAnalysisTaskExecuter.execute(PostExecutionAnalysisTaskExecuter.java:35)
10:05:19.271 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:64)
10:05:19.272 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:58)
10:05:19.272 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:42)
10:05:19.272 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:52)
10:05:19.272 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:53)
10:05:19.272 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43)
10:05:19.272 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.api.internal.AbstractTask.executeWithoutThrowingTaskFailure(AbstractTask.java:305)
10:05:19.273 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.execution.taskgraph.AbstractTaskPlanExecutor$TaskExecutorWorker.executeTask(AbstractTaskPlanExecutor.java:79)
10:05:19.273 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.execution.taskgraph.AbstractTaskPlanExecutor$TaskExecutorWorker.processTask(AbstractTaskPlanExecutor.java:63)
10:05:19.273 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.execution.taskgraph.AbstractTaskPlanExecutor$TaskExecutorWorker.run(AbstractTaskPlanExecutor.java:51)
10:05:19.273 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor.process(DefaultTaskPlanExecutor.java:23)
10:05:19.273 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter.execute(DefaultTaskGraphExecuter.java:88)
10:05:19.273 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.execution.SelectedTaskExecutionAction.execute(SelectedTaskExecutionAction.java:29)
10:05:19.273 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:62)
10:05:19.274 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.execution.DefaultBuildExecuter.access$200(DefaultBuildExecuter.java:23)
10:05:19.274 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.execution.DefaultBuildExecuter$2.proceed(DefaultBuildExecuter.java:68)
10:05:19.274 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.execution.DryRunBuildExecutionAction.execute(DryRunBuildExecutionAction.java:32)
10:05:19.274 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:62)
10:05:19.274 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:55)
10:05:19.274 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.initialization.DefaultGradleLauncher.doBuildStages(DefaultGradleLauncher.java:149)
10:05:19.274 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.initialization.DefaultGradleLauncher.doBuild(DefaultGradleLauncher.java:106)
10:05:19.275 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.initialization.DefaultGradleLauncher.run(DefaultGradleLauncher.java:86)
10:05:19.275 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.launcher.exec.InProcessBuildActionExecuter$DefaultBuildController.run(InProcessBuildActionExecuter.java:80)
10:05:19.275 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.launcher.cli.ExecuteBuildAction.run(ExecuteBuildAction.java:33)
10:05:19.275 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.launcher.cli.ExecuteBuildAction.run(ExecuteBuildAction.java:24)
10:05:19.275 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:36)
10:05:19.275 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:26)
10:05:19.275 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.launcher.cli.RunBuildAction.run(RunBuildAction.java:51)
10:05:19.276 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.internal.Actions$RunnableActionAdapter.execute(Actions.java:171)
10:05:19.276 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:237)
10:05:19.276 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:210)
10:05:19.276 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.launcher.cli.JavaRuntimeValidationAction.execute(JavaRuntimeValidationAction.java:35)
10:05:19.276 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.launcher.cli.JavaRuntimeValidationAction.execute(JavaRuntimeValidationAction.java:24)
10:05:19.276 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:206)
10:05:19.277 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:169)
10:05:19.277 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:33)
10:05:19.277 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:22)
10:05:19.277 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.launcher.Main.doAction(Main.java:33)
10:05:19.277 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.launcher.bootstrap.EntryPoint.run(EntryPoint.java:45)
10:05:19.277 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.launcher.bootstrap.ProcessBootstrap.runNoExit(ProcessBootstrap.java:54)
10:05:19.277 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.launcher.bootstrap.ProcessBootstrap.run(ProcessBootstrap.java:35)
10:05:19.278 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.launcher.GradleMain.main(GradleMain.java:23)
10:05:19.278 [ERROR] [org.gradle.BuildExceptionReporter] Caused by: org.gradle.process.internal.ExecException: Process 'command 'C:\Program Files\Java\jdk1.8.0_31\bin\java.exe'' finished with non-zero exit value 1
10:05:19.278 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.process.internal.DefaultExecHandle$ExecResultImpl.assertNormalExitValue(DefaultExecHandle.java:365)
10:05:19.278 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.process.internal.DefaultJavaExecAction.execute(DefaultJavaExecAction.java:31)
10:05:19.278 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.api.tasks.JavaExec.exec(JavaExec.java:60)
10:05:19.278 [ERROR] [org.gradle.BuildExceptionReporter]    at info.solidsoft.gradle.pitest.PitestTask.super$5$exec(PitestTask.groovy)
10:05:19.278 [ERROR] [org.gradle.BuildExceptionReporter]    at info.solidsoft.gradle.pitest.PitestTask.exec(PitestTask.groovy:175)
10:05:19.279 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:63)
10:05:19.279 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.api.internal.project.taskfactory.AnnotationProcessingTaskFactory$StandardTaskAction.doExecute(AnnotationProcessingTaskFactory.java:218)
10:05:19.279 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.api.internal.project.taskfactory.AnnotationProcessingTaskFactory$StandardTaskAction.execute(AnnotationProcessingTaskFactory.java:211)
10:05:19.279 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.api.internal.project.taskfactory.AnnotationProcessingTaskFactory$StandardTaskAction.execute(AnnotationProcessingTaskFactory.java:200)
10:05:19.279 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:579)
10:05:19.279 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.api.internal.AbstractTask$TaskActionWrapper.execute(AbstractTask.java:562)
10:05:19.279 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeAction(ExecuteActionsTaskExecuter.java:80)
10:05:19.280 [ERROR] [org.gradle.BuildExceptionReporter]    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:61)
10:05:19.280 [ERROR] [org.gradle.BuildExceptionReporter]    ... 44 more
10:05:19.280 [ERROR] [org.gradle.BuildExceptionReporter]

estekhin commented 9 years ago

Enabling pitest.verbose option shows that after some time the tests start to fail with java.lang.StackOverflowError but sadly there is no stacktrace for it.

The first such error in the log is

10:55:48.550 [ERROR] [system.err] 
10:55:48.550 [ERROR] [system.err] 10:55:48 AM PIT >> INFO : SLAVE : 10:55:48 AM PIT >> FINE : Gathering coverage for test Description [testClass=com.db.chronos.client.RingBufferTraceLoggerTraceTest, name=_]
10:55:48.551 [ERROR] [system.err] 
10:55:48.584 [ERROR] [system.err] 10:55:48 AM PIT >> FINE : SLAVE : [TestNG] Running:
  com.db.chronos.client.RingBufferTraceLoggerTraceTest

10:55:48.585 [ERROR] [system.err] 
10:55:48.649 [ERROR] [system.err] 10:55:48 AM PIT >> FINE : SLAVE : 
===============================================
com.db.chronos.client.RingBufferTraceLoggerTraceTest
Total tests run: 2, Failures: 1, Skips: 1
===============================================

10:55:48.649 [ERROR] [system.err] 
10:55:48.652 [ERROR] [system.err] 10:55:48 AM PIT >> FINE : SLAVE : FAIL Description [testClass=com.db.chronos.client.RingBufferTraceLoggerTraceTest, name=trace] -> java.lang.StackOverflowError
10:55:48.652 [ERROR] [system.err] FAIL Description [testClass=com.db.chronos.client.RingBufferTraceLoggerTraceTest, name=_] -> java.lang.StackOverflowError
10:55:48.652 [ERROR] [system.err]

estekhin commented 9 years ago

Modified pitest a little to print the full stack trace inside org.pitest.coverage.execute.ErrorListener#onTestFailure(), got this:

java.lang.StackOverflowError
    at mockit.internal.state.TestRun.getMock(TestRun.java:136)
    at mockit.internal.state.TestRun.updateMockState(TestRun.java:121)
    at org.testng.internal.Parameters.getInjectedParameter(Parameters.java)
    at mockit.integration.testng.internal.TestNGRunnerDecorator$MockParameters.getInjectedParameter(TestNGRunnerDecorator.java:45)
    at org.testng.internal.Parameters.getInjectedParameter(Parameters.java)
    at mockit.integration.testng.internal.TestNGRunnerDecorator$MockParameters.getInjectedParameter(TestNGRunnerDecorator.java:45)
    at org.testng.internal.Parameters.getInjectedParameter(Parameters.java)
    at mockit.integration.testng.internal.TestNGRunnerDecorator$MockParameters.getInjectedParameter(TestNGRunnerDecorator.java:45)
    at org.testng.internal.Parameters.getInjectedParameter(Parameters.java)
    at mockit.integration.testng.internal.TestNGRunnerDecorator$MockParameters.getInjectedParameter(TestNGRunnerDecorator.java:45)
    at org.testng.internal.Parameters.getInjectedParameter(Parameters.java)
...

StefanPenndorf commented 9 years ago

Sorry I've no idea what causes this problem. Maybe jmockit and pitest bytecode generation/insertion interfering with each other?

Maybe it's worth asking the jmockit people what could cause a stack overflow here?

hcoles commented 9 years ago

@KyleRogers have tracked this down to an assumption made in jmockit that there will be only 1 instance of TestNG.

I can fix the code to make this assumption true for when running via pitest, but it's ugly. Will probably need to do this as looking into the TestNG code it sets a static instance member when you call it's constructor. erk.

hcoles commented 9 years ago

I believe this is properly fixed in the latest 1.1.5-SNAPSHOT - it runs without error against https://github.com/estekhin/sandbox-gradle-testng-jmockit-pitest.

Any feedback from more complex projects would be appreciated.

estekhin commented 9 years ago

Master fails to build due to test failures in newly-added tests.

I have built it with -DskipTests and used the 1.1.5-snapshot in my work project - gradle clean pitest successfully run to completion and produced pitest reports for all modules.

@hcoles seems like you have fixed in pitest what, in ideal world, should have been fixed in jmockit and testing =)

estekhin commented 9 years ago

Test failures seems to be due to my host configuration - I have non-standard user home location and, consequently, non-standard local maven repo location. I re-run the build with the proper configuration and it passed.

hcoles commented 9 years ago

Glad it's working - thanks everyone for your help.

hcoles / pitest

unable to use latest versions of pitest, tesng and jmockit together #148