search

STAMP-project / dspot

Automatically detect and generate missing assertions for Junit test cases (also known as test amplification)
https://dspot-demo.stamp-project.eu/
GNU Lesser General Public License v3.0
114 stars 28 forks source link

DSpot generated tests are not all mutation-proof (according to Descartes) #822

Closed gibello closed 5 years ago

gibello commented 5 years ago

Characteristics

Description

Although DSpot statistically increases mutation coverage (as measured by Descartes), some of the amplified tests do not detect code mutations operated by Descartes.

Should DSpot tests be mutation-proof ? It would be an interesting feature.

Steps to reproduce

Note: in the example below, JacocoCoverageSelector was used, because PitMutant generates no amplified test at all... Not sure it is sufficient to explain why tests are not mutation-proof.

Descartes / DSpot usecase (with OW2 Joram) at https://github.com/STAMP-project/descartes-usecases-output/blob/master/OW2/Joram can help reproduce the issue: there is a step-by-step documentation.

A more readable output, generated with STAMP-project/descartes-usecases-output (note that it suggests a DSpot command... to fix the DSpot-generated test ! I didn't try it...): we can see some Ampl* tests still pass when code is mutated by Descartes.

==========================================================================
CRITICAL TEST FAILURE: test suite GREEN upon code mutation
In class org.objectweb.joram.mom.messages.Message, method getClientID (line 322) was updated as follows:
        All methods instructions replaced by: return null;
        The following test(s) still PASS:
                org.objectweb.joram.mom.messages.AmplMessageEncodingTest.run(org.objectweb.joram.mom.messages.AmplMessageEncodingTest)
==========================================================================
==========================================================================
CRITICAL TEST FAILURE: test suite GREEN upon code mutation
In class org.objectweb.joram.mom.messages.Message, method getText (line 567) was updated as follows:
        All methods instructions replaced by: return null;
        The following test(s) still PASS:
                org.objectweb.joram.mom.messages.AmplMessageEncodingTest.run(org.objectweb.joram.mom.messages.AmplMessageEncodingTest)
==========================================================================
==========================================================================
CRITICAL TEST FAILURE: test suite GREEN upon code mutation
In class org.objectweb.joram.mom.messages.Message, method isPersistent (line 201) was updated as follows:
        All method body replaced by: return true
        The following test(s) still PASS:
                org.objectweb.joram.mom.messages.AmplMessageEncodingTest.run(org.objectweb.joram.mom.messages.AmplMessageEncodingTest)
==========================================================================
Suggested DSpot fix(es):
mvn eu.stamp-project:dspot-maven:2.1.1-SNAPSHOT:amplify-unit-tests -Dpath-to-properties=/tmp/stamp2956334477360652493.tmp -Dtest=org.objectweb.joram.mom.messages.AmplMessageEncodingTest -Dtest-criterion=JacocoCoverageSelector -Dgenerate-new-test-class=true -Diteration=1
==========================================================================
danglotb commented 5 years ago

Hello @gibello

I don't understand the issue.

You amplified your tests using the JacocoCoverageSelector. So you increased the instruction coverage and the number of paths executed. But you are expecting that some mutants, generated by descartes are killed?

gibello commented 5 years ago

Hi @danglotb ,

Yes, I expect generated tests to kill mutants, at least in a global STAMP perspective: Descartes considers tests that don't kill mutants as bugs - something that must be fixed by developers.

In a Descartes perspective, a test that does not kill mutants is bad - so DSpot can generate tests that STAMP tools consider bad !

Note that I use Jacoco because PitMutant generates... nothing, although mutations survive (according to Descartes) - which means that they are detectable, so DSpot should generate something ? I would prefer to use PitMutant, in fact... but it doesn't really seem to work. And with Jacoco, DSpot generates tests, considered by Descartes as better than before (the amplified test suite kills more mutations than the existing one), at least in percentage.

Of course, I understand it is difficult to generate tests that detect mutations (even for a human developer, it is difficult to fix them so they detect mutations !). That's why this issue is labeled "feature", not "bug".

This is about science, I guess, not just development ?

danglotb commented 5 years ago

But if you use JacocoCoverageSelector, nothing ensure you that the amplified test methods are killing new mutants.

The fact that they are is a "collateral" benefits and can be explained by the fact that since amplified test methods increase the coverage, the amplified test methods then execute potentially more mutants, and maybe also kill them since DSpot always generate assertions.

I guess it is just a happy coincidence that you kill mutants.

Are you using the same mutants operator between your off-line descartes execution and the descartes inside DSpot?

By default in DSpot, you have the default mutators of descartes.

In fact, this is weird that you cannot improve the mutation score using the PitMutantScoreSelector but it does with JacocoCoverageSelector.

I'm gonna try to reproduce and keep you update.

Thank you.

gibello commented 5 years ago

Thanks @danglotb ! I did not change defaults for Descartes, so I am surprised as well not to fix detected mutations with DSpot + PitMutant. And of course I would prefer to use DSpot in PitMutant mode : much more credible when mixing DSpot + Descartes in my use cases, so I am happy if you fix something there :)

danglotb commented 5 years ago

I identified the problem.

There is an issue with how DSpot run PIT and how it is configured.

I'll propose a patch ASAP.

Thank you.

gibello commented 5 years ago

Hi @danglotb, I just tried to amplify twice: once with Jacoco, then amplify the amplified version with PitMutant.

It does not work, but there are some exceptions saying the test methods are "uncompilable" (sic) when I try to amplified an already amplified test.

Build success, with no test generated. Exceptions look like this:

[INFO] Compiling with -proceedOnError -encoding UTF-8 -cp /tmp/joram/joram/joram/mom/core/target/dspot/tmp_test_sources:/home/gibello/.m2/repository/org/ow2/joram/joram-shared/5.17.0-SNAPSHOT/joram-shared-5.17.0-SNAPSHOT.jar:/home/gibello/.m2/repository/org/ow2/joram/a3-common/5.17.0-SNAPSHOT/a3-common-5.17.0-SNAPSHOT.jar:/home/gibello/.m2/repository/org/objectweb/joram/jcup/5.3.1/jcup-5.3.1.jar:/home/gibello/.m2/repository/org/ow2/joram/a3-rt/5.17.0-SNAPSHOT/a3-rt-5.17.0-SNAPSHOT.jar:/home/gibello/.m2/repository/junit/junit/4.6/junit-4.6.jar:/home/gibello/.m2/repository/org/osgi/org.osgi.compendium/5.0.0/org.osgi.compendium-5.0.0.jar:/home/gibello/.m2/repository/org/osgi/org.osgi.core/6.0.0/org.osgi.core-6.0.0.jar:/home/gibello/.m2/repository/org/ow2/jonas/osgi/monolog/5.2.0/monolog-5.2.0.jar:/home/gibello/.m2/repository/org/objectweb/monolog/monolog/2.1.12/monolog-2.1.12.jar:/tmp/joram/joram/joram/mom/core/target/classes/:/tmp/joram/joram/joram/mom/core/target/test-classes/:/tmp/joram/joram/joram/mom/core/target/dspot/dependencies/: -d /tmp/joram/joram/joram/mom/core/target/test-classes -1.8 -preserveAllLocals -noExit -enableJavadoc -proc:none /tmp/joram/joram/joram/mom/core/target/dspot/tmp_test_sources/org/objectweb/joram/mom/proxies/AmplClientContextEncodingTest.java
eu.stamp_project.dspot.AmplificationException: Every test methods are uncompilable
    at eu.stamp_project.utils.compilation.TestCompiler.compileAndDiscardUncompilableMethods(TestCompiler.java:161)
    at eu.stamp_project.utils.compilation.TestCompiler.compileAndRun(TestCompiler.java:131)
    at eu.stamp_project.dspot.assertgenerator.MethodsAssertGenerator.addAssertions(MethodsAssertGenerator.java:118)
    at eu.stamp_project.dspot.assertgenerator.AssertGenerator.innerAssertionAmplification(AssertGenerator.java:139)
    at eu.stamp_project.dspot.assertgenerator.AssertGenerator.assertionAmplification(AssertGenerator.java:77)
    at eu.stamp_project.dspot.Amplification.assertionsAmplification(Amplification.java:214)
    at eu.stamp_project.dspot.Amplification.amplification(Amplification.java:108)
    at eu.stamp_project.dspot.DSpot._amplify(DSpot.java:274)
    at eu.stamp_project.dspot.DSpot._amplifyTestClass(DSpot.java:266)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
    at eu.stamp_project.dspot.DSpot._amplifyTestClasses(DSpot.java:262)
    at eu.stamp_project.dspot.DSpot.amplifyAllTests(DSpot.java:147)
    at eu.stamp_project.Main.run(Main.java:52)
    at eu.stamp_project.DSpotMojo.execute(DSpotMojo.java:351)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute(MavenCli.java:956)
    at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
    at org.apache.maven.cli.MavenCli.main(MavenCli.java:192)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
[INFO] Could not generate any test with assertions
[INFO] 0 amplified test methods has been selected to be kept. (global: 0)
[INFO] elapsedTime 2172
[WARNING] DSpot could not obtain any amplified test method.
danglotb commented 5 years ago

The problem is that, back in the first day of DSpot, when I used PIT I had to specify the options targetClasses.

If this option was specified, PIT would mutate all the classes in the package groupId.artifactId

So in DSpot, I added the property pitFilterClassesToKeep (the old filter) to specify this value.

However, most users did not notice this property, I was often not specified. It results that PIT fails and a lot of bug report (as issues) was raised here. So I implemented a way to compute on-the-fly this filter based on the AST build by Spoon during the process of DSpot.

Maybe in some recent changes, PIT do not use anymore the groupId.artifactId to target the classes, and thus is less mandatory.

I'm gonna remove this computation on the fly, but let the property since it allows to target a subpackage, which is a feature that some developers would need.

I just tried to amplify twice: once with Jacoco, then amplify the amplified version with PitMutant.

Okay, it should not happen.

However. I'm gonna fix first the issue on the filter, then look at the double amplification.

Would you mind to open a new issue with the maximum information to reproduce your problems?

Thank you.

gibello commented 5 years ago

Done: #825

danglotb commented 5 years ago

You should be able to improve the mutation score using DSpot since I merged #824