vmassol
commented
6 years ago Seems the error is java.lang.ClassCastException: org.apache.xerces.dom.DocumentImpl cannot be cast to com.sun.org.apache.xerces.internal.dom.Doc.
Closed vmassol closed 6 years ago
vmassol
commented
6 years ago Seems the error is java.lang.ClassCastException: org.apache.xerces.dom.DocumentImpl cannot be cast to com.sun.org.apache.xerces.internal.dom.Doc.
danglotb
commented
6 years ago Hi,
The problem happens when DSpot instruments the code to obtain values on the program, to build assertions.
The test method is as follow:
@org.junit.Test(timeout = 10000)
public void testGetDomDocument_withlog0() {
org.w3c.dom.Document o_testGetDomDocument__1 = org.xwiki.xml.XMLUtils.createDOMDocument();
fr.inria.diversify.compare.ObjectLog.log(o_testGetDomDocument__1, "o_testGetDomDocument__1", "testGetDomDocument__1");
}But, the object receive by the method ObjectLog#log, i.e. o_testGetDomDocument__1 is received as a com.sun.org.apache.xerces.internal.dom.Doc and I don't know why.
I tried to isolate the execution of the test method, using a specific and clean classloader, but with no success.
monperrus
commented
6 years ago @danglotb could you add the generated tests here?
danglotb
commented
6 years ago here find a git gist containing the both amplified test.
You will find also a json file, that report killed mutants.
Note that the amplification has been done using only A-amplificaiton , i.e. generation of assertions.
vmassol
commented
6 years ago Thanks @danglotb
Note that I've tested it and it doesn't increase the test coverage unfortunately which is one of the main goal of DSpot I think. So we need some more I think so that we can demonstrate an increase.
monperrus
commented
6 years ago Dspot can increase the coverage and/or the mutation score, it depends on the configuration.
In A-amplification it only increases the mutation score (because it only adds new assertions).
@danglotb could you make a run in I-amplification?
vmassol
commented
6 years ago is it normal that in one of the 2 created tests we have the following line duplicated:
Assert.assertEquals("a < a' && a' < a" => a < a"", escapedText);?
vmassol
commented
6 years ago Ok I've looked at the generated tests. They add an explicit assert for the returned content. It's kind of interesting.
Here are some ideas for strategies to use the generated code (in a maven context).
Strategy 1:
Strategy 2:
monperrus
commented
6 years ago is it normal that in one of the 2 created tests we have the following line duplicated:
It's not normal, but it's not a huge problem (both assertions fail or pass :-)
An important future feature of Dspot is minimization of generated tests (#54 , help wanted), it would remove unnecessary or duplicated code.
monperrus
commented
6 years ago Very interesting suggestions.
Let's discuss Strategy 1.
AFAIU, the title of the strategy would be "push DSpot tests in a special test folder (eg src/generated/test)". Correct?
What I like here, is that a new run on Dspot would be made on top a previous run, in order to see if new mutants can be killed. Correct?
danglotb
commented
6 years ago is it normal that in one of the 2 created tests we have the following line duplicated:
It is explained because DSpot generates two sequences of assertions: one at the begin, one at the end.
This is done in order to detect the change between initial state and tested code.
However, DSpot should verify that between the two sequences, there is something else than assertion.
In this case, we have calls to the contains. I agree that if the methods between the two sequences are pure, we can skip the last sequences.
danglotb
commented
6 years ago I ran DSpot on XWiki-commons-xml. To do this, just add to your properties file the following line: excludedTestCases=testGetDomDocument.
I'm waiting for #265 to be merged, I'll then release again DSpot.
@danglotb could you make a run in I-amplification?
I tried, but it seems that the I-Amplification does not help on XMLUtilsTests : (
Thank you.
monperrus
commented
6 years ago @danglotb What's the configuration to use for xwiki-commons-xml now that we can exclude a test case?
danglotb
commented
6 years ago Retrieve the new release
To exclude test cases, just add the excludedTestCases property:
project=../../
targetModule=xwiki-commons-core/xwiki-commons-xml/
src=src/main/java/
testSrc=src/test/java
javaVersion=8
outputDirectory=output
filter=org.xwiki.*
excludedTestCases=testGetDomDocumentThe command line would be:
java -cp java -cp dspot-1.0.2-jar-with-dependencies.jar fr.inria.stamp.Main --path-to-properties dspot.properties -t org.xwiki.xml.XMLUtilsTest
and you should obtain something like:
======= REPORT =======
PitMutantScoreSelector:
The original test suite kills 861 mutants
The amplification results with 2 new tests
it kills 16 more mutantsJust check the output folder, you will have the output of DSpot.
danglotb
commented
6 years ago I continue on this thread.
I tried to amplify several modules of XWiki-commons. I had one success for now on XWiki-commons-job.
The class org.xwiki.job.internal.DefaultRequestTest is successfully amplified.
======= REPORT =======
PitMutantScoreSelector:
The original test suite kills 798 mutants
The amplification results with 1 new tests
it kills 17 more mutantsYou can find the amplified test cases here
I noticed that for a lot of tests, even PIT is not able to run XWiki. There is an issue with the ConverterManager (see for instance org.xwiki.properties.internal.converter.ArrayListConverterTest) which the old version of the annotation @ComponentList if I am not wrong.
It would be great to know what is missing in the configuration to be able to run it with PIT. And then, we could try the amplification.
If you want to try yourself, find the used property file:
project=../../
targetModule=xwiki-commons-core/xwiki-commons-xml/
src=src/main/java/
testSrc=src/test/java
testResources=src/test/java/
javaVersion=8
outputDirectory=output
filter=org.xwiki.*
excludedClasses=org.xwiki.job.internal.xstream.SafeXStreamTest,org.xwiki.job.internal.DefaultJobStatusStoreTestAnd the command line:
java -cp java -cp dspot-1.0.2-jar-with-dependencies.jar fr.inria.stamp.Main --path-to-properties dspot.properties -t -t org.xwiki.job.internal.DefaultRequestTest
In this case, I-Amplification is also useless.
vmassol
commented
6 years ago It would be much nicer if we could find an example where I-Amplification works. Trying to reproduce your examples now. I'd really like to demonstrate an example where DSpot increases the test coverage :)
vmassol
commented
6 years ago Ok commons-xml works for me too with:
project=../../
targetModule=xwiki-commons-core/xwiki-commons-xml/
src=src/main/java/
testSrc=src/test/java
javaVersion=8
outputDirectory=target/dspot-output
filter=org.xwiki.*
tmpDir=target/dspot
excludedTestCases=createDomDocumentNote: that last week I've renamed the test names so I had to update the excludedTestCases property value.
vmassol
commented
6 years ago @danglotb I noticed that in your dspot properties file for commons-job you've used targetModule=xwiki-commons-core/xwiki-commons-xml/ which doesn't seem correct. Does it mean that this property is not used? :)
vmassol
commented
6 years ago ok I can answer it since I've now been able to run it (I tried both with the error and without it). It matter and your properties above just has a copy paste error :)
danglotb
commented
6 years ago Hi @vmassol
It is a bad copy / paste, sorry for that.
vmassol
commented
6 years ago ok I've now analyzed the result of the amplification of xwiki-commons-job for org.xwiki.job.internal.DefaultRequestTest. It's basically asserting the exact values instead of comparing the 2 instances. That's a small thing but interesting nonetheless. But indeed it doesn't increase the test coverage.
danglotb
commented
6 years ago Hi, I just release DSpot with our fixes.
To totally isolate the Test runner, you have to use the flag -useReflection in the CLI.
This will force DSpot to use the ReflectiveTestRunner which use the reflection to be totally isolated from the runtime of DSpot.
danglotb
commented
6 years ago I just run DSpot on other test classes of XWiki-common-xml
Result are much more better:
on org.xwiki.xml.html.HTMLUtilsTest, I used the following options:
-p dspot.properties --verbose -i 1 -t org.xwiki.xml.html.HTMLUtilsTest -a TestDataMutator
I obtained the following results:
======= REPORT =======
PitMutantScoreSelector:
The original test suite kills 880 mutants
The amplification results with 3 new tests
it kills 5 more mutantsYou can find the amplified test class and the json file here.
For the demo, is suggest to tune the options (test selection / amplifier / pre-computed mutation score) to speed the process.
vmassol
commented
6 years ago To totally isolate the Test runner, you have to use the flag -useReflection in the CLI.
IMO this should be the default since you want to isolate by default, no?
vmassol
commented
6 years ago Result are much more better:
I still need to analyze the results but they don't seem to increase the test coverage though.
vmassol
commented
6 years ago FYI, if you wish to test coverage difference:
mvn clean verify -Pquality -Dxwiki.jacoco.instructionRatio=1.000 once and note down the coverage % number (0.6026 for me, i.e. 60.26%).java -cp /Users/vmassol/dev/dspot/dspot/target/dspot-1.0.2-SNAPSHOT-jar-with-dependencies.jar fr.inria.stamp.Main --path-to-properties dspot.properties --verbose -i 1 -t org.xwiki.xml.html.HTMLUtilsTest -a TestDataMutator <plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<version>3.0.0</version>
<executions>
<execution>
<id>add-test-source</id>
<phase>generate-test-sources</phase>
<goals>
<goal>add-test-source</goal>
</goals>
<configuration>
<sources>
<source>target/dspot-output</source>
</sources>
</configuration>
</execution>
</executions>
</plugin>Edit the source location to match where dspot tests are generated
mvn clean verify -Pquality -Dxwiki.jacoco.instructionRatio=1.000 and note down the new coverage % number. Still 0.6026 for me, even though the newly generated tests are also executed.danglotb
commented
6 years ago The fact is that we used the default selector which is based on PIT.
And so, the amplified does not (necessarily) improve the coverage but inprove the mutation score.
That is to say that the amplified test suite are a better potential to detect fault.
If you want to increase the coverage, we should use BranchCoverageSelector (since the Jacoco selector is not (yet) compatible with the useReflection option)
IMO this should be the default since you want to isolate by default, no?
I don't know, because this mode maybe unstable and is not compatible with all other options.
But at the end, yes, it should be the default mode.
vmassol
commented
6 years ago Note that the reason I'm interested in getting additional test coverage is because I find that this is the killer feature of DSpot, and on the test quality area, there's already Descartes doing this too. So it would be a unique feature of DSpot. WDYT?
monperrus
commented
6 years ago Your strategies can be explained here: https://github.com/STAMP-project/dspot/wiki/Using-DSpot-in-CI
And the discussion about the strategies is appropriate in #191
monperrus
commented
6 years ago otherwise can we close this issue?
vmassol
commented
6 years ago Works now. Example with java -cp /Users/vmassol/dev/dspot/dspot/target/dspot-1.0.4-SNAPSHOT-jar-with-dependencies.jar fr.inria.stamp.Main --path-to-properties dspot.properties --verbose --useReflection no longer generates the classcast error.
Created #284 to have it by default.
Same setup as with #256 but on xwiki-commons-xml.
I got: