search

STAMP-project / dspot

Automatically detect and generate missing assertions for Junit test cases (also known as test amplification)
https://dspot-demo.stamp-project.eu/
GNU Lesser General Public License v3.0
114 stars 28 forks source link

Tests don't pass #126

Closed esauvage closed 6 years ago

esauvage commented 7 years ago

when running the "getting started", tests don't pass. mvn test fails with Failed tests: testRunMockito(fr.inria.diversify.dspot.testrunner.MockitoRunnerTest): expected:<5> but was:<0> test(fr.inria.diversify.dspot.DSpotTest): expected:<1> but was:<2> testResolveDependencies(fr.inria.diversify.dspot.support.MavenDependenciesResolverTest) testFilterTests(fr.inria.diversify.dspot.assertGenerator.RemoveBadTestTest): expected:<1> but was:<4> `Tests in error: ` ` test(fr.inria.diversify.dspot.DSpotMockedTest): java.io.FileNotFoundException: tmpDir/tmp/observations.ser (Aucun fichier ou dossier de ce type)` ` testPitDescartesMode(fr.inria.diversify.mutant.descartes.PitDescartesTest): java.lang.NullPointerException` Tests run: 47, Failures: 4, Errors: 2, Skipped: 0

danglotb commented 7 years ago

Hello @esauvage

Could confirm that all tests pass now? I'll ask to review #128 and merge it to the master branch.

Do not hesitate to open an issue to ask anything ;)

esauvage commented 7 years ago

Hello @danglotb

I'm sorry but no, mvn test don't pass. The error log is too long for CMD (no, that's not a joke). Here is a meaningful extract:

testPit(fr.inria.diversify.mutant.pit.PitTest)  Time elapsed: 0 sec  <<< ERROR!
java.lang.NullPointerException
        at fr.inria.diversify.mutant.pit.PitTest.testPit(PitTest.java:76)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(Framework
Method.java:47)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCal
lable.java:12)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMe
thod.java:44)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMet
hod.java:17)
        at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.
java:26)
        at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.ja
va:27)
        at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRun
ner.java:70)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRun
ner.java:50)
        at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
        at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
        at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provide
r.java:252)
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4
Provider.java:141)
        at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider
.java:112)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(
ReflectionUtils.java:189)
        at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke
(ProviderFactory.java:165)
        at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(Provi
derFactory.java:85)
        at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(Fork
edBooter.java:115)
        at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:
75)

testPitEvosuiteMode(fr.inria.diversify.mutant.pit.PitTest)  Time elapsed: 0 sec
 <<< ERROR!
java.lang.NullPointerException
        at fr.inria.diversify.mutant.pit.PitTest.testPitEvosuiteMode(PitTest.jav
a:35)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(Framework
Method.java:47)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCal
lable.java:12)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMe
thod.java:44)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMet
hod.java:17)
        at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.
java:26)
        at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.ja
va:27)
        at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRun
ner.java:70)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRun
ner.java:50)
        at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
        at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
        at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provide
r.java:252)
        at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4
Provider.java:141)
        at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider
.java:112)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(
ReflectionUtils.java:189)
        at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke
(ProviderFactory.java:165)
        at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(Provi
derFactory.java:85)
        at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(Fork
edBooter.java:115)
        at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:
75)

Running fr.inria.stamp.MainTest
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.001 sec <<< FA
ILURE!
esauvage commented 7 years ago

On a debian, "mvn exec:java -Dexec.mainClass="fr.inria.stamp.Main" -Dexec.args="--example"" don't pass. `[INFO] Scanning for projects... [INFO]
[INFO] ------------------------------------------------------------------------ [INFO] Building dspot 1.0.0 [INFO] ------------------------------------------------------------------------ Downloading: https://stamp-project.github.io/stamp-maven-repository/fr/inria/stamp/descartes/0.1-SNAPSHOT/maven-metadata.xml Downloaded: https://stamp-project.github.io/stamp-maven-repository/fr/inria/stamp/descartes/0.1-SNAPSHOT/maven-metadata.xml (767 B at 0.5 KB/sec) [INFO] [INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ dspot --- 00:00 DEBUG: run maven, phase: [clean, test, -DskipTests], timeout -1 00:10 DEBUG: [INFO] Scanning for projects... [INFO]
[INFO] ------------------------------------------------------------------------ [INFO] Building test-projects 0.0.1-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ example --- [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ example --- [WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent! [INFO] skip non existing resourceDirectory /home/esauvage/dspot/tmpDir/tmp/src/main/resources [INFO] [INFO] --- maven-compiler-plugin:3.2:compile (default-compile) @ example --- [INFO] Changes detected - recompiling the module! [WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent! [INFO] Compiling 1 source file to /home/esauvage/dspot/tmpDir/tmp/target/classes [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ example --- [WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent! [INFO] skip non existing resourceDirectory /home/esauvage/dspot/tmpDir/tmp/src/test/resources [INFO] [INFO] --- maven-compiler-plugin:3.2:testCompile (default-testCompile) @ example --- [INFO] Changes detected - recompiling the module! [WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent! [INFO] Compiling 1 source file to /home/esauvage/dspot/tmpDir/tmp/target/test-classes [INFO] [INFO] --- maven-surefire-plugin:2.17:test (default-test) @ example --- [INFO] Tests are skipped. [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 4.691 s [INFO] Finished at: 2017-07-03T09:20:30+02:00 [INFO] Final Memory: 14M/60M [INFO] ------------------------------------------------------------------------

00:11 DEBUG: timeOut init: 11 00:12 DEBUG: all children process kill (pid: 28834) 00:12 DEBUG: timeOut init: 44 00:12 DEBUG: run maven, phase: [dependency:build-classpath, -Dmdep.outputFile=cp], timeout -1 00:21 DEBUG: timeOut init: 9 00:22 DEBUG: all children process kill (pid: 28834) 00:28 INFO: amplification of example.TestSuiteExample (6 test) 00:28 DEBUG: Error during running test 00:28 WARN: Need a green test suite to run dspot 00:28 DEBUG: total amp test: 0, global: 0 00:28 DEBUG: amp test2 (1/6) 00:28 DEBUG: update test testCriterion 00:29 DEBUG: 1 / 1 test cases failed! 00:29 DEBUG: amp test3 (2/6) 00:29 DEBUG: update test testCriterion 00:29 DEBUG: 1 / 1 test cases failed! 00:29 DEBUG: amp test4 (3/6) 00:29 DEBUG: update test testCriterion 00:29 DEBUG: 1 / 1 test cases failed! 00:29 DEBUG: amp test7 (4/6) 00:29 DEBUG: update test testCriterion 00:29 DEBUG: 1 / 1 test cases failed! 00:29 DEBUG: amp test8 (5/6) 00:29 DEBUG: update test testCriterion 00:29 DEBUG: 1 / 1 test cases failed! 00:29 DEBUG: amp test9 (6/6) 00:30 DEBUG: update test testCriterion 00:30 DEBUG: 1 / 1 test cases failed! 00:30 DEBUG: elapsedTime 2188

======= REPORT ======= Branch Coverage Selector: Initial coverage: 0,00% There is 0 unique path in the original test suite The amplification results with 0 new tests The branch coverage obtained is: NaN% There is 0 new unique path

[WARNING] java.lang.RuntimeException: java.lang.NullPointerException at fr.inria.stamp.Main.runExample(Main.java:106) at fr.inria.stamp.JSAPOptions.parse(JSAPOptions.java:56) at fr.inria.stamp.Main.main(Main.java:21) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:282) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NullPointerException at fr.inria.diversify.dspot.selector.BranchCoverageTestSelector.report(BranchCoverageTestSelector.java:198) at fr.inria.diversify.dspot.DSpot.amplifyTest(DSpot.java:226) at fr.inria.diversify.dspot.DSpot.amplifyTest(DSpot.java:214) at fr.inria.stamp.Main.runExample(Main.java:104) ... 8 more [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 39.196 s [INFO] Finished at: 2017-07-03T09:20:50+02:00 [INFO] Final Memory: 23M/60M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default-cli) on project dspot: An exception occured while executing the Java class. java.lang.NullPointerException -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException e`

danglotb commented 7 years ago

Hello @esauvage ,

Yes, the command line with maven does not work. I think it is because of classloading issue.

I propose you to use following command-lines:

mvn clean package -DskipTests
java -cp target/dspot-1.0.0-jar-with-dependencies.jar fr.inria.stamp.Main --example

which will also run the the example.

danglotb commented 7 years ago

Hello @esauvage

With recent changes into the core of DSpot, thanks to @danzone, I could run all the test on a VM running W10.

Could you update your DSpot's version by pulling the repos and try to run: mvn clean test.

In the other hands, the example still fail using mvn command. I do not know why.

I'm still investigating.

Could you please confirm that your tests pass using maven command?

Cheers.

esauvage commented 7 years ago

Hello,

mvn test don't pass. Well, it almost pass. It just can't delete files at the end, and that's enough to mark it "Failed". I tried in administrative mode, that doesn't work either. The said files don't exist on the filesystem, so it ain't surprising.

By the way, there is a name problem : the test script asks for "mvn.bat", which does not exist. The good name is "mvn.cmd".

Here is the part of the output:

Results :

Tests in error:
testCopyMultipleModuleProject(fr.inria.diversify.dspot.DSpotMultiplePomTest): java.io.IOException: Unable to delete file: tmpDir\tmp\log\logThread-9_1500967355283
test(fr.inria.diversify.dspot.DSpotTest): Unable to delete file: tmpDir\tmp\log\logThread-9_1500967355283
testInheritanceMethod(fr.inria.diversify.dspot.inheritance.InheritanceTest): java.io.IOException: Unable to delete file: tmpDir\tmp\log\logThread-9_1500967355283
test(fr.inria.diversify.dspot.ProjectJSONTest)

Tests run: 56, Failures: 0, Errors: 4, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 03:31 min
[INFO] Finished at: 2017-07-25T09:25:14+02:00
[INFO] Final Memory: 19M/257M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test (default-test) on project dspot: There are test failures.

Regards,

danglotb commented 7 years ago

Hello @esauvage

Could you please send me the full execution trace of maven (by email)?

Thanks.

danglotb commented 7 years ago

Hello @esauvage

Could you please update your version of dspot and retry? Again on my VM every test pass succesfully.

Cheers.

esauvage commented 7 years ago

Hello,

I had the same problem from a fresh newk git clone either.

Regards,

Le 26-07-2017 12:23, Benjamin DANGLOT a écrit :

Hello @esauvage [1]

Could you please update your version of dspot and retry? Again on my VM every test pass succesfully.

Cheers.

-- You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub [2], or mute the thread [3].

Links:

[1] https://github.com/esauvage [2] https://github.com/STAMP-project/dspot/issues/126#issuecomment-318013601 [3] https://github.com/notifications/unsubscribe-auth/AEJKqkd86hY7xbKgCqsB4RCw-9LJabMqks5sRxOigaJpZM4N7Rpl

nicolabertazzo commented 7 years ago

clean_package.txt

danglotb commented 6 years ago

Hi, I'm closing this thread due to inactivity. Do not hesitate to reopen it or a new issue.