search

STAMP-project / dspot

Automatically detect and generate missing assertions for Junit test cases (also known as test amplification)
https://dspot-demo.stamp-project.eu/
GNU Lesser General Public License v3.0
114 stars 28 forks source link

DSpot on Defects4j dataset #1006

Closed firhard closed 6 months ago

firhard commented 2 years ago

Characteristics

Description

Hi,

I am currently trying to run the DSpot test amplification tool on the Defects4j dataset. I have an issue where the bugs inside Compress, Gson, and Jsoup libraries are giving me an error that looks like this when using DSpot to amplify the test suite:

2022-05-09 13:32:11,894 WARN eu.stamp_project.dspot.common.report.error.ErrorReport - DSpot encountered some errors during amplification.
2022-05-09 13:32:11,897 WARN eu.stamp_project.dspot.common.report.error.ErrorReport - DSpot encountered 1 error(s) during amplification.
Something bad happened when DSpot tried to computed the original mutation score.
This is usually due to the value of the command line option --pit-filter-classes-to-keep.
Specify the filter of classes to keep used by PIT. This allow you restrict the scope of the mutation done by PIT. If this is not specified, DSpot will try to build on the fly a filter that takes into account the largest number of classes, e.g. the topest package. 
This is can be also due to a specific configuration of your test suite. If any test fails,
PIT (and so DSpot) won't be able to be executed. Please, check your environment variables,
external files, etc. You can use --excluded-classes and --excluded-test-cases
 to exclude respectively specific test classes ans test cases.
java.lang.RuntimeException: java.lang.RuntimeException: Maven build failed! Enable verbose mode for more information (--verbose)
    at eu.stamp_project.dspot.common.automaticbuilder.maven.MavenAutomaticBuilder.runPit(MavenAutomaticBuilder.java:156)
    at eu.stamp_project.dspot.common.automaticbuilder.maven.MavenAutomaticBuilder.runPit(MavenAutomaticBuilder.java:162)
    at eu.stamp_project.dspot.selector.PitMutantScoreSelector.init(PitMutantScoreSelector.java:109)
    at eu.stamp_project.dspot.common.configuration.AmplificationSetup.firstSelectorSetup(AmplificationSetup.java:129)
    at eu.stamp_project.dspot.DSpot.setupSelector(DSpot.java:86)
    at eu.stamp_project.dspot.DSpot.amplification(DSpot.java:70)
    at eu.stamp_project.dspot.DSpot.run(DSpot.java:59)
    at eu.stamp_project.Main.main(Main.java:20)

This issue does not happen for other libraries in Defects4j. Is there anything that I'm currently missing?

Steps to reproduce

Defects4j command: defects4j checkout -p Time -v 1b -w time_1_fixed I ran the DSpot command with the test files that need to be amplified. DSpot command: java -jar dspot-3.2.0-jar-with-dependencies.jar --absolute-path-to-project-root defects4j/time_1_fixed --test org.joda.time.TestPartial_Constructors -v --pit-filter-classes-to-keep org.joda.time.field.UnsupportedDurationField --gregor-mode=true --pit-version=1.4.10

danglotb commented 2 years ago

Hello @firhard

Thank you for the report.

I would say that the problem is the value you specified to the --pit-filter-classes-to-keep, i.e. org.joda.time.field.UnsupportedDurationField.

Here, you should specify the topest package of the targeted project, e.g. org.joda.time.*.

Please, enable the verbose mode by adding the flag -verbose to see the maven command line used to compute the original mutation score.

I hope this is help you, otherwise, please provide the trace with the verbose mode enabled.

Best,

firhard commented 2 years ago

Hi @danglotb,

Thank you for replying and sorry for taking some time to get back to you. The reason that I have the --pit-filter-classes-to-keep is because I would want it to only generate tests based on killing the mutants inside of org.joda.time.field.UnsupportedDurationField class. I also tried without the --pit-filter-classes-to-keep but it also gives me this error. In Defects4j, it has the d4j.classes.modified property (shown below) of which files being modified on the buggy version and I would only want to amplify the test suite that could help in revealing more mutants on the specific class.

[ERROR] Failed to execute goal org.pitest:pitest-maven:1.4.10:mutationCoverage (default-cli) on project joda-time: Execution default-cli of goal org.pitest:pitest-maven:1.4.10:mutationCoverage failed: Coverage generation minion exited abnormally. Please check the classpath and/or enable test plguin for used test tool.

#File automatically generated by Defects4J
d4j.bug.id=1
d4j.classes.modified=org.joda.time.field.UnsupportedDurationField,org.joda.time.Partial
d4j.dir.src.classes=src/main/java
d4j.dir.src.tests=src/test/java
d4j.project.id=Time
d4j.tests.trigger=org.joda.time.TestPartial_Constructors::testConstructorEx7_TypeArray_intArray

I am currently facing another issue with some other Defects4j datasets (e.g. Lang 20 buggy version), where related to the observer.ser not being generated.

Something bad happened during assertion amplification
java.lang.RuntimeException: java.io.FileNotFoundException: target/dspot/observations.ser (No such file or directory)
        at eu.stamp_project.dspot.assertiongenerator.assertiongenerator.methodreconstructor.observer.testwithloggenerator.objectlogsyntaxbuilder_constructs.ObjectLog.load(ObjectLog.java:353)
        at eu.stamp_project.dspot.assertiongenerator.assertiongenerator.methodreconstructor.observer.testwithloggenerator.objectlogsyntaxbuilder_constructs.ObjectLog.getObservations(ObjectLog.java:314)
        at eu.stamp_project.dspot.assertiongenerator.assertiongenerator.methodreconstructor.Observer.compileRunTests(Observer.java:130)
        at eu.stamp_project.dspot.assertiongenerator.assertiongenerator.methodreconstructor.Observer.getObservations(Observer.java:79)
        at eu.stamp_project.dspot.assertiongenerator.assertiongenerator.MethodReconstructor.addAssertions(MethodReconstructor.java:84)
        at eu.stamp_project.dspot.assertiongenerator.AssertionGenerator.addAssertionsOnPassingTests(AssertionGenerator.java:202)
        at eu.stamp_project.dspot.assertiongenerator.AssertionGenerator.assertPassingAndFailingTests(AssertionGenerator.java:187)
        at eu.stamp_project.dspot.assertiongenerator.AssertionGenerator.assertionAmplification(AssertionGenerator.java:91)
        at eu.stamp_project.dspot.assertiongenerator.AssertionGenerator.removeAndAmplifyAssertions(AssertionGenerator.java:62)
        at eu.stamp_project.dspot.DSpot.assertionAmplification(DSpot.java:202)
        at eu.stamp_project.dspot.DSpot.onlyAssertionGeneration(DSpot.java:98)
        at eu.stamp_project.dspot.DSpot.amplification(DSpot.java:73)
        at eu.stamp_project.dspot.DSpot.run(DSpot.java:59)
        at eu.stamp_project.Main.main(Main.java:20)
Caused by: java.io.FileNotFoundException: target/dspot/observations.ser (No such file or directory)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at eu.stamp_project.dspot.assertiongenerator.assertiongenerator.methodreconstructor.observer.testwithloggenerator.objectlogsyntaxbuilder_constructs.ObjectLog.load(ObjectLog.java:345)
        ... 13 more

It happens in multiple other bugs in Defects4j. I've also attached the --verbose of this issue. dspot.txt

danglotb commented 2 years ago

Hello,

Sorry for the delay.

It seems that the problem comes from the fact that PIT cannot find the test classes.

From your logs, we can see the following:

5:19:04 PM PIT >> INFO : Verbose logging is disabled. If you encounter an problem please enable it before reporting an issue.
5:19:04 PM PIT >> INFO : Sending 1 test classes to minion
5:19:04 PM PIT >> INFO : Sent tests to minion
5:19:05 PM PIT >> INFO : MINION : 5:19:04 PM PIT >> INFO : Checking environment

5:19:05 PM PIT >> INFO : MINION : 5:19:05 PM PIT >> INFO : Found  0 tests

5:19:05 PM PIT >> INFO : MINION : 5:19:05 PM PIT >> INFO : Dependency analysis reduced number of potential tests by 0

5:19:05 PM PIT >> INFO : MINION : 5:19:05 PM PIT >> INFO : 0 tests received

5:19:05 PM PIT >> INFO : Calculated coverage in 0 seconds.
5:19:05 PM PIT >> INFO : Created  1 mutation test units
5:19:05 PM PIT >> INFO : Completed in 0 seconds
/================================================================================
- Timings
================================================================================
> scan classpath : < 1 second
> coverage and dependency analysis : < 1 second
> build mutation tests : < 1 second
> run mutation analysis : < 1 second
--------------------------------------------------------------------------------
> Total  : < 1 second
--------------------------------------------------------------------------------
================================================================================
- Statistics
================================================================================
>> Generated 78 mutations Killed 0 (0%)
>> Ran 0 tests (0 tests per mutation)

In particular: 5:19:05 PM PIT >> INFO : MINION : 5:19:05 PM PIT >> INFO : Found 0 tests and >> Ran 0 tests (0 tests per mutation).

I think that there is a clash between the PIT versions and options used in DSpot. The options used in DSpot might be outdated. I will try to look into this ASAP.

In the meantime, you can try another selector such as Jacoco-based coverage:

--test-selector JacocoCoverageSelector

I will come back to you ASAP, sorry again for the delay and any inconvenience.

firhard commented 2 years ago

Hi @danglotb,

Thank you for updating me about the test-selector parameter. I'm currently using MAJOR mutation testing tool to calculate the number of mutants that it has killed. Is there any reason why the JacocoCoverageSelector could kill more mutants than the PitMutantScoreSelector?

danglotb commented 2 years ago

Hello,

Is there any reason why the JacocoCoverageSelector could kill more mutants than the PitMutantScoreSelector?

Do you mean that the amplified tests obtained with the JacocoCoverageSelector obtain a better mutation score with MAJOR than the amplified tests obtained using the PitMutantScoreSelector?

If so, it might be explained by two reasons:

  1. the mutants generated by MAJOR are different than the ones generated by PIT, therefore the amplified test obtained by the Jacoco selector can be better regarding the mutations of MAJOR.
  2. PitMutantScoreSelector is more difficult to be executed (it requires some tuning on the parameters), then the DSpot's execution might be flawed and the resulting amplified tests might be of lower quality.