DSpot

DSpot is a tool that generates missing assertions in JUnit tests. DSpot takes as input a Java project with an existing test suite. DSpot generates new test cases from existing ones and write them on disk. DSpot supports Java projects built with Maven and Gradle (see the --automatic-builder option)

DSpot ecosystem:

• Jenkins plugin
• Eclipse plugin
• IntelliJ plugin

Getting started

Demo site

Meet our demonstration website at https://dspot-demo.stamp-project.eu/.

Prerequisites

You need Java and Maven. The environment variable JAVA_HOME must point to a valid JDK installation (and not a JRE). DSpot uses the environment variable MAVEN_HOME, ensure that this variable points to your maven installation. Example:

export MAVEN_HOME=path/to/maven/

Releases

We advise you to start by downloading the latest release, see https://github.com/STAMP-project/dspot/releases.

Usage

First Tutorial

After having downloaded DSpot (see the previous section), you can run the provided example by running eu.stamp_project.Main from your IDE, or with

java -jar target/dspot-LATEST-jar-with-dependencies.jar --example

replacing LATEST by the latest version of DSpot, e.g. 2.2.1 would give dspot-2.2.1-jar-with-dependencies.jar

Given a test case as input, with --amplifiers=None, DSpot only adds assertions in the test, for example:

@Test
void test() {
  Tacos tacos = new Tacos();
  Benjamin benjamin = new Benjamin();
  benjamin.eat(tacos);
  assertFalse(benjamin.isHungry());
+  assertTrue(benjamin.isHappy());  // new assertion
}

With some amplifiers, eg --amplifiers=AllAmplifiers, DSpot modifies the setup of the input test and adds assertions in your code, for example:

@Test
void test() {
  Tacos tacos = new Tacos();
  Benjamin benjamin = new Benjamin();
-  benjamin.eat(tacos); // removed method call
-  assertFalse(benjamin.isHungry());
+  assertTrue(benjamin.isHungry());
+  assertFalse(benjamin.isHappy()); // new assertion
}

The modified test is lastly given to a selector, eg --test-selector=PitMutantScoreSelector, that determines if it improves the test suite and should be kept.

The amplification process is illustrated in the figure below. Note that amplified tests are further modified by subsequent iterations determined by for example --iteration=10.

When an amplification is successful, DSpot outputs the improvement on the console and the result of the amplification (the new tests) are written to the output folder specified by configuration property outputDirectory (default to ./target/dspot/output/).

Initial instruction coverage: 30 / 34
88.24%
Amplification results with 5 amplified tests.
Amplified instruction coverage: 34 / 34
100.00%

Want to know more? Read the full articles:

• Automatic Test Improvement with DSpot: a Study with Ten Mature Open-Source Projects (doi:10.1007/s10664-019-09692-y)
• An Approach and Benchmark to Detect Behavioral Changes of Commits in Continuous Integration (doi:10.1007/s10664-019-09794-7)

Command Line Usage

You can then execute DSpot by using:

java -jar /path/to/dspot-LATEST-jar-with-dependencies.jar --absolute-path-to-project-root <path>

Amplify a specific test class

java -jar /path/to/dspot-*-jar-with-dependencies.jar eu.stamp_project.Main --absolute-path-to-project-root <path> --test my.package.TestClass

Amplify specific test classes according to a regex

java -jar /path/to/dspot-LATEST-jar-with-dependencies.jar --absolute-path-to-project-root <path> --test my.package.*
java -jar /path/to/dspot-LATEST-jar-with-dependencies.jar --absolute-path-to-project-root <path> --test my.package.Example*

Amplify a specific test method from a specific test class

java -jar /path/to/dspot-LATEST-jar-with-dependencies.jar --absolute-path-to-project-root <path> --test my.package.TestClass --cases testMethod

Command line options

For options that take list, the used separator is a comma ,, whatever the platform you use.

Maven plugin usage

You can execute DSpot using the maven plugin. For more details, see the dedicated README.

Each command line options is translated into an option for the maven plugin. You must prefix each of them with -D. Examples:

* `--test my.package.MyTestClass1:my.package.MyTestClass2` gives `-Dtest=my.package.MyTestClass1,my.package.MyTestClass2`
* `--output-path output` gives `-Doutput-path=output` 

Configuration

Amplifiers (-a | --amplifiers)

By default, DSpot uses no amplifier because the simplest amplification that can be done is the generation of assertions on existing tests, i.e. it will improve the oracle and the potential of the test suite to capture bugs.

However, DSpot provide different kind of Amplifier:

• StringLiteralAmplifier: modifies string literals: remove, add and replace one random char, generate random string and empty string. It also replaces the existing string by the system path separator (e.g. : on Linux) and the system line separator (e.g. \n on Linux).
• NumberLiteralAmplifier: modifies number literals: replace by boundaries (e.g. Integer_MAX_VALUE for int), increment and decrement values
• CharLiteralAmplifier: modifies char literals: replace by random char and special chars: '\0' and `. It also replaces the existing char by the previous and next one, _i.e._ it computescodechar + 1andcodechar - 1. For example, we haveb, it resuls withb + 1 = candb - 1 = a`
• BooleanLiteralAmplifier: modifies boolean literals: negate the value
• ArrayLiteralAmplifier: modifies array literals: remove and add element in literal, replace literal with empty literal and null
• AllLiteralAmplifiers: combines all literals amplifiers, i.e. StringLiteralAmplifier, NumberLiteralAmplifier, CharLiteralAmplifier, BooleanLiteralAmplifier and ArrayLiteralAmplifier
• MethodDuplicationAmplifier: duplicates an existing method call
• MethodRemove: removes an existing method call
• MethodAdderOnExistingObjectsAmplifier: adds a method call, and generate required parameter
• ReplacementAmplifier: replaces a local variable by a generated one
• FastLiteralAmplifier: a faster amplifier for the literals
• NullifierAmplifier: replaces value with null
• ArrayAmplifier: replaces value of arrays
• ReturnValueAmplifier: creates objects based on the returned value by existing method call
• None: do nothing

If you only want to use these amplifiers and not generate assertions, use --only-input-amplification.

Test Selectors (-s | --test-selector | --test-criterion)

In DSpot, test selectors can be seen as a fitness: it measures the quality of amplified, and keeps only amplified tests that are worthy according to this selector.

The The default selector is PitMutantScoreSelector. This selector is based on PIT, which is a tool to computation mutation analysis. DSpot will keep only tests that increase the mutation score.

Following the list of available test selectors:

• PitMutantScoreSelector: uses PIT to computes the mutation score, and selects amplified tests that kill mutants that was not kill by the original test suite.
• JacocoCoverageSelector: uses JaCoCo to compute instruction coverage and executed paths (the order matters). Selects test that increase the coverage and has unique executed path.
• ExtendedCoverageSelector: uses JaCoCo to compute instruction coverage. Selects tests that cover more instructions than the original test suite on any line of the code under test.
• TakeAllSelector: keeps all amplified tests not matter the quality.
• ChangeDetectorSelector: runs against a second version of the same program, and selects amplified tests that fail. This selector selects only amplified test that are able to show a difference of a behavior betweeen two versions of the same program.

Input Ampl Distributor

In DSpot, the Input Ampl Distributor is a way to select the amplified test methods after the input amplification. It allows to keep interesting and discard unwanted amplified test method.

For now, there are three implementations of the Input Ampl Distributor:

• RandomInputAmplDistributor: This distributor selects randonly amplified test methods.
• TextualDistanceInputAmplDistributor: This distributor selects by maximize their distance of string representation among all the input amplified test methods. The number of amplified selected test methods is specified by the command line option --max-test-amplified.
• SimpleInputAmplDistributor: This distributor selects a fair number of amplified test method per Amplifier per test methods, if possible. The total budget is specified by the command line option `--max-test-amplified, and is the total number of amplified test methods to keep, i.e. it will be divide by the number of Amplifiers and by the number of test methods to be amplified. Example: We have 2 Amplifiers. We apply them to 2 test methods. For each test methods, amplifiers generate 4 new test methods, totally 8 amplified test methods. If the budget is 6, it will select: 3 amplified test methods per amplifier, and 2 for one test method and 2 for the other.

Supported Test Frameworks

DSpot supports:

• JUnit3
• JUnit4
• JUnit5

and

• Google Truth assertions

However, DSpot detects the used test framework at test class level.

Please, do not amplify test classes that mix test frameworks (test methods in JUnit4 and JUnit5 within the same test class.)

If you have such test class, please amplify the different test framework separately.

Supported JDKs & Build Tools

DSpot runs with the following Java versions:

• Java 1.8
• Java 11
• Java 16 (used as JDK, not tested if it can support Java 16 language features)

It can amplify tests of projects built with:

• Maven 3.6
• Gradle 7

Minimizing and Prettifying DSpot Amplified Test Cases

With the DSpot Prettifier you can remove redundant assertions from your amplified test cases & give them more expressive names with the help of code2vec and context2vec.

License and Contributions

DSpot is licensed under LGPLv3. Contributors and pull requests are welcome :-). For more information on development, see the dedicated README-developers.md

Acknowledgement

Dspot is funded by EU H2020 research project STAMP.