iFixFlakies

This plugin automatically generates patches for order-dependent tests by finding and utilizing helper test methods in the test suite. This plugin builds upon code from iDFlakies (https://github.com/idflakies/iDFlakies), and relies on detected order-dependent tests that iDFlakies finds.

More details about iFixFlakies can be found in its paper and website.

Quickstart

First, follow the instructions for iDFlakies to detect order-dependent tests and obtain passing and failing orders for them. Specifically, you will need the .dtfixingtools directory generated by iDFlakies.

Next, build the iFixFlakies plugin within this repository (i.e., running mvn install inside this project's directory). After building the iFixFlakies plugin, you will then need to add it to the Maven project you ran iDFlakies on. (If you had modified the pom.xml to run iDFlakies, you will notice the two modifications are quite similar. You will need to change the existing modification for iDFlakies to be like this modification for iFixFlakies.)

Automatically setting up the pom.xml for iFixFlakies

Run the following command to automatically setup the pom.xml for iFixFlakies. Note that if you made changes to the pom.xml for iDFlakies you will first need to remove those changes before you run the following command.

bash pom-modify/modify-project.sh path_to_maven_project

Manually setting up the pom.xml for iFixFlakies

Copy the following plugin into the Maven project's pom.xml. You do not need to perform this step if you have already completed the instructions in Automatically setting up the pom.xml for iFixFlakies.

<build>
    ...
    <plugins>
        ...
        <plugin>
            <groupId>edu.illinois.cs</groupId>
            <artifactId>testrunner-maven-plugin</artifactId>
            <version>1.2</version>
            <dependencies>
                <dependency>
                    <groupId>edu.illinois.cs</groupId>
                    <artifactId>ifixflakies</artifactId>
                    <version>1.0.0-SNAPSHOT</version>
                </dependency>
            </dependencies>
            <configuration>
                <className>edu.illinois.cs.dt.tools.fixer.CleanerFixerPlugin</className>
            </configuration>
        </plugin>
    </plugins>
</build>

Common use cases of iFixFlakies

Finding one patch

After the pom.xml is setup for the iFixFlakies plugin, run the following command on the Maven project:

mvn testrunner:testplugin \
  -Ddt.minimizer.flaky.list=${json_file} \
  -Ddt.minimizer.original.order=${original_order_file} \
  -Ddt.minimizer.use.original.order=true \
  -Ddt.minimizer.dependent.test=${full_test_name} \
  -Ddiagnosis.run_detection=false \
  -Ddt.find_all=false

-Ddt.minimizer.flaky.list: absolute path to a flaky-list.json file generated by iDFlakies
-Ddt.minimizer.original.order: absolute path to a file containing the fully-qualified names of the tests in the original order of the test suite (this file is automatically generated by iDFlakies as original-order), each line contains one test name
-Ddt.minimizer.use.original.order: controls whether iFixFlakies should use the original order file that is passed in or if iFixFlakies should regenerate an original order file
-Ddt.minimizer.dependent.test: if the dt.minimizer.flaky.list contains more than one OD test, then this argument tells iFixFlakies which specific test to generate a patch for (as opposed to trying to generate patches for all OD tests)
-Ddiagnosis.run_detection: controls whether iFixFlakies should run iDFlakies to first detect OD tests before trying to fix the OD tests. Use the options of iDFlakies to configure the detection of OD tests
-Ddt.find_all: controls whether iFixFlakies should find all possible patches or not

Finding all polluters and cleaners

After the pom.xml is setup for the iFixFlakies plugin, run the following command on the Maven project:

mvn testrunner:testplugin \
  -Ddt.minimizer.flaky.list=${json_file} \
  -Ddt.minimizer.original.order=${original_order_file} \
  -Ddt.minimizer.use.original.order=true \
  -Ddt.minimizer.dependent.test=${full_test_name} \
  -Ddiagnosis.run_detection=false \
  -Dtestplugin.className=edu.illinois.cs.dt.tools.minimizer.MinimizerPlugin \
  -Ddt.minimizer.polluters.one_by_one=true

All other options besides the following are described earlier.

-Dtestplugin.className: controls which plugin testrunner should use. By default, iFixFlakies uses edu.illinois.cs.dt.tools.fixer.CleanerFixerPlugin to generate patches but for finding polluters and cleaners, iFixFlakies only needs to use edu.illinois.cs.dt.tools.minimizer.MinimizerPlugin
-Ddt.minimizer.polluters.one_by_one: controls whether iFixFlakies should use delta debugging to find all polluters and cleaners or whether it should run every test before the OD test to find polluters and all tests inbetween polluters and victims to find cleaners

Cite

If you use iFixFlakies, please cite our corresponding ESEC/FSE paper:

@inproceedings{ShiETAL2019FSE,
    author    = {Shi, August and Lam, Wing and Oei, Reed and Xie, Tao and Marinov, Darko},
    title     = {{iFixFlakies}: {A} Framework for Automatically Fixing Order-Dependent Flaky Tests},
    booktitle = {ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering},
    pages     = {545--555},
    year      = {2019},
}

TestingResearchIllinois / iFixFlakies

readme