RepairThemAll Experiment

This repository contains the raw results of the execution of 11 repair tools on 5 bug benchmarks presented in the following paper:

@inproceedings{RepairThemAll2019,
  author    = {Thomas Durieux and Fernanda Madeiral and Matias Martinez and Rui Abreu},
  title     = {{Empirical Review of Java Program Repair Tools: A Large-Scale Experiment on 2,141 Bugs and 23,551 Repair Attempts}},
  booktitle = {Proceedings of the 27th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE '19)},
  year      = {2019},
  url       = {https://arxiv.org/abs/1905.11973}
}

Benchmark:

• Bears
• Bugs.jar
• Defects4J version 1.4.0
• IntroClassJava
• QuixBugs

The execution framework that has been used is available at: https://github.com/program-repair/RepairThemAll

Repository Structure

The repository is structured as follow:

├── docs: content for the website
├── results: 
│   └── <benchmark>
│       └── <project>
│           └── <bug_id>
│               └── <repair tool>
│                   └── <seed>
│                       ├── grid5k.stderr.log: stderr of the execution
│                       ├── grid5k.stdout.log: stdout of the execution (without the repair)
│                       ├── repair.log: repair log
│                       ├── result.json: standardize output 
│                       └── detailed-result.json: raw output of the repair tool, if it generates a json file
└── script
    └── get_patched_bugs.py: the script that are used to generate the table for the paper

Patched Bugs

The data of this repository is also available as a website: http://program-repair.org/RepairThemAll_experiment

Total generated patch: 67,211

Execution time 314 days, 12:29:19.419491

Chi-square Test of independence

Chi2 value= 170.43487132271886 p-value= 5.945480330471514e-39 Degrees of freedom= 1

Chi2 value= 84.90652479289551 p-value= 3.128091736130167e-20 Degrees of freedom= 1

Chi2 value= 150.4020168750391 p-value= 1.4160845009256217e-34 Degrees of freedom= 1

Chi2 value= 144.80217516680622 p-value= 2.372523759882535e-33 Degrees of freedom= 1

Chi2 value= 10.701973234378928 p-value= 0.0010702132907778191 Degrees of freedom= 1

Chi2 value= 38.10114926497659 p-value= 6.717055566199569e-10 Degrees of freedom= 1

Chi2 value= 39.69012031778793 p-value= 2.976273080413384e-10 Degrees of freedom= 1

Chi2 value= 6.76253623850222 p-value= 0.009309135821381086 Degrees of freedom= 1

Chi2 value= 158.83167769741897 p-value= 2.036659201530019e-36 Degrees of freedom= 1

Chi2 value= 46.25197019724452 p-value= 1.0398193445599021e-11 Degrees of freedom= 1

Chi2 value= 17.333764012540335 p-value= 3.1356574417361234e-05 Degrees of freedom= 1