Project Overview

[rohancme]

Introduction

Existing Tools for test generation end up generating a very large number of tests, not all of which contribute to an increase in the quality of testing. Tools such as Randoop and EvoSuite have been shown to be effective at generating unit tests. PITest is a tool that measures the quality of tests via mutation testing. The aim of this project is to experiment with leveraging these and/or possibly other tools to create a service that can generate high quality tests automatically.

Background

• PITest(http://pitest.org) is a mutation testing framework which measures the effectiveness of existing tests. This can be an important tool to measure test quality.
• As seen in this paper, it seems clear that none of the test generation frameworks perform well in a majority of cases. It will be important to have some combination of existing tools to generate tests. There is an SBST contest (http://sbstcontest.dsic.upv.es/?p=44) that takes place yearly and Randoop and Evosuite seem to be the best tools currently.
• The Exsyst (https://www.st.cs.uni-saarland.de/exsyst/) paper seemed quite promising, and they are in the process of making it open source. If this happens before the end of the semester, EvoSuite could be replaced with Exsyst since it is based on EvoSuite.

  Project Description

This project aims to create a service that can:

• Be invoked via Git hooks
• Analyse existing tests in a codebase to look at the coverage and quality of tests
• Generate unit tests for code that is either not covered, or has tests that are not good based on the results of running a mutation testing tool.
• Use mutation testing to get rid of tests that are not improving test quality.
• Automatically submit a Pull Request with these tests

  Proposed Approach

Link to the Milestones

Initially, it is important to explore existing test generation tools to decide on the ones to be used. There are Search Based Software Testing competitions which compare these tools yearly. This would be a good starting point to narrow down the tools. These tools can also be run on large open source projects and their results compared. It would be important to have multiple runs since most search-based tools do not run deterministically. This will be the first milestone. Once the tools have been decided, the rest of the project can be approached in three phases:

1. Set up the basic workflow [Milestone 2]:
   • Set up the service and allow it to be called via Git hooks.
   • Generate unit tests and submit a pull request
2. Improve test quality [Milestone 3]:
   • Study and come up with a measure of test quality by using results from mutation testing and code coverage statistics.
   • Use the results to prune the number of tests generated in the first phase.
3. Production Ready [Milestone 4]:
   • Only generate test cases for parts of the codebase that do not have sufficient code coverage or which failed mutation testing

     Completed Project Workflow

On completion, the tool should be able to perform the following steps:

1. Run PITest on code that has been pushed.
2. Identify the classes/files that are failing the mutation tests
3. Run Randoop/Evosuite only on those files
4. Run PITest again to identify the tests that are actually improving mutation coverage.
5. Submit the PR with these tests

[rohancme]

@chrisparnin does this look alright? I'll then start breaking down the milestones into issues

[chrisparnin]

Ok looks good. Seems like a good candidate to make towards a paper publication.