JanusGraph / janusgraph

JanusGraph: an open-source, distributed graph database
https://janusgraph.org
Other
5.2k stars 1.16k forks source link

Add support for mutation testing #17

Open mbrukman opened 7 years ago

mbrukman commented 7 years ago

Mutation testing can uncover ineffective tests or uncovered conditions without significant manual effort by mutating source code before running the tests and seeing if they still pass.

Per Wikipedia:

Mutation testing (or Mutation analysis or Program mutation) is used to design new software tests and evaluate the quality of existing software tests. Mutation testing involves modifying a program in small ways. Each mutated version is called a mutant and tests detect and reject mutants by causing the behavior of the original version to differ from the mutant. This is called killing the mutant. Test suites are measured by the percentage of mutants that they kill. New tests can be designed to kill additional mutants. Mutants are based on well-defined mutation operators that either mimic typical programming errors (such as using the wrong operator or variable name) or force the creation of valuable tests (such as dividing each expression by zero). The purpose is to help the tester develop effective tests or locate weaknesses in the test data used for the program or in sections of the code that are seldom or never accessed during execution. Mutation testing is a form of white-box testing.

There is a list of mutation testing tools at the bottom of that article.

Here are a few examples of running various mutation testing tools for Java projects.

dylanht commented 7 years ago

After a quick look through the options listed at the bottom of that page and a few google queries, it looks like PIT is the standout (really the only actively maintained/developed project I found in this niche) and is licensed under APLv2.0 - should we look any further?

fppt commented 7 years ago

I had a play around with with PIT it's very easy to setup and really informative. The only downside is that I have not figured out how to aggregate mutations across multiple modules. So for Janus it might not be that informative since many of the tests are in a single module.