mc-imperial / dredd

Framework for evaluating C/C++ compiler testing tools
Apache License 2.0
11 stars 3 forks source link

Selective mutation instrumentation #218

Open JamesLee-Jones opened 5 months ago

JamesLee-Jones commented 5 months ago

Allow a subset of possible mutants to be instrumented by first performing a pass to collect the possible mutants into a .json, selecting which should be enabled, and then performing a second pass to instrument them.

JamesLee-Jones commented 5 months ago

One issue that will have to be overcome here is avoiding writing to the source file if the mutation trees differ. Currently the mutation tree is built and checked as the mutations are applied. If at some point in this process, we find the two trees differ, we wish to throw and error and tell the user that they must be the same. However, because mutations are applied as the tree is built, the source file would have already been modified at this point.

JamesLee-Jones commented 5 months ago

This is currently implemented in such a way that each 'MutationGroup' can be enabled or disabled and thus that all possible mutants for a source location will either be included or excluded. It may be the case that we would like more fine grained control than this but this becomes much more difficult as one function is used for replacing an entire class of mutants. Therefore, this would either require many more replacement functions, or a change how we currently instrument.

JamesLee-Jones commented 5 months ago

My initial idea was to make each mutation function take in a vector<bool> that would contain whether each possible mutant should be used or not and then each mutant could be wrapped in an if check. After some experimenting, this seems to be significantly too slow on even simple code (as in examples/simple/pi.cc) to be feasible.

Another option is to create a new mutation function for each possible combination of enabled mutants that we encounter. This will lead to huge code bloat but will in turn mean that there should be no extra runtime overhead over the approach that we already use since the functions will be inlined. However, this goes against Dredd's design decision to have a single function for each mutant category.