mc-imperial / dredd

Framework for evaluating C/C++ compiler testing tools
Apache License 2.0
11 stars 3 forks source link

Adding an off by one mutant #307

Open JamesLee-Jones opened 1 month ago

JamesLee-Jones commented 1 month ago

Since mutants are meant to represent common programmer errors, it may be useful to have an off by one mutant. This is not something that can be readily simulated with the mutants currently available in Dredd. This doesn't feel like an unreasonable addition considering we already replace integer and float with the interesting values '0', '1' and '-1'. If Dredd is being used for test case generation, this feels like a mutant that would be necessary to kill with a test.

What do you think @afd, @JonathanFoo0523?

JonathanFoo0523 commented 1 month ago

I felt like it would lead to a lot of equivalent mutations or mutations that can already be killed by the readily available mutations. For example:

  1. For integer constants used in array size initialization, mutating to arg + 1 (overallocation) is likely an equivalent mutation, and mutating to arg - 1 (under-allocation) is likely equivalent or killed by mutation to 0.
  2. For integers in comparison operators (a op b), mutation of the integer literal in the comparison is likely subsumed by mutation of the op itself.
  3. An offset-by-one mutation on expressions such as x + 1 might be partially simulated by Dredd's mutation operator on the lvalue (e.g., ++((*x))), or fully simulated if the lvalue is never used again after evaluation of the expression (e.g., an expression in a return statement).
  4. In some programs, integer literals are used to call functions with the appropriate number of arguments (think about the argument to argc in main(int argc, char **argv)). Decrementing the integer argument by 1 likely has the same effect as mutating to 0. Incrementing the integer argument likely leads to invalid/null pointer access, which might be captured by existing safeguarding code, or the test case produced might not always be reproducible to kill the mutant.

There might indeed be cases where the offset mutation might lead to test cases that aren't captured by current mutation functions, and it might be worthy to investigate if this is indeed the case. I don't remember encountering any mutation testing tools that have an offset mutation operator when I did my research for my FYP, and I am not sure if this indicates they are less useful.