mc-imperial / dredd

Framework for evaluating C/C++ compiler testing tools
Apache License 2.0
11 stars 3 forks source link

Mutation optimisations are leading to missed mutants in the case of mutating unsigned comparison operators #285

Closed afd closed 1 month ago

afd commented 1 month ago

See #284 - this adds an execute test which is failing. If one hacks the scripts that checks execute tests and adds the --no-mutation-opts flag to Dredd, the tests passes. It seems that a number of cases are being inadvertently missed due to optimisations.

JonathanFoo0523 commented 1 month ago

Dredd doesn't produce != mutation on >= in default mode because of the difference in prelude function. Here's the prelude functions

Default mode:

static int __dredd_replace_binary_operator_GE_arg1_unsigned_int_arg2_unsigned_int(unsigned int arg1, unsigned int arg2, int local_mutation_id) {
  if (!__dredd_some_mutation_enabled) return arg1 >= arg2;
  if (__dredd_enabled_mutation(local_mutation_id + 0)) return arg1 == arg2;
  if (__dredd_enabled_mutation(local_mutation_id + 1)) return arg1 > arg2;
  return arg1 >= arg2;
}

no-mutation-opts mode:

static int __dredd_replace_binary_operator_GE_arg1_unsigned_int_arg2_unsigned_int(unsigned int arg1, unsigned int arg2, int local_mutation_id) {
  if (!__dredd_some_mutation_enabled) return arg1 >= arg2;
  if (__dredd_enabled_mutation(local_mutation_id + 0)) return arg1 == arg2;
  if (__dredd_enabled_mutation(local_mutation_id + 1)) return arg1 != arg2;
  if (__dredd_enabled_mutation(local_mutation_id + 2)) return arg1 > arg2;
  if (__dredd_enabled_mutation(local_mutation_id + 3)) return arg1 <= arg2;
  if (__dredd_enabled_mutation(local_mutation_id + 4)) return arg1 < arg2;
  if (__dredd_enabled_mutation(local_mutation_id + 5)) return arg1;
  if (__dredd_enabled_mutation(local_mutation_id + 6)) return arg2;
  return arg1 >= arg2;
}
JonathanFoo0523 commented 1 month ago

Seems like this is intentional. The difference is caused by the line here

afd commented 1 month ago

Thanks - that optimisation will need some careful scrutiny.

afd commented 1 month ago

Add separate execute tests to check that all expected outcomes occur for mutation of each of the following comparison operators when applied to unsigned operands:

afd commented 1 month ago

Attaching a C program used to generate the expectations for mutations on relational operators - used in #291 and #292.

result_generator.c.txt

afd commented 1 month ago

I had gotten confused about the impact of the optimisations implemented in Dredd. They do lead to certain super-mutants being lost, but that's OK.