Closed gkaracha closed 3 years ago
@dorranh I noticed that test_mutated_src
in mutate.py
fails when the mutated source does not compile. Wouldn't it be better to accept the mutation as "tested" instead? Mutation rules for kit for example when mutating kit.ml
might mutate the definition of a function, making the source ill-typed. Instead of having to change the mutation groups manually, I think it best to just move on. This way both locally and on CI we'll be able to execute the mutation script without intervention. What do you think?
@gkaracha, that's a good point. We're starting to bump up against the limits of the script's intended usage (catching some mutations vs exhaustively checking mutations). In principle I don't see an issue with skipping faulty mutations. I think that in the current implementation this attempted mutation will still count against the n_mutations
you specify, but this might actually be desirable behavior since it will help avoid cases where the runtime starts to get long because faulty mutations are being attempted. To improve performance we might also want to consider recording failing mutations in memory so that they are only attempted once.
You're absolutely right, we're past the originally intended usage. For now I would accept the mutations that led to an incompilable source, even if they still count against the specified n_mutations
; this should allow me to progress with the batch-mutation I've been working on as part of #248 (and perhaps another round after that). Later we could improve the script even more, though I am not sure if it will ever become a priority, perfecting the script.
Good point :thinking: For what it's worth, it's this branch that I've been using locally to generate the data for #248 (currently containing 97 mutations to add tests for) with n_mutations = 1000
. I guess after completing a first round of adding tests for the mutations in #248 we could try raising n_mutations
even more and see what we get.
MUTATION_GROUPS
with more mutations. Until now we've had some entries forfixedPoint.ml
, but other newtypes (e.g., tok, kit, ..) did not have any rules.test_mutated_src
accept non-compiling mutations as tested. This silent recovery allows us to run the script on CI on more files without intervention.