tezos-checker / checker

An in-development "robocoin" system for the Tezos blockchain
24 stars 16 forks source link

Updates to the mutation script #253

Closed gkaracha closed 3 years ago

gkaracha commented 3 years ago
  1. Extend MUTATION_GROUPS with more mutations. Until now we've had some entries for fixedPoint.ml, but other newtypes (e.g., tok, kit, ..) did not have any rules.
  2. Make test_mutated_src accept non-compiling mutations as tested. This silent recovery allows us to run the script on CI on more files without intervention.
gkaracha commented 3 years ago

@dorranh I noticed that test_mutated_src in mutate.py fails when the mutated source does not compile. Wouldn't it be better to accept the mutation as "tested" instead? Mutation rules for kit for example when mutating kit.ml might mutate the definition of a function, making the source ill-typed. Instead of having to change the mutation groups manually, I think it best to just move on. This way both locally and on CI we'll be able to execute the mutation script without intervention. What do you think?

dorranh commented 3 years ago

@gkaracha, that's a good point. We're starting to bump up against the limits of the script's intended usage (catching some mutations vs exhaustively checking mutations). In principle I don't see an issue with skipping faulty mutations. I think that in the current implementation this attempted mutation will still count against the n_mutations you specify, but this might actually be desirable behavior since it will help avoid cases where the runtime starts to get long because faulty mutations are being attempted. To improve performance we might also want to consider recording failing mutations in memory so that they are only attempted once.

gkaracha commented 3 years ago

You're absolutely right, we're past the originally intended usage. For now I would accept the mutations that led to an incompilable source, even if they still count against the specified n_mutations; this should allow me to progress with the batch-mutation I've been working on as part of #248 (and perhaps another round after that). Later we could improve the script even more, though I am not sure if it will ever become a priority, perfecting the script.

gkaracha commented 3 years ago

Good point :thinking: For what it's worth, it's this branch that I've been using locally to generate the data for #248 (currently containing 97 mutations to add tests for) with n_mutations = 1000. I guess after completing a first round of adding tests for the mutations in #248 we could try raising n_mutations even more and see what we get.