Testing different configurations

[gkaracha]

The closer we get to addressing #171 the more difficult it becomes to test Checker. Some thoughts:

Unit tests: Our testsuite contains many unit tests that tightly check edge cases, which are specific to our current choice of parameters (specified in checker.yaml). Any change in the value of these parameters can make the testsuite fail, since the edge values change.

Property-based tests: Most property-based randomized tests should be able to pass even with a different choice of parameters, since these tests mostly describe general properties we expect to hold, that are mostly insensitive to the specific choice of parameters. However, now that we generate more code at build time, even property-based tests cannot always be shared. For example, different curves for the drift derivative (see #283) can be expected to have different properties.

Mutation testing: Apart from the challenge of keeping test coverage high while the codebase changes so fast, the results of mutation testing also become less useful:

• We now rely on tezWrapper which is used meaningfully only in e2e tests. Mutating it thus does not yield meaningful results.
• Soon we'll have a ctez wrapper too (see https://github.com/tezos-checker/checker/issues/279#issuecomment-956115938), so that part will also be tested solely in the e2e tests.
• Several files have turned into jinja templates, which at the moment (reasonably) are not mutated.

Certainly Checker will be shipped with a set of default values and we need the testsuite to have sufficient coverage for the default configuration, but ideally we'd have a way to test a wider range of configurations. It is unclear how to achieve that at the moment, but it looks like we'll need to separate (a) general properties that we expect to hold independently of the choice of values in checker.yaml (assuming that they are reasonably set, of course) from (b) configuration-specific tests.

[gkaracha]

Now that all three generalizations (#213, #283, #311) are implemented we can start thinking about how to go about improving our testing setup to accommodate all the different possible configurations. Currently (as of 89b06df) we've kept only the tests that can pass unaltered under all the following configurations (which are the configurations tested on CI):

• curve_type: bang-bang, tracking_type: index, collateral_type: tez
• curve_type: bang-bang, tracking_type: index, collateral_type: fa2
• curve_type: bang-bang, tracking_type: token, collateral_type: fa2

Existing tests whose behavior is not identical for the above configs have been temporarily removed/commented out but they should be the first ones to come back. Lest we forget, let's list them here

• Continuous-curve-specific tests (https://github.com/tezos-checker/checker/commit/1adefcdb7eed1027334253255ada490b0213b7e7)
• Two unit tests in testChecker.ml marked with FIXME: Operations differ between the FA2 deployment and the TEZ deployment
• Three unit tests in testParameters.ml (test_touch_0, test_touch_1, and test_touch_2)