The perturber (part of the test that takes a template plan and produces a plan with a sampled is currently entirely random.
John and Ally suggested that it would be nice to have a systematic mode whereby the tester progressively enumerates every possible perturbance of the plan.
If Go had generators, it would look something like this:
// on one machine
for _, s := range samplesOf(subjects) {
// probably *in parallel* for each compiler, making sure to handle different compiler configuration surfaces
for _, c := range compilers {
for _, m := range c.marches {
for _, o := range c.opts {
yield()
}
}
}
}
The way I see this working is something like the following:
[ ] Abstract the current randomised sampler out into an interface, and create a 'window-based' sampler that returns subjects 0-n, n+1-2n, ..., and wraps over at the end ensuring the sample size is the same. We might be able to ship this separately from the below.
[ ] Abstract the main implementations of the perturber's functionality into a 'policy' interface, with the current randomised policy becoming the default.
[ ] Make a systematic policy that slots into the same interface, but holds a state machine that implements for each machine the loop suggested above.
[ ] (Bonus) Make a mechanism for the perturber and/or sampler to signal when it has run out of permutations, and make it possible for the tester to finish when that occurs.
If we play our cards right, we might be able to make it possible to replace setc with perturb, by making the perturber able to admit a policy that just sets everything to user-specified levels.
The perturber (part of the test that takes a template plan and produces a plan with a sampled is currently entirely random.
John and Ally suggested that it would be nice to have a systematic mode whereby the tester progressively enumerates every possible perturbance of the plan.
If Go had generators, it would look something like this:
The way I see this working is something like the following:
If we play our cards right, we might be able to make it possible to replace
setc
withperturb
, by making the perturber able to admit a policy that just sets everything to user-specified levels.