ssm-lang / Scoria

This is an embedding of the Sparse Synchronous Model, in Haskell!
BSD 3-Clause "New" or "Revised" License
4 stars 0 forks source link

Even better shrinker! #47

Open j-hui opened 2 years ago

j-hui commented 2 years ago

The shrinker is already really great, and it's producing some excellent counterexamples to populate our regression test suite (thanks @Rewbert !). But it could be even better:

I guess the overall theme of this PR is about being more flexible with what constitutes a "smaller" test case---I believe implementing the above may lead to even more intuitive test cases (right now, the test cases are small enough to understand but still require some manual doctoring to be self-explanatory).

Rewbert commented 2 years ago

Changing the types of things is very difficult, as the program might not (most likely not) be type correct anymore. And furthermore, if the shrinker can change an Int32 to an Int64, it can probably also do the opposite, and thus the shrinker will never terminate. When we are shrinking a program we are not maintaining any state, so there's no way to know that we've not already tried to give a variable a certain type.

Concerning the constants, the shrinker is already trying to do a syntactic reduction of expressions. If only a single constant remain after this step it could probably be shrunk, yes, but I don't personally think it's worth the effort. We're just trading one constant for another. I did prepare the expression shrinker to do this though, so you'd just need to write a function SSMExp -> [SSMExp] where the resulting list contains mutations of the expression under scrutiny. Then that function can be plugged into the rest of the machinery. Empty list = unsuccessful shrinking.

The skips can be pruned out, but keeping them won't change any semantics or anything. It should be easy to add this as a last shrinking step though, if we think it's worth the time.

j-hui commented 2 years ago

Ah gotcha. In terms of type errors, @yc2454 's type checker should come in handy, but I do understand your concern.

I think when I have some spare cycles, I'll write these shrinkers myself to get my hands dirty with the shrinker, and see if it's actually useful. For now, I'll punt on this issue.