elidupree / live-prop-test

Fearlessly write both cheap and expensive runtime tests (contracts) for Rust functions.
Apache License 2.0
0 stars 0 forks source link

On errors, apply shrinking if possible #6

Open elidupree opened 4 years ago

elidupree commented 4 years ago

Notes:

First, what shrinking implementation can we use? proptest's implementation is based on strategies rather than values. quickcheck's implementation is based on values, but it also requires that if users implement shrinking, they also have to implement random generation, and it doesn't have an easy way to derive it. Maybe I could write my own shrinking trait, and a derive macro for it. As a more short-term, future-compatible solution, I could just privately use quickcheck's shrinking when available, and fall back to "no shrinking" for types that don't implement it.

Second, how should I consider performance? In the case of expensive tests, we certainly don't want to run shrinking if errors are only being logged rather than panicking. Even in the case of panic, if the test is very expensive, shrinking could take a while. Maybe a good compromise is to set a timeout on shrinking (defaulting to, say, 100ms).

Third, how to consider side effects? We officially support functions with side effects (say, on files or the network). Clearly, there are cases where shrinking shouldn't be applied. Having shrinking be an opt-in setting seems like the safest approach.

elidupree commented 4 years ago

The obvious interface is

#[live_prop_test(
  postcondition = "expression"
  shrink = "true"
)]
fn tested_function() {

}

It would probably be good to make the shrink setting be inherited (i.e. you can apply it to an impl or module, and it affects all methods inside; individual methods can still override it; the default is "false").

I suppose there's no reason we can't make the shrink argument an arbitrary expression (rather than just "true" or "false"); it could even obey the postcondition rules, allowing result and old.

If it evaluates to true after a postcondition failure occurs, we try to shrink all arguments that can be shrunk. To do this, we'll have to have made clones of the arguments at the beginning… It's okay if some arguments can't be shrunk, but they still need to be cloned. Even that won't work properly for Rc<RefCell<T>>, but we'll just document the behavior and let the user deal with any weird cases they create.

Or maybe... I guess it's looking like we may have to implement our own shrinking trait, and that trait could have its own approach to cloning that specifically clones the interior of Rc<T>. (You have to clone the interior to shrink the interior, after all...) But wait, we were thinking about types that don't implement the shrinking trait, like Rc<RefCell<SomeThirdPartyStruct>>. Cloning the interior as a safe default is good, but I don't think we can do that without specialization.

The process should still COMPILE even if there's an argument that can't be cloned. If it hits a failing postcondition, it can just have the message include "tried to shrink, but wasn't able to because "argument_name" didn't implement Clone". And another error message for "none of the arguments implemented Shrink" (with a further special case where functions with 0 arguments don't show an error)

elidupree commented 4 years ago

Should it be possible to configure the timeout for shrinking? Like, there could be a second inheritable setting "shrink_timeout". My instinct is to not allow that, though. Maybe it could be something that you can override using an environment variable if you want, like the error message could say

Note: Shrinking was stopped due to the [default] timeout of [100ms]. Consider rerunning with LIVE_PROP_TEST_SHRINKING_TIMEOUT=1000

But I'm inclined to just not implement a way to override it; it doesn't seem like that would be useful very often, and it has a maintainability cost.