elidupree / live-prop-test

Fearlessly write both cheap and expensive runtime tests (contracts) for Rust functions.
Apache License 2.0
0 stars 0 forks source link

Do something special when a precondition failure occurs inside an outer tested function? #10

Open elidupree opened 4 years ago

elidupree commented 4 years ago

Currently, precondition failures never know what to blame or how to make a regression test.

Live-prop-test could keep an internal call stack, and thus blame the failure on the next tested function outside the one which failed a precondition.

...Except that we didn't necessarily record the inputs to that function.

...But if it happens that we did, shouldn't we take advantage of that?

It would be a little flaky (e.g. we can't necessarily guarantee that it wasn't caused by a failure in an outer precondition that wasn't checked), but we could just mention that in the error message that occurs in this case.

elidupree commented 3 years ago

"Luckily, when this precondition failure occurred, we were already running tests for the containing function, function_name(). So this failure is likely to be the fault of function_name(). Suggested regression test:"

"This precondition failure occurred, inside function_name(), so this failure is likely to be the fault of function_name(). Unfortunately, we weren't running tests for function_name() at the time, so we didn't record the inputs to it."

"...but fortunately, we DID record the inputs to another, outer containing function, third_function_name():"