elidupree / live-prop-test

Fearlessly write both cheap and expensive runtime tests (contracts) for Rust functions.
Apache License 2.0
0 stars 0 forks source link

Be stricter about throttling expensive tests on recursive functions #11

Open elidupree opened 4 years ago

elidupree commented 4 years ago

Right now, test throttling is only based on time already consumed by test functions. But in inner recursive calls, we don't yet know how much time will be spent in postconditions of the outer calls. There's an inherent tension here, and currently, if there is no debt at the start, the code may commit to running all of the tests of all the recursive calls, which could be bad if they are all expensive and there are hundreds of them.

Currently, the code doesn't even store setup debt until the end of the postconditions, so even expensive setups don't lead to any throttling before the recursive calls start exiting.

So we need to refactor the system to make at least two fixes:

But of course, we don't want to create a situation where doing lots of expensive setup means all the postconditions get canceled. This will take some design work.

elidupree commented 3 years ago

But of course, we don't want to create a situation where doing lots of expensive setup means all the postconditions get canceled. This will take some design work.

And also, it seems like the outermost postcondition is generally the most important; there should probably be a rule that at least the outermost postcondition never gets canceled.