proptest-rs / proptest

Hypothesis-like property testing for Rust
Apache License 2.0
1.64k stars 152 forks source link

Implement is_minimal_case #477

Open Andlon opened 1 week ago

Andlon commented 1 week ago

This is a draft for an implementation of the idea suggested in #161.

Currently, using proptest for debugging is very cumbersome because you cannot easily run code only for the minimal failing test case. In my experience, I usually only care about the minimal test case, but if I want to print some debug info about that failing test case, I'll have to print for all test cases, which is borderline useless.

This PR introduces a method is_minimal_case that you can call inside a proptest to determine if the current test case is the minimal failing case. To do this, the minimal test case is run one more time after it has been identified, and a thread local boolean is set before and after.

This allows you to write tests like these:

use proptest::{proptest, prop_assert, is_minimal_case};

proptest! {
    #[test]
    fn test_is_not_five(num in 0 .. 10) {
        if is_minimal_case() {
            eprintln!("Minimal test case is {num:?}");
            export_to_file_for_analysis(num);
        }

        prop_assert!(num != 5);
    }
}

A nice side effect of this is that it makes it much easier to use proptest with a debugger: Simple add an if condition like the one above, and put a breakpoint on a statement inside the branch. Then you can run the debugger only through the minimal failing test case.

I haven't yet written any tests, because I'm frankly not quite sure how best to go about it. I suppose at a minimum we should have a test where a TestRunner is manually invoked with a dummy test case, where we test that the reported minimal test case is what we expect. But I'm not sure how best to write such a test in a way that fits well with the rest of the code base. I'd be very grateful to have some suggestions here.

I'm not sure how to correctly handle forking here - if anything in particular even needs to be done differently. And whether or not I should forward is_from_persistent_seed. Since I'm not so familiar with the code base I don't really understand the ramifications.

Happy to make any necessary adjustments to the PR.

Andlon commented 1 week ago

I have no idea what happened with CI. It seems that it just hangs on some build script?