Provide a way to also supply multiple manual test cases, that is, parameterized or explicit cases.

[Clockwork-Muse]

Often when writing tests, it's good practice to include specific known test cases (usually around boundaries).

Currently, there seems to be these options:

1. Manually write an individual test for each case. This works with one or a small number of boundaries, but becomes burdensome for any large number of similarly-structured cases.
2. Use a macro to generate cases (or a crate that provides such a macro). This works, but generates additional methods under the covers. Failures refer to the failing line in the duplicated method, but picking the case out of the list may be hard in some cases. Too, this requires an additional crate, and almost certainly one that doesn't work with proptest.
3. With proptest, use sample or uniform to select from a list of cases. At lower counts of cases, this should run all of them. However, this is really a misuse of those methods, and may end up confusing the engine besides. Additionally, it can be hard to spot which test case is failing with large manual lists. (4. Might be possible to add manual cases to the regression files? Haven't tried, but this seems like even more of a hack, and that you're more likely to lose cases when tests change)

What I'd like is some way to provide explicit, multiple, test cases, in addition to the ones generated by proptest. All cases should be run (as opposed to only being maybe run, as with sample/uniform). Simplification scenarios would likely need to ignore the cases (especially for composed strategies), but being able to consider them as "known good" data would be a bonus. Ideally, it would also be possible to trace back to which manual case failed (eg, a line number, in addition to the data).

[Andlon]

I would like to chime in and say that this is one of the things I've been missing the most from proptest as well. Sometimes I find myself writing manual test cases with the same test logic exactly as described here, which seems highly redundant and a waste of time.

[rex-remind101]

To try and paraphrase, rather than always having random variables as input to a proptest, you'd like to be able to also run through a set of constant inputs representing specific example test cases? Assuming I'm understanding correctly this sounds beneficial. Do you have a straw man api for how you'd like this to look?

[Clockwork-Muse]

Yes, the idea is to have a set of constant inputs to run against tests, preferably in addition to any random inputs.

As for strawman...

Most of the other test frameworks I'm familiar with in other languages have you decorate the method directly (take C#'s xunit as an example), and usually allow supplying both multiple "simple" cases as well as generators. So something like:

proptest! {
    #[test]
    #[case(5, 7)]
    fn test_add(a in 0..1000i32, b in 0..1000i32) {
        let sum = add(a, b);
        assert!(sum >= a);
        assert!(sum >= b);
    }
}

... would be preferable.

For some higher-order strategies, it might be useful to have a way to essentially concat/require base cases, or ensure that they're included. Something like:

fn some_function(stuff: Vec<String>, index: usize) {
    if index >= 0 && index < stuff.len() {
        let _ = &stuff[index];
    } else {
        let _ = <some default>;
    }
    // Do stuff
}

fn vec_and_index() -> impl Strategy<Value = (Vec<String>, usize)> {
    prop::collection::vec(".*", 1..100).require_empty()
        .prop_flat_map(|vec| {
            let len = vec.len();
            (Just(vec), 0..len)
        })
}

proptest! {
    #[test]
    fn test_some_function((vec, index) in vec_and_index()) {
        some_function(vec, index);
    }
}

... because then the derivation of the rest of the chain would apply as well.

[matthew-russo]

This sounds a lot like what rstest provides.

While in general I think APIs/functionality that support this are very useful, I'm not entirely confident that proptest is the right place for it and it may be better to use something like rstest in addition to proptest if that already meets your needs. It wouldn't require rewriting the core test logic as you could have a shared function e.g.

#[cfg(test)]
mod test {
    fn test_add(a: i32, b: i32) {
        let sum = add(a, b);
        assert!(sum >= a);
        assert!(sum >= b);
    }

    proptest! {
        #[test]
        fn test_add_generated(a in 0..1000i32, b in 0..1000i32) {
            test_add(a, b);
        }
    }

    #[rstest]
    #[case(5, 7)]
    #[case(10, 10)]
    fn test_add_well_known(#[case] a: i32, #[case] b: i32) {
        test_add(a, b);
    }
}

Is the use of another crate that already has fixture/table-based testing functionality something that would work for you and if not, I'd be curious to hear why proptest is a better home for this type of functionality. I'm open to hearing arguments in any direction but I'd like to try and understand the problem space and specific needs more.

Some counter arguments against what I've said are that the above example is a bit more verbose and other related projects like Python's Hypothesis which we list as our inspiration include this functionality via example

[Clockwork-Muse]

I previously looked at rstest and specifically rejected it due to ergonomics; the casing results in multiple additional methods being generated (this covers point 2 in the original post).

I would be willing to use a different external crate that did not behave this way.

(.... it may be that internally proptest does generate such additional test methods, however if so it abstracts it. In contrast most of the other crates I'm aware of are very loud about the additional methods)

[matthew-russo]

Thanks for the quick reply -- i missed the focus on tracing failures/line numbers/case detection in your original post so thanks for calling attention to that!

[Andlon]

I would just like to point out that using an intermediate function for the test body doesn't work very well, because the assertions are different: normal Rust code uses assert!, but proptest uses prop_assert!. You can technically use assert, but this floods the output stream (terminal) while minimizing, and as a result minimizing can be orders of magnitude slower to boot (because for every failing test you end up running the panic handler, which prints to terminal).

[matthew-russo]

Thanks for the context. I think we have enough to go off to come up with an initial prototype. We can probably try to get something on a branch and then get some feedback on it before we try to get it on to the main branch.