Feature Idea: Explicit test cases

[shepmaster]

While reading Property-based Testing using Hypothesis in Python, I noticed this code:

@given(text())
@example('')
def test_decode_inverts_encode(s):
    assert decode(encode(s)) == s

What I find clever is the @example bit. When quickcheck finds a problem, I usually end up copy-pasting the body of the test and then hardcoding the witness. However, if there were some way of extending the list of inputs with hardcoded values, that could make that case a bit easier sometimes.

The Python syntax matches most closely to a Rust attribute:

#[quickcheck]
#[quickcheck_example(vec![], "no values")]
fn double_reversal_is_identity(xs: Vec<isize>) -> bool {
    xs == reverse(&reverse(&xs))
}

(I took the opportunity to extend it with a string describing the test case)

[BurntSushi]

We should probably figure out how this fits into the API before jumping to compiler plugin stuff, but in principle, this does seem like a good idea.

I think this is a dupe of #40 though.

[shepmaster]

how this fits into the API

Oh absolutely; I just didn't want to prescribe anything without looking at the API and thinking about it. The super-high level was enough to convey the intent.

a dupe of #40

It's certainly super-related, although the phrasing "export" and "a file containing the test cases" doesn't immediately bring this to mind. Your comment of "run tests with these arguments" fits much closer. Would you like me to close in favor of #40?

[BurntSushi]

I think I was looking specifically at this comment: https://github.com/BurntSushi/quickcheck/issues/40#issuecomment-69496420

I'm fine leaving this one open though. The issue titles do suggest different things, you're right.

[Eh2406]

Hypothesis has 2 great features that are related to this.

1. You can add an example to be run within the quickcheck process as a decorator. See the OP for examples.
2. It saves all failing test to a file to ensure that they get run again in the next test session.

[m4rw3r]

I made a quick and dirty implementation of this here: https://github.com/m4rw3r/quickcheck/tree/wip_explicit_test_cases I am not happy with the API at the moment, especially not how parameters need to be specified.

Example usage (we guarantee that the empty list will be covered every time):

quickcheck! {
    #[example( ((vec![],), ()) )]
    fn peek_pred(i: Vec<u8>) -> bool {
        let first = i.first().cloned();
        peek().parse(&i[..]) == (&i[..], Ok(first))
    }
}

note the extra ((... ,), ()); this is because quickcheck also calls result on the returned value of the predicate to be tested, the second part of the tuple is therefore the parameter to bool which is (). The tuple itself is required because it is a function and might have multiple parameters.

[BurntSushi]

@m4rw3r Awesome! Can you show an example of how to use it without any macros?

[m4rw3r]

@BurntSushi There is a test here which makes sure that an empty vector is tested every time: https://github.com/m4rw3r/quickcheck/blob/3d46898165f324cafb2e30c4bb5d13a472f17112/src/tests.rs#L214-L224

But the example above expands to:

#[test]
fn peek_pred() {
    fn prop(i: Vec<u8>) -> bool {
        let first = i.first().cloned();
        peek().parse(&i[..]) == (&i[..], Ok(first))
    }
    let v = vec![(( vec![], ), () )];
    quickcheck_(prop as fn($($arg_ty),*) -> $ret, v);
}

And for multiple parameters, here is another example:

#[test]
fn token_pred() {
    use quickcheck::quickcheck_;

    fn prop(i: Vec<u8>, c: u8) -> bool {
        match i.first().cloned() {
            Some(t) if t == c => token(c).parse(&i[..]) == (&i[1..], Ok(t)),
            _                 => token(c).parse(&i[..]) == (&i[..], Err(Error::expected(c)))
        }
    }

    // Let's make sure nulls and empty list work
    let explicit = vec![
        ((vec![],      b'\0'), ()),
        ((vec![],      b'a'),  ()),
        ((vec![b'\0'], b'\0'), ()),
        ((vec![b'\0'], b'a'),  ())
    ];

    quickcheck_(prop as fn(Vec<u8>, u8) -> bool, explicit)
}

quickcheck_ is the version which allows for a list of specific values. Would have been nice to just be able to write:

let explicit = vec![
    (vec![],      b'\0'),
    (vec![],      b'a'),
    (vec![b'\0'], b'\0'),
    (vec![b'\0'], b'a'),
];

But the Testable impl for functions is implemented on functions returning Testable, so it needs to use a tuple or something similar to be able to also specify the parameters for the returned Testable.

[m4rw3r]

An addition to the above: It seems like the nested Testable in the fn(...) -> T implementation does not actually expose what values it used on failure, its returned arguments list is overwritten: https://github.com/BurntSushi/quickcheck/blob/6c7a4a85402e09301427700e9eed38c73a8f6cf9/src/tester.rs#L292

This makes me wonder if it really is necessary to unify them all (eg. bool, Result and fn) under Testable and instead have some other trait which can be coerced into a success/failure value to be returned from the fn(...) -> T which implements Testable.

[BurntSushi]

It sounds like proptest adding something like this. It might be good to try it out instead of quickcheck if this is important to your workflow!