How to achieve consistent code coverage?

[CBenoit]

Is there a way to achieve a consistent code coverage? I’m generating a coverage report using cargo-llvm-cov, but some tests where the problem space is very big are causing some lines to not be covered for each individual cargo test run. Everything end up covered when enough test cases are generated so I’m not too much concerned about this fact, but the inconsistent coverage report is a bit troublesome. I imagine I could try to find all seeds which exercise all the branches, but that’s not very realistic for me to do this (unless this is automated, but proptest is not coverage-based). I’m thinking that some environment variable to set the seed used to generate the seeds (so that we always generate the coverage report using the same set of seeds) would be good enough, but I don’t think there is such thing as of today.

[matthew-russo]

I would think of proptest as a tool in your toolbox but not the end-all-be-all of your testing. Without a ton of info on your use case, I typically rely on unit tests for ensuring specific branches are hit. You can use rstest or similar crates to run tests against a set of explicit values (https://github.com/la10736/rstest).

We have some plans to better support this directly in proptest and have chatted with the rstest maintainer on better integration between proptest and rstest. This is generally tracked in #284.

Another strategy, if unit tests don't work for you or if you still want more explicit, consistent coverage with proptest, is to make newtype wrappers that guarantee a certain branch is hit. For example if you have a function

fn is_even(u: usize) -> bool {
    // pretend you can't just write this as a single line `u % 2 == 0` so this example works with an if statement
    if u % 2 == 0 {
        true
    } else {
        false
    }
}

and you want to validate that it works for both even and odd numbers you could do something like

#[cfg(test)]
mod test {
    use super::is_even;

    struct Even(usize);
    impl Arbitrary for Even {
        /* impl that only generates even numbers */
    }

    struct Odd(usize);
    impl Arbitrary for Odd {
        /* impl that only generates odd numbers */
    }

    proptest! {
        #[test]
        fn is_even_returns_true_for_even_nums(e in any::<Even>()) {
            assert!(is_even(e.0));
        }

        #[test]
        fn is_even_returns_false_for_odd_nums(o in any::<Odd>()) {
            assert!(i!s_even(o.0));
        } 
    }
}

A contrived example but hopefully it illustrates how you can use more specific types to force certain branches to be executed.

I'm going to close this for now but if you have any further questions, feel free to reopen

[CBenoit]

Thank you for your answer. I appreciate you took time to write such an elaborate answer. However, maybe I wasn’t clear in my initial post, I’m sorry about that.

The problem for me is NOT to ensure all branches are hit when using proptest. My problem is that I can’t properly track the coverage of my codebase because obviously the proptest tests are random and because of that the coverage percent is unstable. In my case, I can run the coverage analysis 10 times and the results will vary by roughly ±2% even though I don’t change anything in the code. Because of that I can’t really track commits which are reducing or improving the coverage; there is just too much noise.

Obviously, if I write many manual tests and handwrite all the relevant values, then that’s not a problem anymore. However, this is kind of missing the point I my opinion.

For example, the approach I suggested would help:

I’m thinking that some environment variable to set the seed used to generate the seeds (so that we always generate the coverage report using the same set of seeds) would be good enough, but I don’t think there is such thing as of today.

Such an environment variable could be set in order to give up randomness when running the coverage analysis task (and I think it’s fine).

As an aside, I’m happy to hear that you are working with the rstest maintainer. I happen to be already using rstest crate along proptest, and I’m curious to see what this collaboration will bring to the table. :slightly_smiling_face:

EDIT: again, this is off-topic, but in case someone who is actually looking to write a test which verifies that a certain code path is exercised read this issue, I want to mention that cov-mark is a great, underrated option for that.

Also, instead of defining a canonical strategy for a newtype wrapper, I think that a good alternative for the example above is to define a custom strategy (typically by writing a function returning something like impl Strategy<Value = usize>).

EDIT 2:

I'm going to close this for now but if you have any further questions, feel free to reopen

FYI, external users don’t have the permission to re-open issues, so I’ll leave you decide whether this should be re-opened or not.