proptest-rs / proptest

Hypothesis-like property testing for Rust
Apache License 2.0
1.74k stars 161 forks source link

Proptest fails to find trivial failing example (but Hypothesis does) #500

Open MaxG87 opened 2 months ago

MaxG87 commented 2 months ago

Coming from Python, where I use Hypothesis to write my tests whenever possible, I wanted to make use of proptest for new Rust projects.

Today I came across a very simple example where proptest fails to detect an edge-case bug in a naive implementation, while hypothesis does not. I suspect this is an opportunity to better understand how to use proptest, so I wanted to ask for an explanation.

The bug is in a function calculating the total number of pages I need to request, based on a page size count and the total number of elements, total_count. The erroneous implementation is total_count / count + 1. The bug is that, if total_count is a multiple of count, the result is to big by 1. So, for 2 and 2, the result is 2 instead of 1. Obviously, if I request a page big enough to contain all elements, a single page suffices.

Hypothesis reliable and quickly finds this bug. To my big surprise, proptest reliably does not find a bug here. Furthermore, if I hardcode let count = 2 in the test body, proptest fails to reduce total_count to 2 too.

Could someone please help me to understand this behaviour?

The Rust code is the following:

fn get_total_pages(total_count: usize, count: usize) -> usize {
    total_count / count + 1
}

#[cfg(test)]
mod tests {
    use super::*;
    use proptest::prelude::*;

    proptest! {
        #[test]
        fn test_get_total_pages(
             total_count in 0usize..1000000,
             count in 1usize..100000,
        ) {
            let result = get_total_pages(total_count, count);
            let ub = count * result;
            let lb = ub - count;

            if total_count % count == 0 {
                prop_assert_eq!(ub, total_count);
                prop_assert!(false); // apparently never hit
            } else {
                prop_assert!(lb < total_count && total_count < ub);
            }
        }
    }
}

The Python code is:

from hypothesis import given, settings
from hypothesis import strategies as st

@given(
    total_count=st.integers(min_value=0),
    count=st.integers(min_value=1),
)
def test_get_total_pages(total_count: int, count: int) -> None:
    result = total_count // count + 1
    ub = count * result
    lb = ub - count
    if total_count % count == 0:
        assert ub == result
    else:
        assert lb < total_count <= ub