Infinite Repetition/Never Ending Test with `f32` and `f64`.

[Alexhuszagh]

Issue

I've got a test that uses the following structure:

quickcheck! {
    fn f32_quickcheck(f: f32) -> bool {
        if f.is_special() {
            true
        } else {
            let string = f.to_string();
            f == string.parse::<f32>()
        }
    }

The minimum failing input is:

f32::from_bits(0b11001111000000000000000000000000);   // -2147483600.0

Adding print debugging with --nocapture shows that this value, -2147483600.0, is called repeatedly, without ever stopping, if the checks pass. The actual implementation uses parsing for an arbitrary radix (using big-integer arithmetic), and a float formatter, but pressing enter in the terminal has printed values rapidly being produced afterwards, showing that the quickcheck is completing each iteration faster than the newline can show (and therefore that the test is stalling out indefinitely due to quickcheck, not the actual test code).

A similar result also occurs for f64 with the following structure:

quickcheck! {
    fn f64_quickcheck(f: f64) -> bool {
        if f.is_special() {
            true
        } else {
            let string = f.to_string();
            f == string.parse::<f64>()
        }
    }

The minimal failing input is:

f64::from_bits(0b1100001111100000000000000000000000000000000000000000000000000000) // f=-9223372036854776000.0

It's worth noting that these have a practically identical bit pattern, which may help debugging this issue.

The currently rustc/QuickCheck version is:

$ rustc --version
rustc 1.55.0-nightly (b41936b92 2021-07-20)

[[package]]
name = "quickcheck"
version = "1.0.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "588f6378e4dd99458b60ec275b4477add41ce4fa9f64dcba6f15adccb19b50d6"
dependencies = [
 "env_logger",
 "log",
 "rand 0.8.4",
]

Debugging/Solution

Is there any way to print the seed so I can more reliably debug these issues? Although the same infinite loop is reproducing on each subsequent run, this is somewhat difficult to debug, given that it uses a random seed and never exits, making it very hard to submit a PR.

[neithernut]

Also reproduces with i32::MIN but not with e.g. i32::MIN + 1.

Reproducer:

use quickcheck::Arbitrary;
i32::MIN.shrink().for_each(|v| println!("{}", v));

Looking at the SignedShrinker, it's quite obvious that the shrinker is quaranteed to yield a result if it was initialized from i32::MIN: https://github.com/BurntSushi/quickcheck/blob/defde6fb0ce20b0c8c4e672aa9ae821f7d1f5b38/src/arbitrary.rs#L827 Which results in the endless iteration.

I'm honestly surprised this didn't surface any earlier.

[neithernut]

Can you verify that #296 solves your problem?

[Alexhuszagh]

Can you verify that #296 solves your problem?

Yep, it does. Wonderful, thank you.

[Alexhuszagh]

Weirdly, I'm getting another issue with a stack overflow for cases failing with the bit pattern 0b11001110111111111111111111110000 or 0b11001111000000000000000000000000 for a signed 32-bit integer type. Both these cases fail, and I'm unsure specifically which one causes the indefinite repetition.

It then attempts to shrink as follows (I believe, this is just the data passed to my test function being printed out as bits):

0b11001110111111111111111111111000
0b11001110111111111111111111111100
0b11001110111111111111111111111110
0b11001110111111111111111111111111
0b11001111000000000000000000000000
0b10000000000000000000000000000000
0b11001110100000000000000000000000
0b11001110110000000000000000000000
0b11001110111000000000000000000000
0b11001110111100000000000000000000
0b11001110111110000000000000000000
0b11001110111111000000000000000000
0b11001110111111100000000000000000
0b11001110111111110000000000000000
0b11001110111111111000000000000000
0b11001110111111111100000000000000
0b11001110111111111110000000000000
0b11001110111111111111000000000000
0b11001110111111111111100000000000
0b11001110111111111111110000000000
0b11001110111111111111111000000000
0b11001110111111111111111100000000
0b11001110111111111111111110000000
0b11001110111111111111111111000000
0b11001110111111111111111111100000

This repeats indefinitely until there's a fatal error due to a stack overflow. I've been using your branch of proptest until the changes are merged, but this also reproduces on the main quickcheck repository. This also reproduces on v1.0.3, so it should be another issue with the shrinker.

[neithernut]

Can you be more specific about what types you use? i32? u32?

Also, I this may warrant a new issue, though it definitely looks related.

[Alexhuszagh]

Can you be more specific about what types you use? i32? u32?

Also, I this may warrant a new issue, though it definitely looks related.

I'm using f32, so it should be using a signed shrinker. I'll see if I can create a minimal failing case.

[neithernut]

Looks like I failed to fix the issue properly.

The shrinker is bounded but it does yield the original value, which results in an endless recursion as quickcheck executes the test with the exact same value again and again. As I can't reproduce the issue with a i32 I suspect some information loss in the conversion to f32. I'll have another look later.

[neithernut]

I just applied some additional band-aids. Could you check whether you can reproduce the issue with 0c279d6f15d262b471d94ac70ad4e4d1ac5b6fad (part of #296)?

[Alexhuszagh]

@neithernut Seems like it has, both #285 and this bug seem to be fixed by the changes. I can't reproduce it exactly with the cases, since running 50 or so repeats never had had those specific bit patterns occur, however, all the failing cases in the shrinker before had a bit pattern of 0b1100111XXXXXXXXXXXXXXXXXXXXXXXXX, so I created the following test:

#[cfg(test)]
mod tests {
    use quickcheck::quickcheck;

    quickcheck! {
        fn case_failure(a: f32) -> bool {
            let mask = 0b11001110000000000000000000000000;
            a.to_bits() & mask != mask
        }
    }
}

The minimum failing case, as expected was 0b11001110000000000000000000000000, or -536870900.0, which is fixed in your branch, but overflows the stack in the master branch. This now seems to be a further extension of #285.

0b11001000000000000000000000000000 does not cause a stack overflow on either, while 0b11001100000000000000000000000000 does produce a stack overflow for the master branch, so I believe this is a useful metric that the issue is fixed.

[KamilaBorowska]

I get an infinite loop with the following test case:

quickcheck! {
    fn something(x: f64) -> bool {
        let x = x.abs() % 355.0;
        x.cosh().acosh() - x < 1e-10
    }
}

[neithernut]

I have trouble reproducing the issue (both on master and with #296). Does this happen with some specific initial values of x?

[KamilaBorowska]

It's random for me :(, may necessitate running the test multiple times.

[neithernut]

To be clear: it does reproduce on master. That much is expected since no fix was merged since the original issue report. However, I dedicated quite a bit of CPU time in an attempt to reproduce the issue with the changes in #296 applied, without any success. So... can you actually reproduce your issue with the fix(es) in #296 or "only" on current master (aka tag 1.0.3 aka defde6fb0ce20b0c8c4e672aa9ae821f7d1f5b38)?

[KamilaBorowska]

Yeah, I only reproduced this with 1.0.3.

[fosskers]

It seems this might still be an issue?

[neithernut]

296 was supposed to fix this, but it was never merged.

...and that's not the only issue still open with an existing fix. quickcheck appears to be practically abandoned. Depending on your situation, you may want to consider alternatives such as proptest or qcheck (which is a fork).

[fosskers]

Related: https://github.com/image-rs/imageproc/pull/522