Shrink counter example in case of exception

[choeger]

Right now, if your code throws an exception you are drowning in the output, it would be nice to be able to shrink the counterexample in such a case as well.

e.g.:

% cat feature.ml                                                                                                                                           :(
open QCheck

let foo_length l = if (List.mem 42 l) then raise (Failure "I cannot ...") else List.length l

let test =
  QCheck.(
     Test.make
    ~name:"Extremely Optimized Length Method" ~count:10_000 ~max_fail:3
    (list small_int)
    (fun l -> foo_length l = List.length l))

let () =
  QCheck.Test.check_exn test

% ocamlbuild -package qcheck feature.native
Finished, 4 targets (0 cached) in 00:00:00.
% ./feature.native 
Fatal error: exception test `Extremely Optimized Length Method`
  raised exception `Failure("I cannot ...")`
  on [5; 2; 3; 7; 7; 25; 9; 9; 1; 4; 8; 1; 233; 516; 7; 640; 1771; 61; 113; 0; 92; 242; 5; 798; 90; 2; 430; 4; 7; 667; 8; 3; 64; 962; 95; 54; 722; 7; 8; 8; 819; 0; 8; 7; 1; 3; 507; 2359; 66; 321; 6; 4; 3; 335; 0; 77; 1; 6; 0; 6; 8; 29; 2; 929; 251; 48; 9; 25; 90; 6; 88; 4; 867; 47; 0; 0; 31; 5; 41; 80; 43; 965; 71; 337; 89; 9; 720; 2; 67; 329; 9; 4; 38; 57; 81; 5; 125; 8; 0; 7; 80; 7; 6; 6; 6467; 57; 275; 1; 33; 925; 8; 6; 6; 8; 8; 2; 7; 435; 0; 6; 29; 388; 87; 96; 9; 9; 7592; 9; 4; 220; 2; 61; 6; 615; 2; 9; 99; 478; 78; 308; 0; 229; 59; 2; 14; 92; 6; 27; 81; 6; 2; 200; 32; 1; 6; 8; 8; 5; 15; 867; 2; 996; 7988; 5; 7; 519; 879; 5; 8; 2; 2; 369; 8; 6; 670; 6726; 2; 60; 74; 0; 7383; 27; 4; 9; 7; 49; 4; 6; 97; 2; 15; 8; 8662; 9; 85; 50; 91; 5; 90; 7; 68; 37; 1325; 6; 5; 1; 4; 165; 0; 998; 6; 3; 35; 40; 1; 1245; 849; 8; 12; 73; 8; 7; 5754; 31; 195; 1; 54; 4; 4518; 4; 72; 409; 3; 85; 803; 13; 5; 3; 860; 5734; 934; 3; 24; 944; 84; 0; 21; 957; 272; 411; 6; 4; 5; 7; 0; 0; 6; 5; 5; 4; 92; 872; 5; 48; 25; 2; 7; 7; 8; 13; 2355; 382; 7; 5; 8; 891; 96; 99; 2689; 85; 9; 4; 6; 5; 65; 57; 289; 3; 9292; 41; 7; 54; 3; 640; 6; 10; 29; 1243; 1; 1; 3; 81; 2; 9; 9; 4; 23; 8; 4; 5; 8; 8860; 2; 0; 5; 0; 1; 73; 1; 25; 883; 6; 1; 6; 21; 3; 9; 1; 57; 901; 515; 6; 121; 3; 738; 264; 8; 83; 25; 2; 24; 66; 13; 2; 0; 5687; 5; 7; 12; 9; 5; 86; 14; 2; 4; 97; 0; 653; 695; 7; 8; 218; 1; 8; 0; 6; 6; 2; 218; 7; 481; 4; 35; 741; 57; 92; 414; 0; 5; 4; 701; 0; 56; 9; 87; 7; 2; 7; 94; 7; 1; 868; 59; 48; 7; 8; 84; 3849; 9; 686; 68; 1728; 15; 4; 626; 0; 7; 18; 422; 2; 4; 788; 833; 1; 44; 566; 811; 7; 9; 5; 2; 333; 8849; 9719; 462; 83; 3; 3; 4; 8454; 4; 0; 4; 2; 8; 525; 9; 661; 8; 52; 3; 51; 95; 2; 7; 681; 4312; 9; 82; 34; 5; 660; 611; 7; 0; 9; 940; 15; 500; 6; 0; 98; 86; 743; 6083; 5572; 4; 0; 7046; 7; 723; 5; 2; 4; 340; 9; 48; 4; 7; 1; 91; 488; 92; 7; 683; 65; 1; 5; 256; 5; 7; 8; 7; 3; 97; 2; 3; 977; 53; 1; 5; 0; 6; 4; 69; 792; 0; 230; 6; 1; 8; 866; 3; 35; 59; 4; 0; 1; 1; 466; 604; 776; 5; 6; 6; 9; 2; 8; 2384; 1; 497; 0; 1; 1; 6; 35; 5; 1; 3; 93; 5; 2; 81; 8; 4; 2; 7897; 152; 5981; 0; 4; 6; 42; 1; 98; 71; 69; 5; 968; 711; 0; 6; 1311; 206; 958; 425; 5; 3; 1; 39; 4; 6; 27; 51; 0; 267; 90; 64; 50; 237; 3; 1; 147; 3; 8; 83; 8; 484; 0; 4; 7; 3; 86; 50; 8; 80; 26; 8; 6; 51; 2; 0; 5; 8308; 0; 3; 195; 418; 91; 57; 6; 1; 641; 2; 7; 360; 953; 0; 74; 752; 8; 20; 6; 83; 8; 63; 4; 883; 123; 85; 577; 0; 83; 4; 27; 70; 1; 9; 51; 129; 304; 935; 11; 6; 1; 0; 7; 5; 24; 48; 5; 0; 278; 5; 5; 0; 830; 157; 3; 97; 4; 45; 12; 5; 7; 4; 2; 3; 6441; 102; 606; 40; 441; 3; 2; 5; 4; 11; 0; 261; 9; 53; 64; 1; 12; 669; 2; 3; 9; 2; 5; 1; 6; 9; 90; 6; 71; 8; 164; 107; 93; 44; 93; 56; 6; 9573; 9; 15; 6; 7; 94; 3; 78; 41; 16; 335; 7; 444; 42; 3; 66; 78; 32; 300; 7; 40; 9; 6; 0; 704; 6; 0; 694; 3; 6; 131; 3; 65; 6; 257; 6; 373; 73; 8; 44; 800; 1; 621; 3; 9; 89; 80; 514; 2; 84; 438; 5992; 1892; 340; 7984; 4; 6; 2; 352; 3; 940; 657; 3; 6; 80; 106; 892; 3; 8; 51; 70; 1; 6; 3; 99; 6; 0; 6; 29; 43; 7; 55; 810; 66; 4; 832; 7; 32; 948; 6; 6; 7; 27; 8; 1; 7; 76; 3; 628; 90; 26; 19; 957; 2; 4; 27; 1; 5; 886; 82; 171; 2; 7; 4; 390; 9; 45; 4; 765; 10; 2; 5; 8; 3; 10; 735; 3; 2; 606; 3; 9; 7; 35; 2; 9; 10; 282; 5; 3; 1; 13; 39; 353; 93; 9; 954; 2957; 2575; 67; 547; 81; 29; 212; 3; 4; 2; 3; 35; 3; 6; 4; 64; 14; 63; 293; 0; 2; 3; 4; 727; 25; 5; 0; 4; 41; 8; 49; 7; 3; 650; 51; 5; 2; 7; 7; 5; 1; 955; 3; 2; 0; 8; 8; 7; 687; 7; 5; 851; 514; 1]
%                                                                                                                                                          :(

[c-cube]

A question is: if a test fails by raising an exception, shall we shrink it until it doesn't fail at all, or until it fails another way? In other words, if an input of size 1000 triggers a raise Not_found error, and after shrinking to size 500 it doesn't raise anymore but doesn't pass the test either, should we stop at 501 or continue shrinking?

[c-cube]

@choeger can you try with the newest commit?

[c-cube]

@choeger ping, I'd like to close this issue if it's solved.