search

gfredericks / test.chuck

A utility library for test.check
Eclipse Public License 1.0
214 stars 26 forks source link

Output seed information when `checking` fails during generation #78

Open SevereOverfl0w opened 9 months ago

SevereOverfl0w commented 9 months ago

If generation fails during the bindings phase of the checking macro any seed information is lost. This makes it difficult to reproduce failures.

(deftest fails
  (checking
    "a"
    [i gen/small-integer
     :let [x (/ 1 i)]]
    (is (pos? x))))

Produces this output:

ERROR in (fails) (Numbers.java:188)
Uncaught exception, not in assertion.
expected: nil
  actual: java.lang.ArithmeticException: Divide by zero
 at clojure.lang.Numbers.divide (Numbers.java:188)
    clojure.lang.Numbers.divide (Numbers.java:3877)
    com.gfredericks.test.chuck.clojure_test_test$fn__4276$fn__4277$fn__4284.invoke (clojure_test_test.cljc:55)
    clojure.test.check.rose_tree$fmap.invokeStatic (rose_tree.cljc:77)
    clojure.test.check.rose_tree$fmap.invoke (rose_tree.cljc:73)
    clojure.test.check.rose_tree$fmap$fn__618.invoke (rose_tree.cljc:77)
    clojure.core$map$fn__5884.invoke (core.clj:2759)
    clojure.lang.LazySeq.sval (LazySeq.java:42)
    clojure.lang.LazySeq.seq (LazySeq.java:51)
    clojure.lang.RT.seq (RT.java:535)
    clojure.core$seq__5419.invokeStatic (core.clj:139)
    clojure.core$map$fn__5884.invoke (core.clj:2750)
    clojure.lang.LazySeq.sval (LazySeq.java:42)
    clojure.lang.LazySeq.seq (LazySeq.java:51)
    clojure.lang.RT.seq (RT.java:535)
    clojure.core$seq__5419.invokeStatic (core.clj:139)
    clojure.core/seq (core.clj:139)
    clojure.test.check.rose_tree$permutations$iter__629__635$fn__636$iter__631__640$fn__641.invoke (rose_tree.cljc:102)
    clojure.lang.LazySeq.sval (LazySeq.java:42)
    clojure.lang.LazySeq.seq (LazySeq.java:51)
    clojure.lang.RT.seq (RT.java:535)
    clojure.core$seq__5419.invokeStatic (core.clj:139)
    clojure.core/seq (core.clj:139)
    clojure.test.check.rose_tree$permutations$iter__629__635$fn__636.invoke (rose_tree.cljc:102)
    clojure.lang.LazySeq.sval (LazySeq.java:42)
    clojure.lang.LazySeq.seq (LazySeq.java:51)
    clojure.lang.RT.seq (RT.java:535)
    clojure.core$seq__5419.invokeStatic (core.clj:139)
    clojure.core$map$fn__5884.invoke (core.clj:2750)
    clojure.lang.LazySeq.sval (LazySeq.java:42)
    clojure.lang.LazySeq.seq (LazySeq.java:51)
    clojure.lang.RT.seq (RT.java:535)
    clojure.core$seq__5419.invokeStatic (core.clj:139)
    clojure.core$map$fn__5884.invoke (core.clj:2750)
    clojure.lang.LazySeq.sval (LazySeq.java:42)
    clojure.lang.LazySeq.seq (LazySeq.java:51)
    clojure.lang.RT.seq (RT.java:535)
    clojure.core$seq__5419.invokeStatic (core.clj:139)
    clojure.core$empty_QMARK_.invokeStatic (core.clj:6195)
    clojure.core$empty_QMARK_.invoke (core.clj:6195)
    clojure.test.check$shrink_loop.invokeStatic (check.cljc:263)
    clojure.test.check$shrink_loop.invoke (check.cljc:242)
    clojure.test.check$failure.invokeStatic (check.cljc:314)
    clojure.test.check$failure.invoke (check.cljc:297)
    clojure.test.check$quick_check.invokeStatic (check.cljc:228)
    clojure.test.check$quick_check.doInvoke (check.cljc:59)
    clojure.lang.RestFn.invoke (RestFn.java:425)
    clojure.lang.AFn.applyToHelper (AFn.java:156)
    clojure.lang.RestFn.applyTo (RestFn.java:132)
    clojure.core$apply.invokeStatic (core.clj:671)
    clojure.core$apply.invoke (core.clj:662)
    com.gfredericks.test.chuck.clojure_test_test$fn__4276$fn__4277.invoke (clojure_test_test.cljc:52)
    com.gfredericks.test.chuck.clojure_test$_testing.invokeStatic (clojure_test.cljc:103)
    com.gfredericks.test.chuck.clojure_test$_testing.invoke (clojure_test.cljc:101)
    com.gfredericks.test.chuck.clojure_test_test$fn__4276.invokeStatic (clojure_test_test.cljc:52)
    com.gfredericks.test.chuck.clojure_test_test/fn (clojure_test_test.cljc:51)
    clojure.test$test_var$fn__9761.invoke (test.clj:717)
    clojure.test$test_var.invokeStatic (test.clj:717)
    clojure.test$test_var.invoke (test.clj:708)
    clojure.test$test_vars$fn__9787$fn__9792.invoke (test.clj:735)
    clojure.test$default_fixture.invokeStatic (test.clj:687)
    clojure.test$default_fixture.invoke (test.clj:683)
    clojure.test$test_vars$fn__9787.invoke (test.clj:735)
    clojure.test$default_fixture.invokeStatic (test.clj:687)
    clojure.test$default_fixture.invoke (test.clj:683)
    clojure.test$test_vars.invokeStatic (test.clj:731)
    clojure.test$test_all_vars.invokeStatic (test.clj:737)
    clojure.test$test_ns.invokeStatic (test.clj:758)
    clojure.test$test_ns.invoke (test.clj:743)
    user$eval227$fn__350.invoke (form-init6834647596943620637.clj:1)
    clojure.lang.AFn.applyToHelper (AFn.java:156)
    clojure.lang.AFn.applyTo (AFn.java:144)
    clojure.core$apply.invokeStatic (core.clj:669)
    clojure.core$apply.invoke (core.clj:662)
    leiningen.core.injected$compose_hooks$fn__154.doInvoke (form-init6834647596943620637.clj:1)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.core$apply.invokeStatic (core.clj:667)
    clojure.core$apply.invoke (core.clj:662)
    leiningen.core.injected$run_hooks.invokeStatic (form-init6834647596943620637.clj:1)
    leiningen.core.injected$run_hooks.invoke (form-init6834647596943620637.clj:1)
    leiningen.core.injected$prepare_for_hooks$fn__159$fn__160.doInvoke (form-init6834647596943620637.clj:1)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.lang.AFunction$1.doInvoke (AFunction.java:31)
    clojure.lang.RestFn.invoke (RestFn.java:408)
    clojure.core$map$fn__5884.invoke (core.clj:2759)
    clojure.lang.LazySeq.sval (LazySeq.java:42)
    clojure.lang.LazySeq.seq (LazySeq.java:51)
    clojure.lang.Cons.next (Cons.java:39)
    clojure.lang.RT.boundedLength (RT.java:1793)
    clojure.lang.RestFn.applyTo (RestFn.java:130)
    clojure.core$apply.invokeStatic (core.clj:669)
    clojure.test$run_tests.invokeStatic (test.clj:768)
    clojure.test$run_tests.doInvoke (test.clj:768)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.core$apply.invokeStatic (core.clj:667)
    clojure.core$apply.invoke (core.clj:662)
    user$eval227$fn__360$fn__413.invoke (form-init6834647596943620637.clj:1)
    user$eval227$fn__360$fn__361.invoke (form-init6834647596943620637.clj:1)
    user$eval227$fn__360.invoke (form-init6834647596943620637.clj:1)
    user$eval227.invokeStatic (form-init6834647596943620637.clj:1)
    user$eval227.invoke (form-init6834647596943620637.clj:1)
    clojure.lang.Compiler.eval (Compiler.java:7181)
    clojure.lang.Compiler.eval (Compiler.java:7171)
    clojure.lang.Compiler.load (Compiler.java:7640)
    clojure.lang.Compiler.loadFile (Compiler.java:7578)
    clojure.main$load_script.invokeStatic (main.clj:475)
    clojure.main$init_opt.invokeStatic (main.clj:477)
    clojure.main$init_opt.invoke (main.clj:477)
    clojure.main$initialize.invokeStatic (main.clj:508)
    clojure.main$null_opt.invokeStatic (main.clj:542)
    clojure.main$null_opt.invoke (main.clj:539)
    clojure.main$main.invokeStatic (main.clj:664)
    clojure.main$main.doInvoke (main.clj:616)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.lang.Var.applyTo (Var.java:705)
    clojure.main.main (main.java:40)

Ran 5 tests containing 20 assertions.
0 failures, 1 errors.
Tests failed.
SevereOverfl0w commented 9 months ago

After poking at this a little bit, it seems that we'd need upstream changes in test.check to make this work. When doing a call-gen in quick-check the exception would need to be caught and some handling put in place. Calling :reporter-fn with any relevant information before rethrowing the exception seems like the minimal set of changes to me.

gfredericks commented 8 months ago

Okay; we can leave this open and you or somebody else could pursue those changes in test.check.

frenchy64 commented 4 months ago

@gfredericks I took a shot at it. https://github.com/gfredericks/test.chuck/pull/80

What do you think about the test.check changes? Worth proposing on TCHECK-159?

https://github.com/frenchy64/test.check/pull/1/files

gfredericks commented 4 months ago

I'm not maintaining test.check anymore, so my opinion doesn't matter too much.

frenchy64 commented 4 months ago

Firstly, thanks @gfredericks for all the years of maintainership. In my book, test.check under your direction has been the most successful Clojure library of them all. All that to say, I really appreciate your work and congratulations on an amazing tenure.

Secondly, my hit rate on getting patches merged from JIRA is probably about 5%. Would you consider a patch to test.chuck with this definition of quick-check, and then using it in checking?

gfredericks commented 4 months ago

Thanks for the kind words!

Copypasting the definition of quickcheck into this library sounds like it could gen gnarly in the future if implementation details in test.check change.

Is there no way to do this in test.chuck, perhaps by wrapping generators with a fmap that has a try/catch, and then wrapping the checking expression similarly? That'd amount to turning a generator-exception into a test failure, which is a bit weird, but would at least result in the seed being output.

frenchy64 commented 4 months ago

@gfredericks great idea, I will try it.