search

gfredericks / test.chuck

A utility library for test.check
Eclipse Public License 1.0
214 stars 26 forks source link

Should catch unexpected exceptions and fail the test suite #59

Open metametadata opened 5 years ago

metametadata commented 5 years ago

Example:

(ns unit.a.b
  (:require 
    [clojure.test.check.generators :as gen]
    [com.gfredericks.test.chuck.clojure-test :as chuck]
    [clojure.test :as ct :refer :all]))

(deftest foo
  (chuck/checking
    "bar"
    10
    [x gen/int]
    (throw (ex-info "oops" {}))))

Expected:

<somehow report the exception with stacktrace>
...
0 failures, 1 errors.
Tests failed.

Actual:

Tests failed, smallest case: [{x 0}] 
Seed 1534810136116

Ran 1 tests containing 0 assertions.
0 failures, 0 errors.

My current workaround is to redefine the reporting method so that it always "fails" the test via (is false):

(defmethod ct/report ::chuck/shrunk
  [m]
  (is false "Property test failed, see logs for more info")
  (println
    ;"\nCase:\n" (with-out-str (pp/pprint (:fail m)))
    "\nShrunk:\n" (with-out-str (pp/pprint (-> m :shrunk :smallest)))
    (str "\n(num-tests: " (:num-tests m)
         ", total-nodes-visited: " (-> m :shrunk :total-nodes-visited)
         ", depth: " (-> m :shrunk :depth)
         ", failing-size: " (:failing-size m)
         ", seed: " (:seed m) ")"))
  (println "Error:" (pr-str (-> m :result-data :clojure.test.check.properties/error))))

Output:

FAIL in (foo) (b.clj:21)
bar
Property test failed, see logs for more info
expected: false
  actual: false

Shrunk:
 [{x 0}]

(num-tests: 1, total-nodes-visited: 0, depth: 0, failing-size: 0, seed: 1534810550729)
Error: #error {
 :cause "oops"
 :data {}
 :via
 [{:type clojure.lang.ExceptionInfo
   :message "oops"
   :data {}
   :at [clojure.core$ex_info invokeStatic "core.clj" 4739]}]
 :trace
 [[clojure.core$ex_info invokeStatic "core.clj" 4739]
  [clojure.core$ex_info invoke "core.clj" 4739]
  ...]}

Ran 1 tests containing 1 assertions.
1 failures, 0 errors.
Tests failed.

Workaround output for the expected failure:

(deftest foo
  (chuck/checking
    "bar"
    1000
    [x gen/int]
    (is (not= 5 x))))
lein test :only unit.a.b/foo

FAIL in (foo) (b.clj:21)
bar
Property test failed, see logs for more info
expected: false
  actual: false

Shrunk:
 [{x 5}]

(num-tests: 114, total-nodes-visited: 3, depth: 0, failing-size: 113, seed: 1534810835278)
Error: nil

lein test :only unit.a.b/foo

FAIL in (foo) (b.clj:179)
bar
expected: (not= 5 x)
  actual: (not (not= 5 5))

Ran 1 tests containing 2 assertions.
2 failures, 0 errors.
Tests failed.
gfredericks commented 5 years ago

I'm having trouble reproducing this.

E.g.:

$ for file in $(ls); do echo === $file ===; cat $file | sed 's/^/> /'; done && clojure the-test.clj | tail
=== deps.edn ===
> {:deps {com.gfredericks/test.chuck {:mvn/version "0.2.9"}
>         org.clojure/test.check     {:mvn/version "0.9.0"}}}
=== the-test.clj ===
> (ns unit.a.b
>   (:require
>     [clojure.test.check.generators :as gen]
>     [com.gfredericks.test.chuck.clojure-test :as chuck]
>     [clojure.test :as ct :refer :all]))
> 
> (deftest foo
>   (chuck/checking
>     "bar"
>     10
>     [x gen/int]
>     (throw (ex-info "oops" {}))))
> 
> (run-tests)
  [clojure.main$script_opt invokeStatic "main.clj" 338]
  [clojure.main$script_opt invoke "main.clj" 333]
  [clojure.main$main invokeStatic "main.clj" 424]
  [clojure.main$main doInvoke "main.clj" 387]
  [clojure.lang.RestFn applyTo "RestFn.java" 137]
  [clojure.lang.Var applyTo "Var.java" 702]
  [clojure.main main "main.java" 37]]}))

Ran 1 tests containing 1 assertions.
1 failures, 0 errors.
metametadata commented 5 years ago

I see, thanks. I used lein test inside the existing project, so maybe something else in lein user profile or project.clj causes the issue. I'll reply when I find the real cause.

metametadata commented 5 years ago

Looks like the problem is reproducible because of [org.clojure/test.check "0.10.0-alpha2"]. Exception is caught as expected in 0.9.0 and latest version 0.10.0-alpha3.

Unfortunately, the default reporting is not very helpful: it is very long and noisey (~300 lines in 0.9.0 and ~500 in 0.10.0-alpha3, mostly stacktraces) with smallest shrunk case somewhere in the middle. So I guess I still have to redefine the reporting method.

metametadata commented 5 years ago

Ah, seems like the redefined ct/report is able to catch unexpected exception only in the "broken" [org.clojure/test.check "0.10.0-alpha2"]. It's not convenient, because sometimes the shrunk case is the reason for the exception, but it's very difficult to find :smallest in the output. Is it possible to customize exception reporting somehow or shall I issue a new feature request?

gfredericks commented 5 years ago

I'm confused by these two statements, which seem contradictory:

Exception is caught as expected in 0.9.0 and latest version 0.10.0-alpha3

seems like the redefined ct/report is able to catch unexpected exception only in the "broken" [org.clojure/test.check "0.10.0-alpha2"]

If you could post some code (and deps.edn) that doesn't do what you want and explain what you'd like it to do, I can sort this out quicker.

metametadata commented 5 years ago

Sure, sorry for confusion.

Exception is caught as expected in 0.9.0 and latest version 0.10.0-alpha3.

The library catches the unexpected exception and reports it. So it is good.

seems like the redefined ct/report is able to catch unexpected exception only in the "broken" [org.clojure/test.check "0.10.0-alpha2"]

I meant that I can't redefine what the library outputs when it reports the unexepected error. Unless the broken "0.10.0-alpha2" is used (in such case my own custom defmethod ct/report ::chuck/shrunk catches the unexpected exception, for some reason).

The example:

; deps.clj
{:deps {com.gfredericks/test.chuck {:mvn/version "0.2.9"}
        org.clojure/test.check     {:mvn/version "0.10.0-alpha3"}}}
; the-test.clj
(ns the-test
  (:require
    [clojure.test.check.generators :as gen]
    [com.gfredericks.test.chuck.clojure-test :as chuck]
    [clojure.test :as ct :refer :all]))

(deftest foo
  (chuck/checking
    "bar"
    10
    [x gen/int]
    (throw (ex-info "oops" {}))))

(run-tests)

This is the output:

ᐅ clojure the-test.clj

Testing the-test

FAIL in (foo) (clojure_test.cljc:30)
bar
{:shrunk {:total-nodes-visited 0, :depth 0, :pass? false, :result #error {
 :cause oops
 :data {}
 :via
 [{:type clojure.lang.ExceptionInfo
   :message oops
   :data {}
   :at [clojure.core$ex_info invokeStatic core.clj 4739]}]
 :trace
 [[clojure.core$ex_info invokeStatic core.clj 4739]
  [clojure.core$ex_info invoke core.clj 4739]
  [the_test$fn__3748$fn__3749$fn__3753$fn__3755 invoke the-test.clj 12]
  [com.gfredericks.test.chuck.clojure_test$capture_reports_STAR_ invokeStatic clojure_test.cljc 67]
  [com.gfredericks.test.chuck.clojure_test$capture_reports_STAR_ invoke clojure_test.cljc 63]
  [the_test$fn__3748$fn__3749$fn__3753 invoke the-test.clj 8]
  [clojure.lang.AFn applyToHelper AFn.java 154]
  [clojure.lang.AFn applyTo AFn.java 144]
  [clojure.core$apply invokeStatic core.clj 657]
  [clojure.core$apply invoke core.clj 652]
  [clojure.test.check.properties$apply_gen$fn__1113$fn__1114 invoke properties.cljc 30]
  [clojure.test.check.properties$apply_gen$fn__1113 invoke properties.cljc 29]
  [clojure.test.check.rose_tree$fmap invokeStatic rose_tree.cljc 77]
  [clojure.test.check.rose_tree$fmap invoke rose_tree.cljc 73]
  [clojure.test.check.generators$fmap$fn__463 invoke generators.cljc 102]
  [clojure.test.check.generators$gen_fmap$fn__437 invoke generators.cljc 58]
  [clojure.test.check.generators$call_gen invokeStatic generators.cljc 42]
  [clojure.test.check.generators$call_gen invoke generators.cljc 38]
  [clojure.test.check$quick_check invokeStatic check.cljc 126]
  [clojure.test.check$quick_check doInvoke check.cljc 59]
  [clojure.lang.RestFn invoke RestFn.java 425]
  [clojure.lang.AFn applyToHelper AFn.java 156]
  [clojure.lang.RestFn applyTo RestFn.java 132]
  [clojure.core$apply invokeStatic core.clj 661]
  [clojure.core$apply invoke core.clj 652]
  [the_test$fn__3748$fn__3749 invoke the-test.clj 8]
  [com.gfredericks.test.chuck.clojure_test$_testing invokeStatic clojure_test.cljc 102]
  [com.gfredericks.test.chuck.clojure_test$_testing invoke clojure_test.cljc 100]
  [the_test$fn__3748 invokeStatic the-test.clj 8]
  [the_test$fn__3748 invoke the-test.clj 7]
  [clojure.test$test_var$fn__9209 invoke test.clj 716]
  [clojure.test$test_var invokeStatic test.clj 716]
  [clojure.test$test_var invoke test.clj 707]
  [clojure.test$test_vars$fn__9235$fn__9240 invoke test.clj 734]
  [clojure.test$default_fixture invokeStatic test.clj 686]
  [clojure.test$default_fixture invoke test.clj 682]
  [clojure.test$test_vars$fn__9235 invoke test.clj 734]
  [clojure.test$default_fixture invokeStatic test.clj 686]
  [clojure.test$default_fixture invoke test.clj 682]
  [clojure.test$test_vars invokeStatic test.clj 730]
  [clojure.test$test_all_vars invokeStatic test.clj 736]
  [clojure.test$test_ns invokeStatic test.clj 757]
  [clojure.test$test_ns invoke test.clj 742]
  [clojure.core$map$fn__5587 invoke core.clj 2747]
  [clojure.lang.LazySeq sval LazySeq.java 40]
  [clojure.lang.LazySeq seq LazySeq.java 49]
  [clojure.lang.Cons next Cons.java 39]
  [clojure.lang.RT boundedLength RT.java 1785]
  [clojure.lang.RestFn applyTo RestFn.java 130]
  [clojure.core$apply invokeStatic core.clj 659]
  [clojure.test$run_tests invokeStatic test.clj 767]
  [clojure.test$run_tests doInvoke test.clj 767]
  [clojure.lang.RestFn invoke RestFn.java 408]
  [clojure.test$run_tests invokeStatic test.clj 772]
  [clojure.test$run_tests invoke test.clj 767]
  [the_test$eval3773 invokeStatic the-test.clj 14]
  [the_test$eval3773 invoke the-test.clj 14]
  [clojure.lang.Compiler eval Compiler.java 7062]
  [clojure.lang.Compiler load Compiler.java 7514]
  [clojure.lang.Compiler loadFile Compiler.java 7452]
  [clojure.main$load_script invokeStatic main.clj 278]
  [clojure.main$script_opt invokeStatic main.clj 338]
  [clojure.main$script_opt invoke main.clj 333]
  [clojure.main$main invokeStatic main.clj 424]
  [clojure.main$main doInvoke main.clj 387]
  [clojure.lang.RestFn applyTo RestFn.java 137]
  [clojure.lang.Var applyTo Var.java 702]
  [clojure.main main main.java 37]]}, :result-data #:clojure.test.check.properties{:error #error {
 :cause oops
 :data {}
 :via
 [{:type clojure.lang.ExceptionInfo
   :message oops
   :data {}
   :at [clojure.core$ex_info invokeStatic core.clj 4739]}]
 :trace
 [[clojure.core$ex_info invokeStatic core.clj 4739]
  [clojure.core$ex_info invoke core.clj 4739]
  [the_test$fn__3748$fn__3749$fn__3753$fn__3755 invoke the-test.clj 12]
  [com.gfredericks.test.chuck.clojure_test$capture_reports_STAR_ invokeStatic clojure_test.cljc 67]
  [com.gfredericks.test.chuck.clojure_test$capture_reports_STAR_ invoke clojure_test.cljc 63]
  [the_test$fn__3748$fn__3749$fn__3753 invoke the-test.clj 8]
  [clojure.lang.AFn applyToHelper AFn.java 154]
  [clojure.lang.AFn applyTo AFn.java 144]
  [clojure.core$apply invokeStatic core.clj 657]
  [clojure.core$apply invoke core.clj 652]
  [clojure.test.check.properties$apply_gen$fn__1113$fn__1114 invoke properties.cljc 30]
  [clojure.test.check.properties$apply_gen$fn__1113 invoke properties.cljc 29]
  [clojure.test.check.rose_tree$fmap invokeStatic rose_tree.cljc 77]
  [clojure.test.check.rose_tree$fmap invoke rose_tree.cljc 73]
  [clojure.test.check.generators$fmap$fn__463 invoke generators.cljc 102]
  [clojure.test.check.generators$gen_fmap$fn__437 invoke generators.cljc 58]
  [clojure.test.check.generators$call_gen invokeStatic generators.cljc 42]
  [clojure.test.check.generators$call_gen invoke generators.cljc 38]
  [clojure.test.check$quick_check invokeStatic check.cljc 126]
  [clojure.test.check$quick_check doInvoke check.cljc 59]
  [clojure.lang.RestFn invoke RestFn.java 425]
  [clojure.lang.AFn applyToHelper AFn.java 156]
  [clojure.lang.RestFn applyTo RestFn.java 132]
  [clojure.core$apply invokeStatic core.clj 661]
  [clojure.core$apply invoke core.clj 652]
  [the_test$fn__3748$fn__3749 invoke the-test.clj 8]
  [com.gfredericks.test.chuck.clojure_test$_testing invokeStatic clojure_test.cljc 102]
  [com.gfredericks.test.chuck.clojure_test$_testing invoke clojure_test.cljc 100]
  [the_test$fn__3748 invokeStatic the-test.clj 8]
  [the_test$fn__3748 invoke the-test.clj 7]
  [clojure.test$test_var$fn__9209 invoke test.clj 716]
  [clojure.test$test_var invokeStatic test.clj 716]
  [clojure.test$test_var invoke test.clj 707]
  [clojure.test$test_vars$fn__9235$fn__9240 invoke test.clj 734]
  [clojure.test$default_fixture invokeStatic test.clj 686]
  [clojure.test$default_fixture invoke test.clj 682]
  [clojure.test$test_vars$fn__9235 invoke test.clj 734]
  [clojure.test$default_fixture invokeStatic test.clj 686]
  [clojure.test$default_fixture invoke test.clj 682]
  [clojure.test$test_vars invokeStatic test.clj 730]
  [clojure.test$test_all_vars invokeStatic test.clj 736]
  [clojure.test$test_ns invokeStatic test.clj 757]
  [clojure.test$test_ns invoke test.clj 742]
  [clojure.core$map$fn__5587 invoke core.clj 2747]
  [clojure.lang.LazySeq sval LazySeq.java 40]
  [clojure.lang.LazySeq seq LazySeq.java 49]
  [clojure.lang.Cons next Cons.java 39]
  [clojure.lang.RT boundedLength RT.java 1785]
  [clojure.lang.RestFn applyTo RestFn.java 130]
  [clojure.core$apply invokeStatic core.clj 659]
  [clojure.test$run_tests invokeStatic test.clj 767]
  [clojure.test$run_tests doInvoke test.clj 767]
  [clojure.lang.RestFn invoke RestFn.java 408]
  [clojure.test$run_tests invokeStatic test.clj 772]
  [clojure.test$run_tests invoke test.clj 767]
  [the_test$eval3773 invokeStatic the-test.clj 14]
  [the_test$eval3773 invoke the-test.clj 14]
  [clojure.lang.Compiler eval Compiler.java 7062]
  [clojure.lang.Compiler load Compiler.java 7514]
  [clojure.lang.Compiler loadFile Compiler.java 7452]
  [clojure.main$load_script invokeStatic main.clj 278]
  [clojure.main$script_opt invokeStatic main.clj 338]
  [clojure.main$script_opt invoke main.clj 333]
  [clojure.main$main invokeStatic main.clj 424]
  [clojure.main$main doInvoke main.clj 387]
  [clojure.lang.RestFn applyTo RestFn.java 137]
  [clojure.lang.Var applyTo Var.java 702]
  [clojure.main main main.java 37]]}}, :time-shrinking-ms 1, :smallest [{x 0}]}, :failed-after-ms 4, :num-tests 1, :seed 1536598020405, :fail [{x 0}], :result #error {
 :cause oops
 :data {}
 :via
 [{:type clojure.lang.ExceptionInfo
   :message oops
   :data {}
   :at [clojure.core$ex_info invokeStatic core.clj 4739]}]
 :trace
 [[clojure.core$ex_info invokeStatic core.clj 4739]
  [clojure.core$ex_info invoke core.clj 4739]
  [the_test$fn__3748$fn__3749$fn__3753$fn__3755 invoke the-test.clj 12]
  [com.gfredericks.test.chuck.clojure_test$capture_reports_STAR_ invokeStatic clojure_test.cljc 67]
  [com.gfredericks.test.chuck.clojure_test$capture_reports_STAR_ invoke clojure_test.cljc 63]
  [the_test$fn__3748$fn__3749$fn__3753 invoke the-test.clj 8]
  [clojure.lang.AFn applyToHelper AFn.java 154]
  [clojure.lang.AFn applyTo AFn.java 144]
  [clojure.core$apply invokeStatic core.clj 657]
  [clojure.core$apply invoke core.clj 652]
  [clojure.test.check.properties$apply_gen$fn__1113$fn__1114 invoke properties.cljc 30]
  [clojure.test.check.properties$apply_gen$fn__1113 invoke properties.cljc 29]
  [clojure.test.check.rose_tree$fmap invokeStatic rose_tree.cljc 77]
  [clojure.test.check.rose_tree$fmap invoke rose_tree.cljc 73]
  [clojure.test.check.generators$fmap$fn__463 invoke generators.cljc 102]
  [clojure.test.check.generators$gen_fmap$fn__437 invoke generators.cljc 58]
  [clojure.test.check.generators$call_gen invokeStatic generators.cljc 42]
  [clojure.test.check.generators$call_gen invoke generators.cljc 38]
  [clojure.test.check$quick_check invokeStatic check.cljc 126]
  [clojure.test.check$quick_check doInvoke check.cljc 59]
  [clojure.lang.RestFn invoke RestFn.java 425]
  [clojure.lang.AFn applyToHelper AFn.java 156]
  [clojure.lang.RestFn applyTo RestFn.java 132]
  [clojure.core$apply invokeStatic core.clj 661]
  [clojure.core$apply invoke core.clj 652]
  [the_test$fn__3748$fn__3749 invoke the-test.clj 8]
  [com.gfredericks.test.chuck.clojure_test$_testing invokeStatic clojure_test.cljc 102]
  [com.gfredericks.test.chuck.clojure_test$_testing invoke clojure_test.cljc 100]
  [the_test$fn__3748 invokeStatic the-test.clj 8]
  [the_test$fn__3748 invoke the-test.clj 7]
  [clojure.test$test_var$fn__9209 invoke test.clj 716]
  [clojure.test$test_var invokeStatic test.clj 716]
  [clojure.test$test_var invoke test.clj 707]
  [clojure.test$test_vars$fn__9235$fn__9240 invoke test.clj 734]
  [clojure.test$default_fixture invokeStatic test.clj 686]
  [clojure.test$default_fixture invoke test.clj 682]
  [clojure.test$test_vars$fn__9235 invoke test.clj 734]
  [clojure.test$default_fixture invokeStatic test.clj 686]
  [clojure.test$default_fixture invoke test.clj 682]
  [clojure.test$test_vars invokeStatic test.clj 730]
  [clojure.test$test_all_vars invokeStatic test.clj 736]
  [clojure.test$test_ns invokeStatic test.clj 757]
  [clojure.test$test_ns invoke test.clj 742]
  [clojure.core$map$fn__5587 invoke core.clj 2747]
  [clojure.lang.LazySeq sval LazySeq.java 40]
  [clojure.lang.LazySeq seq LazySeq.java 49]
  [clojure.lang.Cons next Cons.java 39]
  [clojure.lang.RT boundedLength RT.java 1785]
  [clojure.lang.RestFn applyTo RestFn.java 130]
  [clojure.core$apply invokeStatic core.clj 659]
  [clojure.test$run_tests invokeStatic test.clj 767]
  [clojure.test$run_tests doInvoke test.clj 767]
  [clojure.lang.RestFn invoke RestFn.java 408]
  [clojure.test$run_tests invokeStatic test.clj 772]
  [clojure.test$run_tests invoke test.clj 767]
  [the_test$eval3773 invokeStatic the-test.clj 14]
  [the_test$eval3773 invoke the-test.clj 14]
  [clojure.lang.Compiler eval Compiler.java 7062]
  [clojure.lang.Compiler load Compiler.java 7514]
  [clojure.lang.Compiler loadFile Compiler.java 7452]
  [clojure.main$load_script invokeStatic main.clj 278]
  [clojure.main$script_opt invokeStatic main.clj 338]
  [clojure.main$script_opt invoke main.clj 333]
  [clojure.main$main invokeStatic main.clj 424]
  [clojure.main$main doInvoke main.clj 387]
  [clojure.lang.RestFn applyTo RestFn.java 137]
  [clojure.lang.Var applyTo Var.java 702]
  [clojure.main main main.java 37]]}, :result-data #:clojure.test.check.properties{:error #error {
 :cause oops
 :data {}
 :via
 [{:type clojure.lang.ExceptionInfo
   :message oops
   :data {}
   :at [clojure.core$ex_info invokeStatic core.clj 4739]}]
 :trace
 [[clojure.core$ex_info invokeStatic core.clj 4739]
  [clojure.core$ex_info invoke core.clj 4739]
  [the_test$fn__3748$fn__3749$fn__3753$fn__3755 invoke the-test.clj 12]
  [com.gfredericks.test.chuck.clojure_test$capture_reports_STAR_ invokeStatic clojure_test.cljc 67]
  [com.gfredericks.test.chuck.clojure_test$capture_reports_STAR_ invoke clojure_test.cljc 63]
  [the_test$fn__3748$fn__3749$fn__3753 invoke the-test.clj 8]
  [clojure.lang.AFn applyToHelper AFn.java 154]
  [clojure.lang.AFn applyTo AFn.java 144]
  [clojure.core$apply invokeStatic core.clj 657]
  [clojure.core$apply invoke core.clj 652]
  [clojure.test.check.properties$apply_gen$fn__1113$fn__1114 invoke properties.cljc 30]
  [clojure.test.check.properties$apply_gen$fn__1113 invoke properties.cljc 29]
  [clojure.test.check.rose_tree$fmap invokeStatic rose_tree.cljc 77]
  [clojure.test.check.rose_tree$fmap invoke rose_tree.cljc 73]
  [clojure.test.check.generators$fmap$fn__463 invoke generators.cljc 102]
  [clojure.test.check.generators$gen_fmap$fn__437 invoke generators.cljc 58]
  [clojure.test.check.generators$call_gen invokeStatic generators.cljc 42]
  [clojure.test.check.generators$call_gen invoke generators.cljc 38]
  [clojure.test.check$quick_check invokeStatic check.cljc 126]
  [clojure.test.check$quick_check doInvoke check.cljc 59]
  [clojure.lang.RestFn invoke RestFn.java 425]
  [clojure.lang.AFn applyToHelper AFn.java 156]
  [clojure.lang.RestFn applyTo RestFn.java 132]
  [clojure.core$apply invokeStatic core.clj 661]
  [clojure.core$apply invoke core.clj 652]
  [the_test$fn__3748$fn__3749 invoke the-test.clj 8]
  [com.gfredericks.test.chuck.clojure_test$_testing invokeStatic clojure_test.cljc 102]
  [com.gfredericks.test.chuck.clojure_test$_testing invoke clojure_test.cljc 100]
  [the_test$fn__3748 invokeStatic the-test.clj 8]
  [the_test$fn__3748 invoke the-test.clj 7]
  [clojure.test$test_var$fn__9209 invoke test.clj 716]
  [clojure.test$test_var invokeStatic test.clj 716]
  [clojure.test$test_var invoke test.clj 707]
  [clojure.test$test_vars$fn__9235$fn__9240 invoke test.clj 734]
  [clojure.test$default_fixture invokeStatic test.clj 686]
  [clojure.test$default_fixture invoke test.clj 682]
  [clojure.test$test_vars$fn__9235 invoke test.clj 734]
  [clojure.test$default_fixture invokeStatic test.clj 686]
  [clojure.test$default_fixture invoke test.clj 682]
  [clojure.test$test_vars invokeStatic test.clj 730]
  [clojure.test$test_all_vars invokeStatic test.clj 736]
  [clojure.test$test_ns invokeStatic test.clj 757]
  [clojure.test$test_ns invoke test.clj 742]
  [clojure.core$map$fn__5587 invoke core.clj 2747]
  [clojure.lang.LazySeq sval LazySeq.java 40]
  [clojure.lang.LazySeq seq LazySeq.java 49]
  [clojure.lang.Cons next Cons.java 39]
  [clojure.lang.RT boundedLength RT.java 1785]
  [clojure.lang.RestFn applyTo RestFn.java 130]
  [clojure.core$apply invokeStatic core.clj 659]
  [clojure.test$run_tests invokeStatic test.clj 767]
  [clojure.test$run_tests doInvoke test.clj 767]
  [clojure.lang.RestFn invoke RestFn.java 408]
  [clojure.test$run_tests invokeStatic test.clj 772]
  [clojure.test$run_tests invoke test.clj 767]
  [the_test$eval3773 invokeStatic the-test.clj 14]
  [the_test$eval3773 invoke the-test.clj 14]
  [clojure.lang.Compiler eval Compiler.java 7062]
  [clojure.lang.Compiler load Compiler.java 7514]
  [clojure.lang.Compiler loadFile Compiler.java 7452]
  [clojure.main$load_script invokeStatic main.clj 278]
  [clojure.main$script_opt invokeStatic main.clj 338]
  [clojure.main$script_opt invoke main.clj 333]
  [clojure.main$main invokeStatic main.clj 424]
  [clojure.main$main doInvoke main.clj 387]
  [clojure.lang.RestFn applyTo RestFn.java 137]
  [clojure.lang.Var applyTo Var.java 702]
  [clojure.main main main.java 37]]}}, :failing-size 0, :pass? false}
expected: (not-exception? (:result result))
  actual: (not (not-exception? #error {
 :cause "oops"
 :data {}
 :via
 [{:type clojure.lang.ExceptionInfo
   :message "oops"
   :data {}
   :at [clojure.core$ex_info invokeStatic "core.clj" 4739]}]
 :trace
 [[clojure.core$ex_info invokeStatic "core.clj" 4739]
  [clojure.core$ex_info invoke "core.clj" 4739]
  [the_test$fn__3748$fn__3749$fn__3753$fn__3755 invoke "the-test.clj" 12]
  [com.gfredericks.test.chuck.clojure_test$capture_reports_STAR_ invokeStatic "clojure_test.cljc" 67]
  [com.gfredericks.test.chuck.clojure_test$capture_reports_STAR_ invoke "clojure_test.cljc" 63]
  [the_test$fn__3748$fn__3749$fn__3753 invoke "the-test.clj" 8]
  [clojure.lang.AFn applyToHelper "AFn.java" 154]
  [clojure.lang.AFn applyTo "AFn.java" 144]
  [clojure.core$apply invokeStatic "core.clj" 657]
  [clojure.core$apply invoke "core.clj" 652]
  [clojure.test.check.properties$apply_gen$fn__1113$fn__1114 invoke "properties.cljc" 30]
  [clojure.test.check.properties$apply_gen$fn__1113 invoke "properties.cljc" 29]
  [clojure.test.check.rose_tree$fmap invokeStatic "rose_tree.cljc" 77]
  [clojure.test.check.rose_tree$fmap invoke "rose_tree.cljc" 73]
  [clojure.test.check.generators$fmap$fn__463 invoke "generators.cljc" 102]
  [clojure.test.check.generators$gen_fmap$fn__437 invoke "generators.cljc" 58]
  [clojure.test.check.generators$call_gen invokeStatic "generators.cljc" 42]
  [clojure.test.check.generators$call_gen invoke "generators.cljc" 38]
  [clojure.test.check$quick_check invokeStatic "check.cljc" 126]
  [clojure.test.check$quick_check doInvoke "check.cljc" 59]
  [clojure.lang.RestFn invoke "RestFn.java" 425]
  [clojure.lang.AFn applyToHelper "AFn.java" 156]
  [clojure.lang.RestFn applyTo "RestFn.java" 132]
  [clojure.core$apply invokeStatic "core.clj" 661]
  [clojure.core$apply invoke "core.clj" 652]
  [the_test$fn__3748$fn__3749 invoke "the-test.clj" 8]
  [com.gfredericks.test.chuck.clojure_test$_testing invokeStatic "clojure_test.cljc" 102]
  [com.gfredericks.test.chuck.clojure_test$_testing invoke "clojure_test.cljc" 100]
  [the_test$fn__3748 invokeStatic "the-test.clj" 8]
  [the_test$fn__3748 invoke "the-test.clj" 7]
  [clojure.test$test_var$fn__9209 invoke "test.clj" 716]
  [clojure.test$test_var invokeStatic "test.clj" 716]
  [clojure.test$test_var invoke "test.clj" 707]
  [clojure.test$test_vars$fn__9235$fn__9240 invoke "test.clj" 734]
  [clojure.test$default_fixture invokeStatic "test.clj" 686]
  [clojure.test$default_fixture invoke "test.clj" 682]
  [clojure.test$test_vars$fn__9235 invoke "test.clj" 734]
  [clojure.test$default_fixture invokeStatic "test.clj" 686]
  [clojure.test$default_fixture invoke "test.clj" 682]
  [clojure.test$test_vars invokeStatic "test.clj" 730]
  [clojure.test$test_all_vars invokeStatic "test.clj" 736]
  [clojure.test$test_ns invokeStatic "test.clj" 757]
  [clojure.test$test_ns invoke "test.clj" 742]
  [clojure.core$map$fn__5587 invoke "core.clj" 2747]
  [clojure.lang.LazySeq sval "LazySeq.java" 40]
  [clojure.lang.LazySeq seq "LazySeq.java" 49]
  [clojure.lang.Cons next "Cons.java" 39]
  [clojure.lang.RT boundedLength "RT.java" 1785]
  [clojure.lang.RestFn applyTo "RestFn.java" 130]
  [clojure.core$apply invokeStatic "core.clj" 659]
  [clojure.test$run_tests invokeStatic "test.clj" 767]
  [clojure.test$run_tests doInvoke "test.clj" 767]
  [clojure.lang.RestFn invoke "RestFn.java" 408]
  [clojure.test$run_tests invokeStatic "test.clj" 772]
  [clojure.test$run_tests invoke "test.clj" 767]
  [the_test$eval3773 invokeStatic "the-test.clj" 14]
  [the_test$eval3773 invoke "the-test.clj" 14]
  [clojure.lang.Compiler eval "Compiler.java" 7062]
  [clojure.lang.Compiler load "Compiler.java" 7514]
  [clojure.lang.Compiler loadFile "Compiler.java" 7452]
  [clojure.main$load_script invokeStatic "main.clj" 278]
  [clojure.main$script_opt invokeStatic "main.clj" 338]
  [clojure.main$script_opt invoke "main.clj" 333]
  [clojure.main$main invokeStatic "main.clj" 424]
  [clojure.main$main doInvoke "main.clj" 387]
  [clojure.lang.RestFn applyTo "RestFn.java" 137]
  [clojure.lang.Var applyTo "Var.java" 702]
  [clojure.main main "main.java" 37]]}))

Ran 1 tests containing 1 assertions.
1 failures, 0 errors.

I find this report to be very difficult to interpret, because it's long (~380 lines in 0.10.0-alpha3, ~230 in 0.9.0) and buries :smallest somewhere in the middle. The solution could be to allow customization of this report (e.g. also via defmethod).

gfredericks commented 5 years ago

I think this hinges on this function, which does something different depending on whether the tests threw an exception or not. The else branch of the if delegates to a special ct/report type of ::shrunk, while the then branch just relies on the default reporting of is. I think the then branch could be refactored to use a second special ct/report type (::error?) that includes the generated data in a similar way that the ::shrunk impl does.

I think this would give you what you need, does that sound right?

metametadata commented 5 years ago

Yes, sounds good!

gfredericks commented 5 years ago

Is this something you can help with with a pull request?

metametadata commented 5 years ago

Sure, will do in spare time.