gfredericks / test.chuck

A utility library for test.check
Eclipse Public License 1.0
214 stars 26 forks source link

Reported assertion count #45

Closed dpassen closed 7 years ago

dpassen commented 8 years ago

When running the sample code from the README,

(require '[clojure.test :refer [deftest is]]
         '[com.gfredericks.test.chuck.clojure-test :refer [checking]])

(deftest my-test
  (checking "that positive numbers are positive" 100
    [x gen/s-pos-int]
    (is (pos? x))
    (is (> x 0))))

I expect the number of assertions to be 2, one for each is clause. When running this (clojure 1.8, lein 2.6.1, test.check 0.9.0, and test.chuck 0.2.6), the assertion count is 3.

Within a larger private project, I see that each successful checking clause brings along with it an implicit, passing assertion.

gfredericks commented 8 years ago

I can see how this is confusing, and it might be worth making the fix I assume you have in mind (decrementing the assertion count somehow); however I think this interaction between test.check and clojure.test is inherently messy, since the tests are being run many times but we're reporting assertion counts as if it were run once.

E.g., what should we report for a test like this?

(deftest my-test
  (checking "that positive numbers are positive" 100
    [x gen/int]
    (if (pos? x)
      (is (> x 0)))))
dpassen commented 8 years ago

That's certainly a more difficult problem. Perhaps, it could report the number of assertions hit through the generator. For simple cases like the gen/s-pos-int this would be 2 * number of iterations. The gen/int test would have varying number of assertions on each run.

Feel free to close since there's no common solution.