gfredericks / test.chuck

A utility library for test.check
Eclipse Public License 1.0
215 stars 26 forks source link

Summary report from test.check is not being shown when using `checking` #28

Closed nberger closed 8 years ago

nberger commented 8 years ago

When test.check detects a failure, it outputs something like the following:

{:result false, :seed 1442512581845, :failing-size 12, :num-tests 13, :fail [8], :shrunk {:total-nodes-visited 9, :depth 2, :result false, :smallest [5]}}

Since test.chuck 0.1.22 that output is being swallowed by test.chuck and it doesn't show up in the test output. I created a test to verify the correct behavior in 0.1.21 in the tc-output-test-0.1.21, and added the same test for current master in fix-missing-output-from-tests. The broken build is in https://circleci.com/gh/nberger/test.chuck/48

As a comparison of the actual output, let's run the following test:

(deftest a-failing-test
  (checking "all ints lt 5" 100
             [i gen/int]
             (is (< i 5))))

Output in 0.1.21

› grep chuck project.clj              
  :profiles {:dev {:dependencies [[com.gfredericks/test.chuck "0.1.21"]

› lein test                           

FAIL in (a-failing-test) (clojure_test.clj:17)
all ints lt 5
{:result false, :seed 1442512581845, :failing-size 12, :num-tests 13, :fail [8], :shrunk {:total-nodes-visited 9, :depth 2, :result false, :smallest [5]}}
expected: (not-falsey-or-exception? (:result result))
  actual: (not (not-falsey-or-exception? false))

FAIL in (a-failing-test) (core_test.clj:27)
all ints lt 5
expected: (< i 5)
  actual: (not (< 5 5))

Ran 1 tests containing 2 assertions.
2 failures, 0 errors.
Tests failed.

Output in 0.1.22

› grep chuck project.clj 
  :profiles {:dev {:dependencies [[com.gfredericks/test.chuck "0.1.22"]

› lein test                           

FAIL in (a-failing-test) (core_test.clj:27)
all ints lt 5
expected: (< i 5)
  actual: (not (< 5 5))

Ran 1 tests containing 2 assertions.
1 failures, 0 errors.
Tests failed.

Output in 0.2.0

› grep chuck project.clj 
  :profiles {:dev {:dependencies [[com.gfredericks/test.chuck "0.2.0"]

› lein test

FAIL in (a-failing-test) (core_test.clj:27)
all ints lt 5
expected: (< i 5)
  actual: (not (< 5 5))

Ran 1 tests containing 2 assertions.
1 failures, 0 errors.
Tests failed.