gfredericks / test.chuck

A utility library for test.check
Eclipse Public License 1.0
214 stars 26 forks source link

The `:fail` and `:shrunk :smallest` reported values include binding names instead of just the bounded values #31

Closed nberger closed 8 years ago

nberger commented 8 years ago

The change to use test.chuck/for-all instead of test.check/for-all in test.chuck.ct made the test failure report :fail and :shrunk :smallest entries to change from reporting a vector of bounded values to report a map from bounded names to bounded values. As an example, let's consider the following failing test:

(deftest a-failing-test
  (checking "int by int lt 25" 100
    [i gen/int
     j gen/int]
    (is (< (* i j) 25))))

Under the current code in master, it reports the following as the failure summary:

{:result false, :seed 1447369243259, :failing-size 18, :num-tests 19, :fail [{i 9, j 4}], :shrunk {:total-nodes-visited 9, :depth 1, :result false, :smallest [{i 7, j 4}]}}

Before the change introduced by #30, it would return the following report:

{:result false, :seed 1447369123712, :failing-size 9, :num-tests 10, :fail [6 7], :shrunk {:total-nodes-visited 12, :depth 2, :result false, :smallest [4 7]}}

So, it's now reporting a map representing the binding names and bounded values {i 9, j 4}, instead of a vector with the bounded values [9 4]

I'm not sure if this change in behavior is a bug or a feature :). Seems like a feature, as now the :fail entry can be taken and put as the binding of a let to run the test body with the same bound values.

If it's a feature, then it's pretty easy to fix the failing test that is making the build to be red, by just changing the expected output.

What do you think @gfredericks?

nberger commented 8 years ago

I'm going to merge #36 right now. It just fixes the test expectation, seems reasonable to me.