Add some feedback mechanism to monitor progress

[cemerick]

If the parameter space is large (leading to long shrinking search times) or the per-test runtime is long (which lengthens high-count test runs generally), individual calls to quick-check can take a very long time to complete, with no indication of progress. This is irritating in general, but can be particularly painful if e.g. the function under test + the generated arguments produce an infinite loop themselves, or the shrinking function is flawed such that it produces an infinite seq.

The easy way out would be for quick-check and shrink-loop to report progress somewhere every N iterations or M seconds, perhaps along with a sampling of the generated arguments (though that could get verbose really fast). This would basically fall out of the simplest possible integration with clojure.test…

[reiddraper]

Definitely useful. The two implementations that I'm most familiar with (Haskell and Erlang) print a dot, . after each test run. If the test fails, they also print something to that effect. I'm pretty sure they both offer a 'no-verbose' option too. I tend to like quiet tests by default, but an options hash makes it easy to switch back and forth.

This would basically fall out of the simplest possible integration with clojure.test…

Yeah, I'm definitely interested in this, but don't yet have good enough knowledge of clojure.test

[cemerick]

Yeah, I generally prefer silent tests too, but I think that preference has come mostly from working with relatively short-running unit tests. Even longer functional suites take just a fraction of the time that I find myself running "generative" tests in order to feel comfortable with the coverage.

I assume the scope of a "test run" is a single trial / loop through quick-check? If so, that might be overkill (I'm running 10M trials of each property at the moment).

I've done more clojure.test hacking than I care to admit, so I'll whip up some integration there and see how it shakes out in #4.

[reiddraper]

I assume the scope of a "test run" is a single trial / loop through quick-check? If so, that might be overkill (I'm running 10M trials of each property at the moment).

Yes, heh. The Erlang and Haskell implementations also do 100 tests by default. I think encouraging you to write quicker tests and doing a default of 1000 might be better. Maybe a simple dot every N% will be flexible enough for now?

[cemerick]

Some initial clojure.test integration is working; the output I've set up so far is like this:

user=> (clojure.test/test-all-vars 'cemerick.test-sedan)
Passing trial 0 / 5000 for property #<core$partial$fn__4190 clojure.core$partial$fn__4190@7ce9702>
Passing trial 3383 / 5000 for property #<core$partial$fn__4190 clojure.core$partial$fn__4190@7ce9702>
{:result true, :num-tests 5000}

FAIL in (quickcheck-number-encoding*) (:)
expected: true
  actual: false
Shrinking #<core$partial$fn__4190 clojure.core$partial$fn__4190@2d7a8b8c> starting with parameters [6008778112100936364 1128103527268812779.651968794899666900587364329479847527247891277800782755894250252372692821421647106636637406825559474841828089607828M]
{:shrunk {:total-nodes-visited 121, :depth 2, :smallest [1502194528025234091 1128103527268812779.651968794899666900587364329479847527247891277800782755894250252372692821421647106636637406825559474841828089607828M]}, :result false, :num-tests 1545, :fail [6008778112100936364 1128103527268812779.651968794899666900587364329479847527247891277800782755894250252372692821421647106636637406825559474841828089607828M]}
nil

The "passing trial" lines are emitted every 10s (customizable, and disabled by default). The dots thing could be added easily (I'll make sure it's available before I create a PR), I just haven't seen the value yet: my objective is just to know that the testing is proceeding, so a timed message makes more sense than trying to figure out how many trials should pass between dots (seems impossible to default sanely, given the variability in property runtimes).

Property names are a problem, at least while using HOFs; I can print out the current clojure.test var's name, as long as people maintain a 1:1 correspondence between property quickchecks and test vars.

Finally, all of the work so far depends upon clojure.test for the output, meaning that no output is produced by running quick-check directly (test-ns or test-all-vars sets up the output bindings). That's either good or bad, depending on your perspective. The nice thing about it is that the actual reporting of progress is entirely customizable (you could easily override the clojure.test multimethods dispatches that I've set up exclusively for simple-check).

Does this sound sane?

[reiddraper]

Some initial clojure.test integration is working; the output I've set up so far is like this:

Looks like a great start to me.

The dots thing could be added easily (I'll make sure it's available before I create a PR), I just haven't seen the value yet:

One benefit of the dots is you get a good feel for how long each test takes. That being said, if they're fast enough that running 1k, 10k, etc. only takes a second or so, that value starts to dwindle.

Property names are a problem, at least while using HOFs; I can print out the current clojure.test var's name, as long as people maintain a 1:1 correspondence between property quickchecks and test vars.

+1 to starting with clojure.test vars name for now. Good enough

Finally, all of the work so far depends upon clojure.test for the output, meaning that no output is produced by running quick-check directly (test-ns or test-all-vars sets up the output bindings). That's either good or bad, depending on your perspective. The nice thing about it is that the actual reporting of progress is entirely customizable (you could easily override the clojure.test multimethods dispatches that I've set up exclusively for simple-check).

Does this sound sane?

Yes, +1 to all of that

[cemerick]

Aside from any tests I might add and README docs, the above commit is complete AFAICT. If it looks decent, let me know what defaults seem right to you. I continue to prefer the periodic reporting (which ends up being silent for anything reasonably interactive given the 10s period), but I can certainly see that not being a common preference.