`for-all`: a construct for generative testing

[philomates]

A more uniform way to use clojure.test.check/quick-check with facts. Allows you to easily check multiple facts given generated values and auto-shrinks failing values before showing output.

So something that was once:

(require
  '[clojure.test.check :as tc]
  '[clojure.test.check.properties :as prop]
  '[clojure.test.check.generators :as gen])
(use 'midje.repl)

(tc/quick-check
  10
  (prop/for-all*
    [gen/s-pos-int gen/int]
    (fn [positive-num int]
      (fact (+ positive-num int) => pos?))))
;; =>
FAIL at (form-init7448724912800023367.clj:1)
Actual result did not agree with the checking function.
Actual result:
0
Checking function: pos?

{:fail [2 -2]
 :failing-size 4
 :num-tests 5
 :result false
 :seed 1510234310986
 :shrunk {:depth 2 :result false :smallest [1 -1] :total-nodes-visited 4}}

becomes

(require  '[clojure.test.check.generators :as gen])
(use 'midje.repl)

(for-all
  [positive-num gen/s-pos-int
   int          gen/int]
  (fact (+ positive-num int) => pos?))
;; =>
FAIL at (form-init7448724912800023367.clj:4)
Actual result did not agree with the checking function.
Actual result:
0
Checking function: pos?

quick-check seed:
1510234310986

quick-check shrunken failing values:
{int -1 positive-num 1}

Sort of similar formula (which is unfinished?) and is implemented in a way similar to tabular

With quick-check options specified:

(for-all "generative tests"
  [strictly-pos gen/s-pos-int
   any-integer  gen/int]
  {:max-size 10
   :num-tests 15
   :seed 1510160943861}
  (fact "Summing an integer to a positive integer should be positive? Really?"
    strictly-pos => integer?
    {:x (+ strictly-pos any-integer)} => (contains {:x pos?})))

Joint work with @rafaeldff

Some future work would be to use for-all to build a spec fuzzer: Take a function, grab its clojure.spec spec and evaluated the function over generated example arguments to ensure that the evaluated result matches the spec.

[marick]

You probably already know this, but there's stale code in there that was working toward generative tests. Grep for "formula". (There's also a file FORMULAS-BACKLOG-AND-FEATURE-IDEAS.md in root

[marick]

You probably already know this, but there's stale code in there that was working toward generative tests. Grep for "formula". (There's also a file FORMULAS-BACKLOG-AND-FEATURE-IDEAS.md in root

[marick]

You probably already know this, but there's stale code in there that was working toward generative tests. Grep for "formula". (There's also a file FORMULAS-BACKLOG-AND-FEATURE-IDEAS.md in root)

[marick]

You probably already know this, but there's stale code in there that was working toward generative tests. Grep for "formula". (There's also a file FORMULAS-BACKLOG-AND-FEATURE-IDEAS.md in root)

On Nov 8, 2017, at 11:14 AM, Phillip Mates notifications@github.com wrote:

Joint work with @rafaeldff https://github.com/rafaeldff Sort of similar to tabular (more details to come as progress is made)

(for-all "generative tests" [strictly-pos gen/s-pos-int any-integer gen/int] (fact "Summing an integer to a positive integer should be positive? Really?" strictly-pos => integer? {:x (+ strictly-pos any-integer)} => (contains {:x pos?}))) With quick-check options specified:

(for-all "generative tests" [strictly-pos gen/s-pos-int any-integer gen/int] {:max-size 10 :num-tests 15 :seed 1510160943861} (fact "Summing an integer to a positive integer should be positive? Really?" strictly-pos => integer? {:x (+ strictly-pos any-integer)} => (contains {:x pos?}))) https://user-images.githubusercontent.com/94817/32562837-f8e40d4e-c4af-11e7-89ce-036a207b2ec3.png You can view, comment on, or merge this pull request online at:

https://github.com/marick/Midje/pull/416 https://github.com/marick/Midje/pull/416 Commit Summary

Initial generative import add some validation Merge branch 'master' into pm/generative refactor logging move files around File Changes

M project.clj https://github.com/marick/Midje/pull/416/files#diff-0 (1) M src/midje/emission/api.clj https://github.com/marick/Midje/pull/416/files#diff-1 (3) M src/midje/emission/plugins/default.clj https://github.com/marick/Midje/pull/416/files#diff-2 (4) M src/midje/emission/plugins/silence.clj https://github.com/marick/Midje/pull/416/files#diff-3 (1) M src/midje/emission/plugins/util.clj https://github.com/marick/Midje/pull/416/files#diff-4 (2) A src/midje/parsing/0_to_fact_form/generative.clj https://github.com/marick/Midje/pull/416/files#diff-5 (75) M src/midje/parsing/1_to_explicit_form/metadata.clj https://github.com/marick/Midje/pull/416/files#diff-6 (1) M src/midje/sweet.clj https://github.com/marick/Midje/pull/416/files#diff-7 (19) Patch Links:

https://github.com/marick/Midje/pull/416.patch https://github.com/marick/Midje/pull/416.patch https://github.com/marick/Midje/pull/416.diff https://github.com/marick/Midje/pull/416.diff — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/marick/Midje/pull/416, or mute the thread https://github.com/notifications/unsubscribe-auth/AAEY5ULTu2qKwZGel91iDVREO21imWSAks5s0eF3gaJpZM4QWumN.

[philomates]

@marick ah, I had missed formula, it does almost the same thing :smile: Any idea how complete the implementation is / if people currently use it (cc/ @AlexBaranosky)?

If it is largely unused, I would vote to remove it to prevent confusion between the two constructs. Especially since it seems like shrinking and integration with test.check is still incomplete with formula, right?

[sovelten]

:+1: :cool:

[rafaeldff]

Just wanted to check if anyone has any issues with bringing test.check as a dependency. Clojure specs decided to use it as an optional dependency, going to great lengths to allow users to write tests with no references to test.check namespaces. I personally don't have a problem with the dependency, but I wanted to get others opinions.

[marick]

The formulas were never officially announced. As I recall, they didn't have narrowing. I never used it myself (except in the sense of keeping the formula tests working as Midje changed).

I'd recommend scavenging it for ideas or code, but otherwise continuing on your current path.