marick / Midje

Midje provides a migration path from clojure.test to a more flexible, readable, abstract, and gracious style of testing
MIT License
1.69k stars 129 forks source link

`expr =not=> checker` is true when `expr` evaluates to an exception #438

Closed philomates closed 6 years ago

philomates commented 6 years ago

In midje you can do

(fact (throw (ex-info "Not 1" {})) =not=> 1)

which is technically true. That said, this can be problematic because it hides exceptions that may come up from bugs (happened to my colleague the other day)

For example:

(defn critical-code [] (throw (ex-info "important issue I want to have visibility on" {})))

;; this passes, hiding buggy behavior
(fact (critical-code) =not=> "bad")

;; produces the failure as expected
(fact (critical-code) => "good")

FAIL at (form-init5640994841513193915.clj:1)
Expected:
"good"
Actual:
clojure.lang.ExceptionInfo: important issue I want to have visibility on {}
  user$critical_code.invokeStatic(form-init5640994841513193915.clj:1)
  user$critical_code.invoke(form-init5640994841513193915.clj:1)

Proposal: =not=> should behave the same as => when the left side throws an exception.

 (fact (critical-code) =not=> 1)
;; should produce the following result

FAIL at (form-init5640994841513193915.clj:1)
Expected:
1
Actual:
clojure.lang.ExceptionInfo: important issue I want to have visibility on {}
  user$critical_code.invokeStatic(form-init5640994841513193915.clj:1)
  user$critical_code.invoke(form-init5640994841513193915.clj:1)
philomates commented 6 years ago

This is probably not feasible, given the following example

(fact
  (throw-exception "msg") =not=> (throws "another-message" Error #(= "another-message" (.getMessage %))))

There is no way to know when the checker is validating something about the exception thrown on the right side of the arrow, so it is impossible to know when the exception should be treated as a check failure or not