weavejester / eftest

Fast and pretty Clojure test runner
424 stars 40 forks source link

Double-count issue when using two reporters #23

Open AndrewGnagy opened 7 years ago

AndrewGnagy commented 7 years ago

I'm attempting to use a custom reporter to report results to both junit and the pretty-printer.

What I've found is that this will double-count the number of assertions and failures. For example:

    Ran 5 tests in 6.059 seconds
    10 assertions, 2 failures, 2 errors.

(Should be 1 assertion per test for 5 total. 1 failure and 1 error.)

A work-around suggested by @miikka is binding clojure.test/*report-counters* to nil for one of the reports:

    (defn report [m]
      (pretty/report m)
      (binding [clojure.test/*report-counters* nil]
        (xml-report m)))

Which works.

Is there a more proper way of handling this?

weavejester commented 7 years ago

Because of the way clojure.test works, that might be the best solution. We could add in something that combines multiple reporters automatically, or potentially we could try and create a new reporter abstraction we could layer on top of clojure.test/report, but adding new abstractions is something I tend to be cautious about.

Deraen commented 7 years ago

I'm using following function in boot-alt-test to combine collection of reporters, should we add this to eftest.report?

(defn combined-reporter
  "Combines the reporters by running first one directly,
  and others with clojure.test/*report-counters* bound to nil."
  [report & rst]
  (fn [m]
    (report m)
    (doseq [report rst]
      (binding [clojure.test/*report-counters* nil]
        (report m)))))
aviflax commented 6 years ago

~FWIW, I just wrote my own version of such a function, and I’m not encountering the double counting.~

Update: Ah, never mind, I didn’t read the TP closely enough. I am indeed seeing the same issue with my function.

My function ```clojure (defn- multi-report "Accepts n reporting functions, returns a reporting function that will call them all for their side effects and return nil. I tried to just use juxt but it didn’t work. Maybe because some of the reporting functions provided by eftest are multimethods, I don’t know." [& fs] (fn [event] (doseq [f fs] (f event)))) ```
aviflax commented 6 years ago

Ah, never mind, I didn’t read the TP closely enough. I am indeed seeing the same issue with my function.

buzzdan commented 5 years ago

i'm upvoting the need for a duel reporter --> i wanna be able to run lein eftest in my CI that produces a junit report while outputing to console in development. (or just use both)