search

lambdaisland / kaocha

Full featured next gen Clojure test runner
https://cljdoc.org/d/lambdaisland/kaocha/1.0.861/doc/1-introduction
Eclipse Public License 1.0
797 stars 83 forks source link

Compatibility with fulcro-spec #15

Closed plexus closed 5 years ago

plexus commented 5 years ago

fulcro-spec is an elaborate testing library that builds on top of clojure.test.

Fulcro-spec has its own organizational structure for tests, and has custom clojure.test events that correspond with them, like :begin-specification or :end-provided. It's intended to be used with its own reporter that knows how to handle these events.

The Issue

When Kaocha sees an event type it doesn't know about, it forwards it to the original clojure.test/report (so ignoring rebinding). If no defmethod's exist for these types then clojure.test/report just prints out the event map, and since the :kaocha/testable is in there this gets really noisy.

Why is it like that

In this case fulcro-spec provides both custom events and a custom reporter, expecting you to rebind clojure.test/report. Other libraries (e.g. assertion libraries) provide custom events but no custom reporter, instead extending the clojure.test/report method to handle these events. Both approaches are within scope of the intended use of clojure.test, but they are not compatible.

So Kaocha rebinds the reporter, but still forwards unknown events to the original reporter, assuming they will be handled on that level.

Possible solutions

As a first step we should check if clojure.test/report has a multimethod implementation for a given event type before forwarding the event. The default behavior of printing the event map is never useful.

Note that this could also be solved presently by adding the fulcro-spec events type to the kaocha hierarchy, making kaocha aware of them and thus preventing the forwarding.

Kaocha reporters and general compatibility

I'll have to try out fulcro-spec to better understand what its reporter is bringing to the table, and how well it plays with other libraries. E.g. I suspect that you can't use matcher-combinators with fulcro-spec's reporter.

Going forward we should make sure fulcro-spec can also work with Kaocha's reporters, since these provide facilities for extension by multiple parties, preventing some of the problems that extending clojure.test/report directly has.

awkay commented 5 years ago

I'll give a shot at patching fulcro-spec. Thanks for the details.

awkay commented 5 years ago

https://github.com/fulcrologic/fulcro-spec/issues/10

awkay commented 5 years ago

spec 2.1.3 sets the multimethods in t/report to do nothing for the special events. This cleans up the output.

plexus commented 5 years ago

Hi @awkay, Kaocha is now more conservative in what it sends to t/report, it will only forward events if there is a (non-default) implementation of t/report that will handle it.

I know it's no longer an issue for Fulcro but if people are using other assertion libraries this might prevent them getting some very noisy output.

Thanks for the input and for trying out Kaocha!