Broken by `cljs.test` implementation of async tests

[crisptrutski]

So dug around, and discovered the reason for failures - the summary is not coming back through HtmlUnit because cljs.test/run-tests no longer has a return value. (http://dev.clojure.org/jira/browse/CLJS-988)

Looked around for ways to get at the summary (whether or not async tests exist), and the situation is not that great -

It's using a continuation passing style, but provides no easy way to add a call on the end. Also the function that turns test namespaces into functions for the CPS chain includes adding a final call, which generates the report side effect and wipes out the summary state. So we can't chain after that.

So we can't even recompose the existing functions, we'd have to mess with the function chain (cljs.test/run-block (concat (butlast (run-tests-block env namespaces..)) [fn-to-throw-summary-exception])).

Our other affordance is a custom reporter, but this again is clumsy The way the multimethods work (dispatch on two keys) for reporting mean we'll have to reimplement a whole bunch of the basic crud steps just to be able to override the one that we care about.

Options:

1. Do the awful thing above about building manual CPS chain
2. Do the awful thing with copy-pasta custom test reporter
3. Do something else awful to spy on when tests are done, then get the summary stats from the :message capture of the report printout.
4. Request addition of :on-complete style optional key in the test environment, which takes a function to be called with summary in that final CPS call that writes report and clears state.

Think (4) is the way to go, and happy to provide the CLJS patch - but scared of JIRA etc ;)

What do you think?

(btw hacked around this with local fork and got it working, "just composes" with watch, really awesome)

[pandeiro]

Chris, first of all, thanks for investigating this and for the thorough report. Awesome work.

Admittedly I've been letting this project simmer while I am busy with other work and while I think about how to move forward in light of #1. That's another question and not really related to this, though.

In terms of what I think, I'm still trying to understand everything but I think (2) is potentially interesting if most of the "work" is copy-paste and the implementation part is just that one method. That would at least prevent having to make a huge investment in time in order to address this, right?

The option (3) is, if I remember correctly, somewhat similar to how cemerick's clojurescript.test deals with retrieving async test results. Whether it's "awful" or not might be a matter of taste (and our alternatives).

If you want to try to do (4), JIRA is pretty manageable (hell, I did it!) -- David has posted good instructions here. It would be a great idea to check with him in #clojurescript maybe, if that's an option, to see if he's receptive to the idea.

[crisptrutski]

(3) certainly is the intended extension point - I took a whack, but it's not quite enough - boot process isn't catching any exceptions.

Opening the tempfiles with a browser, the exception comes through (with a weird quirk that it's wrapped and rethrown by Firebase, because the done calls are threaded through Firebase callbacks.

Should be possible to get past that with a setTimeout. Perhaps the wrapping is breaking the exception? Or maybe async + htmlunit is just a lot of bother (didn't tackle it very neatly, just threw some of the "wait wait wait" style functions at it, hoping on fortune.

[crisptrutski]

OK so the problem is definitely html unit - browser is throwing the exception loud and clear now. Strangely the hang-around attempts are keeping it alive, longer than the time it takes to run tests in Chrome. Not sure why exception is not materializing.

Thinking of just adding slimer/phantom support to get around this :ghost: