Faster cider-test feedback

[r0man]

When running tests via clojure-test-mode it was nice that you got instant feedback while running tests. A failing test at the beginning of a test run could already be taken care of while other tests were still running.

With cider-test you get the test results after all tests have finished. Would it be possible to stream the test results back ealier, while other tests are still running?

For larger projects this makes quiete a difference, and I wonder if there's interest in restoring this behaviour?

Otherwise I quite like cider-test.

Thanks again for this, Roman.

[jeffvalk]

This might take some effort, but is certainly possible.

@r0man Out of curiousity, how long do your tests take for a single namespace?

[r0man]

@jeffvalk Some integration tests with databases, message queues etc involved take easily > 10sec :(

[bbatsov]

Guess it'd be nice we showed the progress of the tests with ...... (green dots per passing tests, red otherwise, as in xunit test frameworks).

[r0man]

Progress would be also nice. The most important thing to me is to see failing tests and their error message as early as possible.

[bbatsov]

Btw, if we get to implementing this we should probably add some option to fail fast (stop running the tests at the first failure).

[jeffvalk]

Following 51bf460, I rather expect this issue will get bumped.

[bbatsov]

Guess so. It seems to me like the most important thing related to tests that remains to be done.

[bbatsov]

@jeffvalk Do you plan to work on this eventually?

[jeffvalk]

At some point, sure. Can't promise it very soon though; I'm a bit short on time.

I should note that I've never personally needed this, so anyone who has is welcome to put in their two cents wrt user experience.

[bbatsov]

Sure. I was just going over some tickets a day ago and was wondering what's next for us. Unfortunately my time for CIDER was progressively dwindled as well.

[bbatsov]

@r0man Not sure if you're still interested in this after so much time, but now I know you're more than capable of implementing it, so you might consider tackling it at some point. ;-)

[vemv]

Perhaps a simple "fail-fast" feature would suffice, otherwise one can imagine leaving things in a "concurrent" state (user editing, tests still running) prone to complexities.

[bbatsov]

I view the fail-fast as orthogonal to streaming the results, but it's also definitely the simpler feature to implement.

I don't think that multiple test runs would be a big problem - after all that's not different from today (I don't even remember what we do currently on this front, if anything; serializing the requests seems the the easiest thing). Most command-line test runner show gradual progress, so I definitely think it won't be bad for CIDER to do the same. Not a big prio obivously - we lived without this for a long time and as a workarond people can just use something like Kaocha or run their tests in the REPL/terminal.

[vemv]

I reckon that the simplest definition for this task would be:

• introduce an option, e.g. cider-test-stream?
• if true, when running tests, *cider-test-report* pops up as soon as the first test has failed, showing a test failure report as usual
• any subsequent failed tests are appended to *cider-test-report*
• The results header (Tested 1 namespaces \n Ran 11 assertions, in 1 test functions \n 1 failures) is updated as each test passes/fails

Assuming this approach, what are your thoughts on "streaming" these buffer contents?

Perhaps a particularly simple approach would be to replace the whole buffer's contents each time an update is ready, instead of trying to edit it smartly (by editing the headers and appending things). Probably Emacs wouldn't jitter.

[vemv]

Guess it'd be nice we showed the progress of the tests with ...... (green dots per passing tests, red otherwise, as in xunit test frameworks).

This would be nice, but my impression is that Emacs is pretty bad at displaying such widgets?

[bbatsov]

Perhaps a particularly simple approach would be to replace the whole buffer's contents each time an update is ready, instead of trying to edit it smartly (by editing the headers and appending things). Probably Emacs wouldn't jitter.

Redrawing is an option, but you can also have a slightly different format then and have the summary at the end which will eliminate the need for this. Alternatively we can show the progress in one buffer (or the minibuffer) and show the same summary as now when all the tests have been run. I'm guessing that might be easier to do.

I don't have a strong preference about this, so feel free to go with whatever is easiest to implement.

This would be nice, but my impression is that Emacs is pretty bad at displaying such widgets?

It doesn't have to be widget per se - it can just be some text, as terminal runners do it, and a summary people can interact with following this.

[r0man]

Hi, I totally forgot about this issue and haven't longed for it in the recent years. Either I got used to it, or my tests got faster. I think it would still be a nice improvement. I won't have time to work on this right now, though. I have another NREPL project going on and will be on holidays for some time soon.

[vemv]

Either I got used to it, or my tests got faster. I think it would still be a nice improvement.

Same!

I ended up feeling pretty convinced about the potential value of this feature. Sometimes in enterprise codebases, test namespaces are larger than one would personally want.

It also particularly makes sense for testing a whole project (vs just a single ns). Without streaming, it would be prohibitively non-interactive.

Redrawing is an option, but you can also have a slightly different format then and have the summary at the end which will eliminate the need for this.

Good thinking. It also makes sense from a UX point of view - if I see output being appended to the bottom, the summary should also be rendered at the bottom - otherwise it would seem easy to forget to scroll to top to see the summary.

I'll give this one a shot if it fits with the rest of my planned work.