Tooling friendly output for console runner

[rundis]

Hi !

I'm about to add initial support for elm-test in my elm-light plugin. I'll most likely add support for running tests through node-elm-test. Anyways it would be great if the output could have an output that was more tooling (and ci) friendly. The obvious choice might be xunit xml (ci friendly), all though I'd prefer json (tooling friendly). The more context information that can be provided in the output, the larger the room for "innovation" for tooling ! (suite, testfile, testfunction, assertion, line/col number, exception/errors).

On a related note, would it be possible to stream the results, rather than waiting for all tests to complete before reporting. Also would that play well with node-elm-test @rtfeldman ?

A few thoughts on what I'd like to achieve (long term):

• A user can open a elm test file in Light Table (it's recognized as a test file by convention, say in test/tests folder for your project)
• User selects command to run test
• Test is run
• Results are shown inline, any failed asserttions/errors are shown/highlighted on their relevant code line (or code range).
• when relevant (many/slow tests); user is notified on progress of the test run
• Running all tests option
• Continuous testing/auto-test option etc...

I guess some synchronization of efforts with elm-check would be great to.

[deadfoxygrandpa]

I had already wanted to add something like xunit output, but you mention json output? Are you thinking of some sort of specific json standard that you can point me toward? The point behind adding xunit would be to make it easier to plug into existing systems.

I also plan to add result streaming, I just haven't got around to it yet.

[rundis]

xunit output would undoubtably be a big win in terms of supporting existing tools.

The reason I mention json (think of it as a additional very nice to have feature!) is;

• The xunit format is very OO (/java) inspired with things like classname and package (ref xsd)
• xml sucks (so much easier to work with json).
• streaming results might be easier, say for a suite you can emit when suite starts, then emit individual results as they happen without needing to be limited by sending a complete testsuite xml (with xml you must send a suite with a start-tag and end-tag for it to be valid)
• Flexibility to have much richer information :-)

But no I don't have any json std format to refer to. Just pitching an idea really.