xUnit Driver

[12AT7]

Our team is definitely liking mettle!

I was looking into what it would take to add an xUnit driver to support Jenkins dashboards (presumably with the xUnit plugin). Is the proper approach basically just to copy verbose.hpp and verbose.cpp, rename to xunit.hpp and xunit.cpp, and modify? I found the format described at http://reflex.gforge.inria.fr/xunit.html#xunitReport.

Is that basically what we would have to do to implement this feature, or is there any other previous ideas about how to integrate mettle with a continuous integration workflow, that should be considered instead?

[jimporter]

It depends on what you want to do. You don't have to patch mettle directly if you don't want; you could just write a new test driver that spawns a subprocess for each test file, and then listens to the test events as they're emitted over the pipe. If you pass --output-fd=N to the subprocess, where N is a file descriptor, the individual test file should send test events over that fd. The communication is done via bencode, and you can see how the mettle driver parses the events here: https://github.com/jimporter/mettle/blob/master/src/mettle/log_pipe.hpp

In the long term (post-1.0), I'd like to support different loggers somehow so people can integrate with their CI systems, but I'm not sure how I want to handle that yet. Chances are it'll be something like what I just said above, or some sort of plugin system for the mettle driver where you can add new kinds of output formats. Currently, I'm leaning towards the former, since the mettle driver is a) pretty simple, and b) designed primarily for interactive use.

[jimporter]

There are also some subleties with the xUnit format that might take some thought; one of the big ones is that I'm not sure if xUnit allows nested test suites (mettle does). The docs you posted don't say one way or another, but they seem to imply that a test suite can only contain test cases, not other test suites.

P.S.: I'm glad to see you're having a good time with mettle!

[jimporter]

@JohnGalbraith I'm starting work on a built-in implementation of this now, and wanted to get your thoughts on a few things:

1. I notice that your implementation in #37 writes one XML file per mettle test file. Is that something common to how xUnit works everywhere, or was that just something that worked for your use case? I'm fine with either doing one massive XML file or a directory of files, although in the latter case, I worry a bit about stale files (e.g. if you delete a mettle test file but don't clean the xUnit dir, that file would stick around unless mettle wipes the dir first).
2. On that note, I just want to confirm: it would be weird to print all the XML on standard output, right? (Writing the XML to a file actually seems like a nice thing for mettle, since it means that you still get the summary of your failures printed to stdout, and that won't mess up your XML files.)
3. I don't see any support in your PR for doing multiple runs in a single invocation of mettle. Do you think there's value in adding support for this to the native implementation, and if so, do you have any ideas for how it should work? I was thinking something like emitting files that look like run-1/path/to/test.xml. Or I could just throw an error if the user specifies >1 run. :)

If you're curious, where I'm at right now is that I made an extremely-simplistic C++ representation of the DOM and build up a document similar to how your PR #37 works, and then I just spit it out on stdout for testing (eventually, this will be a file or collection of files). Luckily, writing XML is pretty simple (especially since mettle effectively mandates UTF-8), so I won't need to add any extra dependencies to mettle.

[12AT7]

Hey Jim, this sounds great!

I don't really consider myself an expert on xUnit or DevOps in general, but maybe I can fake it. 8-)

On 1), I actually do not know how other systems work, but I did discover that at least the tools that I experimented with (mostly Allure [http://allure.qatools.ru/]) accept multiple XML files. This suggests that other unit test systems might also drop multiple files. It does not feel to me like it is a big deal one way or the other; if writing more than one file is convenient, then there appears to be no harm in doing so. Also, having stale files is not really an issue on (the few) devops systems I have seen, because each automatic build starts with a clean directory specific to that particular build. This is true on Bamboo, which is a terrible CI system, and CircleCI, which is among the best. I am pretty sure Jenkins starts with clean directories also, but not certain. And looking back at my Mettle experiences so far, I have never once looked at those XML files on my development system after I finished writing that script; they are only ever consumed on the CI system automatically. So stale XML files, in my experience, is a total non-issue.

On 2), I cannot imagine a situation where I would routinely want the XML on stdout. XML is wretched to read by eye, while Mettle's console output is gorgeous. If I got the XML on stdout, in almost all cases I would be directing that to a file, and this would be in scripts. Again on the automatic CI system, it seems natural to have the human readable console log appear in the build spew along with the compiler messages, and simultaneously write the XML to files to be consumed by Allure or similar graphical dashboard tooling.

On 3), I am not super clear on the use case for this. Do you mean that I would be able to run my test suite three times, and have three separately saved outputs? I don't see this being particularly useful, unless I missed the point. Normally, I will manually run the test suite just to get a tentative green light before I push a commit to a feature branch. Multiple runs would occur on the CI system automatically, as I tested different compilers, optimization settings, environments, etc., and each of those would get a unique output directory for the whole build. On merges, those tests are run only in automation.

Actually, now I see your point: the "mettle --runs" option! Honestly, I have never used that feature 8-) In that case, it's not just a question of where to store the files, but what your dashboard should actually show and making sure that we don't confuse Allure (which is the whole point of using xUnit). Replicants seem to be a similar feature to parameterization in this regard, so whereas now the name of a test might be "test addition (int)" to show the parameter, it seems most useful to have it appear as "test addition (int)[2]" or whatever to indicate the replicant also. So I guess my first reaction is that no, we don't want separate XML output directories, but rather a distinctive name for the specific test inside the file.

I am looking forward to checking this out!

[jimporter]

@JohnGalbraith Thanks for the comments! I have a WIP patch for this on the xunit branch. At least for the time being, I went with a single XML file for all the tests (rather than one XML file per test file) since that's a little easier, and I also disabled the ability to do multiple runs when using the xunit logger. The --runs option is mainly there for debugging intermittent failures, so it's probably not super-necessary when using xunit.

Since you have a Python version of this in #37 that I'm guessing you're using locally, you might also be interested in how I handle the started_suite and ended_suite events. mettle is designed so that when a suite starts, subsuites run first, then the tests for the suite itself. That means that you need to keep a stack of the suites around so you can append the current test to whatever suite is on top. Here's what my patch does:

https://github.com/jimporter/mettle/blob/555eea8d39123153ee93b3abe2245e2e9a252854/src/libmettle/log/xunit.cpp#L65-L81

[jimporter]

mettle is designed so that when a suite starts, subsuites run first, then the tests for the suite itself.

Whoops, this is a lie! Tests run first, then subsuites. (However, I've been seriously considering making them run in whatever order they're specified in mettle, so doing something like the above is the safest long-term solution.)