Tetha
commented
6 years ago Moin,
I've done a first simple, ham-fisted iteration over at my fork. I'd like some feedback, especially because I'm no go expert. Feel free to poke me with improvements whenever :)
Overall, I've introduced a package 'reporting' with an interface 'Reporter' and I've mostly tried to push existing informational fmt.Printf-calls into methods of this Reporter interface. Names are open to change. I've also added a QuietReporter because I wasn't sure if I need to worry about the parallel tests being in parallel and then I'd need concurrency and whatever. A /dev/null-reporter does the right thing even in a concurrent case and a very silent mode might be desirable.
There's a few things I'm bumping into by now:
- I'm unsure how to handle the external diff tool. By now I'd have to push the stdout/stderr to a reporter and then the reporter has to setup some async handling via channels, and who knows what's going to happen if there are parallel diffs going on. By intuition, I'd almost see if I can find a simple go json package, or at worst, implement some json diffing myself. That's a bunch of work, but process handling is ugly as well.
- I'm running into circular dependencies between testcase and reporting in my first iteration. I have to pass reporters into the testcase.Compare method, so there's one dependency, but I'd like to pass testcases structs into the reporter to offer a maximum of information to the reporting method. I'll see if I either do something about the compare method, or if I introduce some kind of digest structs for the reporting, which just contain informations about the tests. I'm tending towards the latter, because this results in a more expressive API of the reporting interface, which should make it easier to test the reporting implementations. It'd require some transformation code in testcase and maybe the test runner, but that should be small.
- multiple reporters should be simple by making a fan-out-reporter with a list of reporters and that's it.
- I havent thought about the parallel tests, or command line handling and so on at all.
I'd like some feedback especially about the first two parts before I progress here. But from there, most reporting issues should be rather simple to handle even with the minimal API I have from pushing prints around.
magnusbaeck
Currently, the progress and results of testing is only reported to stdout as user messages, which is fine for human consumption but makes it hard to present the results in a structured way for machine consumption (like CI systems).
Instead of overloading the current code paths with various output formats, let's solve this with the observer pattern so that the core code would report its progress to the zero, one, or more TestResultProcessor pointers (or whatever name we come up with) that it's given upon initialization (based on user options) and let those entities worry about what to echo to the terminal, write to files (and in what format), or process in some other way we haven't thought of yet.
Some observers we'll want to implement early on: