Open honzajavorek opened 8 years ago
Good idea I think. Might improve the integration possibilities. +1
+1 Would be useful to add something like this https://www.npmjs.com/package/newman#json-reporter-options. This would make it easier to integrate with CI tools
I think this should be implemented using the TAP protocol: https://testanything.org/
Let's do this. The idea is:
--reporter=cli
, --reporter=apiary
, or --reporter=tap
to indicate which reporter they want to use. They should be able to somehow use multiple reporters (e.g. to see CLI output and to produce xunit file at the same time), on both NIX and Windows. It needs investigation whether this is possible with TAP. We also need to write docs on this with ready-made examples for NIX and Windows. And we should test the basic cases. This is gonna be a breaking change for Dredd, so let's have some solid "migration docs" in place when the version hits the npm. The --output
option would possibly disappear.hooks.log
, console.log
, and hook runners stdout?Following needs to be answered before we do this:
| > < >> <<
redirections, I did not expect that. Thus this is probably a non-issue, I'd test it though.@garyluu @juliengerboud @mlambley I noticed you are using the HTML reporter. What is your use case for it? Why do you prefer it?
Thinking about this further:
dot
, xunit
, nyan
reporters can be easily provided by tap-
libraries.html
and markdown
reporters we have right now are so low-quality I'm actually surprised anyone is using them. The HTML tap-
libraries are better than what we have, so let's recommend them. I could not find a single person in Dredd issues using the markdown
reporter. I wouldn't be very sad to deprecate it without replacement. After all, anyone is welcome to build whatever they like on top of TAP, the data are going to be there.So the remaining issue is how to support multiple reporters. I've seen quite a few people here to use Dredd with dot reporter together with apiary or html reporters, as in
as a Dredd user I can see whether anything failed in the console and I can go to the file to see the verbose output
TAP steals stdout, so that wouldn't be possible. And the stderr would be probably ocuppied by redirected server and hooks output... 🤔
didn't end up using the html reporter, but i was trying to create a web link that quickly displayed the results
Regarding the stdout
noise of parent process coming from server and hook instances - it could be good idea to leverage subprocess.send() method to handle incoming information from server/hooks and decide in main Dredd's process what should go to stdout
instead of just writing directly to main process' stdout
from server/hook processes.
@honzajavorek The idea was to "store" the report as an artifact of TeamCity build. This way, users checking a build creating the report would just had to click on the file and the report would have been display directly into the browser.
With @michalholasek we designed following steps to take:
--names
option, and other parts of Dredd.Regarding the server/hooks output, we can capture it as suggested in https://github.com/apiaryio/dredd/issues/593#issuecomment-251946163 and put it into special TAP comments as part of the stdout. The stderr would be reserved for output of the debug
utility.
@garyluu @juliengerboud Thanks for sharing! Did you need the CLI output as well or was the HTML file sufficient for you? I mean, if you run Dredd like you described, did you care about what it outputs to the console at all?
The last time I tried, I needed the CLI output as well (in order to get the summary): https://github.com/apiaryio/dredd/issues/1001
@honzajavorek The CLI log is quite useful! It is easier to read IMHO.
It's been... quite a while, but I'll try to remember my use case. I wanted the cli to give brief information (a single line for each test, showing perhaps method, url, result). The problem I had was that the cli reporter was cluttered with json dumps of failed tests. There were two options for the cli reporter: show errors inline or show errors at the end. I wanted the option to not show errors at all.
The dot reporter wasn't ideal either because you couldn't see the test information (and if memory serves, it still dumped json data anyway).
A HTML file is a much easier place to review long strings of json and identify where the errors are. I wanted the cli reporter to give summary info, with detailed information in the html.
I don't know how TAP will fit into this, but my suggestion was to have an interface for creating custom reporters and be able to configure dredd to say "read extra reporters from this directory". I think another suggestion I had was to create a "quiet cli reporter" which was a copy of cli reporter with all of the "show errors" code removed.
...or a different easily machine-readable format. Would significantly simplify writing some integration tests (such as those in #338). Filing this issue so people can add other use cases - probably not worth implementing just because of couple of tests.