ciscoheat / buddy

Your friendly BDD testing library for Haxe!
MIT License
96 stars 24 forks source link

Ideas for reporters. #23

Closed brendanjerwin closed 10 years ago

brendanjerwin commented 10 years ago

I'm using Buddy primarily on a code base that targets Flash, so accept my twisted perspectives with that in mind. :)

I eventually want to get my Buddy specs running in Continous Integration, and I'm also a big fan of command line test runners.

The flash tests, though, need to run in the Flash Debugger (I use the open command on my Mac).

It'd be great if all the tests, no matter the target, reported their results back to a central command line test runner, who was then in charge of displaying the results in a common manner.

I've recently done some testing where my tests work against a node.js server running along side them. It seems to work well enough (once all the cross-origin stuff was worked out..) Perhaps a test runner could start a server and then launch the individual target's test runners. Those 'child' runners would communicate their results back to the server on a well-known port which would then handle the job of actually displaying the results on screen (or writing them to file in the JUnit-style for CI!)

ciscoheat commented 10 years ago

Hi, do I understand correctly that you want a more flexible way of reporting? I think that would be done best by implementing your own Reporter (it has a simple interface, https://github.com/ciscoheat/buddy/blob/master/src/buddy/reporting/Reporter.hx) and do what you want from there. I can add both a command-line and compilation option to specify a custom reporter. What do you think?

For CI right now, Buddy reports the test status on command-line as 0 of ok, and 1 if failed. It uses Travis for its own testing: https://travis-ci.org/ciscoheat/buddy

brendanjerwin commented 10 years ago

More interested in being able to run the tests in a target-appropriate context, flash and js in the browser, for instance, but have the results always arrive in the same way.

In my use-case, the tests run fairly nicely, in my terminal, when I run the neko tests, but the flash tests run in the flash debugger, in their own window, with flash rendering the text... No way to further process the results for things like CI.

This may be a more critical need for the flash target, since I see you've found a way to wire in most all of the other targets in your travis config.

brendanjerwin commented 10 years ago

One more use-case: running Flash tests within a browser. There are times I need to exercise Flash in that particular sandbox situation as well...

ciscoheat commented 10 years ago

After a bit of messing around, flash is now integrated in Travis. There is now a -D fdb-ci flag that will modify the swf file to output to trace and most importantly, exit with a status code when tests are done.

Check out https://github.com/ciscoheat/buddy/blob/master/flash-travis-setup.sh and https://github.com/ciscoheat/buddy/blob/master/buddy.flash.hxml if you want to use this technique. Maybe it'll solve your problems?

ciscoheat commented 10 years ago

And now in version 0.14.0 you can easily add custom reporters: https://github.com/ciscoheat/buddy#customizing-reporting

Now you should have everything you need to solve this. :)

ciscoheat commented 10 years ago

Let me know if you need anything else, otherwise I'm closing this.

brendanjerwin commented 9 years ago

I'm finally attempting to wrestle the fdb tool from the Flex SDK into submission. I have high hopes! I'll let you know if it pans out. (I'm on OS X BTW.)

brendanjerwin commented 9 years ago

Alright. Wrestling isn't going well...

I can't seem to get the output into my shell at all. Compiling with the fdb_ci switch does cause Flash to exit, but nothing is showing up in the console when launched with fdb or with open file.swf. If I don't run with that switch, but do use the Trace Reporter, I see that it is still using the normal Haxe trace facility, writing the output on the flash window.

I thought I might be able to get to the flash trace file but cat ~/Library/Preferences/Macromedia/Flash\ Player/Logs/flashlog.txt shows nothing. So it seems like the output is just not going anywhere I can access.

Is there supposed to be an override of the normal trace somewhere?

ciscoheat commented 9 years ago

I've been using trace to send output to flashlog.txt, after some tinkering: https://github.com/ciscoheat/buddy/blob/master/flash-travis-setup.sh and https://github.com/ciscoheat/buddy/blob/master/flash-travis-run.sh

But I'm not actually using the trace output, exiting with a status code is the only thing needed for Travis.

brendanjerwin commented 9 years ago

Oops. I had failed to compile the .swf with -D fdb. Now I'm getting trace output.