a plugin-based reporting mechanism for shunit2

[GoogleCodeExporter]

I see that you already have a ticket offering JUnit/XML ( #16 ) and another 
suggesting TAP ( #8 ).  Neither of those formats collects all of the data that 
I need.  I suggest a generalized plugin architecture is a better answer than 
coding these specific formats into shunit2.

I wanted shunit2 to send results to my Pandokia test reporting system, so I 
implemented a plugin architecture and a plugin that produces data files in the 
pandokia format.

I propose that you consider this plugin architecture (but not the pandokia 
plugin) for inclusion in a future version of shunit2.  I am attaching:

- a patch to shunit2 that implements a plugin architecture for shunit2 to write 
test reports in other formats; this includes a default plugin (built in to the 
shunit2 script) that preserves the existing behaviour of shunit2.

- an example plugin that makes something that looks vaguely like XML; this 
could be enough clues for someone more familiar with JUnit/XML or TAP to write 
a suitable plugin for those formats.  (I can't implement them myself because I 
don't know anything about them, I don't have a test environment, and I don't 
need them.)

The plugin that produces Pandokia logs does not belong in shunit2, but you can 
see it at 
https://svn.stsci.edu/trac/ssb/etal/browser/pandokia/trunk/commands/shunit2_plug
in_pdk .  It might be useful as an example while you are evaluating the patch.

The general principle of the patch is this:

Various points in the shunit2 execution are functions that can be redefined.  
If you invoke "shunit --plugin name file.sh", it uses ". shunit_plugin_$name" 
to load functions from that file into the shunit2 script.  The various plugin 
functions are either redefined by the plugin or left at the default 
implementation, as appropriate.  Test execution then proceeds as usual.

Some notable features of the code:

The plugin entry points are argv processing, init, execute a test, final 
report, and finish.  If you identify additional entry points, they are easy to 
define as further patches to shunit2.

The global variable _shunit_last_test_status reports the status of the most 
recently completed test.  _shunit_assertPass, _shunit_assertFail, and 
_shunit_assertSkip update the status.  The status is available to the plugin to 
include in a report.  I see that shunit2 tests often contain multiple 
assertions in a single test function.  The status is only allowed to go from 
less serious to more serious during a single test function.  That is, Pass can 
change to Fail if another assertion is executed, but a second assertion can 
never change Fail to Pass.

I principally run tests with "shunit2 filename", not by writing the file to 
contain ". shunit2"; this is the mode that the plugin works best.  I know of 
two problems related to this in the patch as it stands now:

1) Plugins can be loaded by "shunit2 --plugin".  I did not yet figure out a 
good way to make the plugin architecture work universally in 
__SHUNIT_MODE_SOURCED.  The problem I found is that shunit2 is sourced many 
times, so the plugin init and finish are called more than once each.  I think 
there is a solution that involves counting how deeply the shunit processes are 
nested, but I have not worked out the details.

2) I can't seem to run any of the tests as single files, except for 
shunit2_test_standalone.sh;  for example, if I type "shunit2 
shunit2_test_misc.sh", it gives an error instead of running tests.  That same 
error comes out if I try to run shunit2_test.sh with a plugin.  I think that 
the case of "shunit2 file.sh" should work even if file.sh ends with ". 
shunit2"; somehow the second invocation of shunit2 should recognize that it is 
redundant and not do anything.  I think that is a separate patch, though.

I am including a sample test file and output that it produces in various modes:

sample.shunit - a sample set of tests

PDK_DEFAULT.LOG.sample - pandokia log file from "pdkrun --test_run sample 
sample.shunit"; possibly not especially interesting to you, but for me, this 
was the point of doing all this.  The format is human readable and fairly 
obvious.

example.txt - stdout of "shunit2 --plugin example --file sample.xml 
sample.shunit"

sample.xml - xml file produced by "shunit2 --plugin example --file sample.xml 
sample.shunit"

regression.txt - output of "./shunit2_test.sh > regression.txt" shows that the 
original tests still pass

The actual patch is attached as:

shunit2.patch - svn diff of shunit2 comparing 
http://shunit2.googlecode.com/svn/trunk/source/2.1/src/shunit2 rev 337 to my 
changed copy

shunit2_plugin_example - example plugin; not useful itself, but as a basis for 
experimenting

You'll note that this patch includes only code, no documentation.  If you 
accept the patch, I offer to write documentation updates.  If you decline the 
patch, I still have my locally modified copy, but I do not need to document it 
further.

I would appreciate your comments about whether you view this as something that 
you would incorporate into shunit2.  Obviously, with the amount of work I've 
put in to it, I will need to make alternate plans if you are not interested.  
If you have an immediate response, even if it is not definite, please let me 
know.

Regards,

Mark

Original issue reported on code.google.com by sienkiew...@gtempaccount.com on 16 Sep 2011 at 5:57

Attachments:

• sample.shunit
• PDK_DEFAULT.LOG.sample
• example.txt
• sample.xml
• regression.txt
• shunit2.patch
• shunit2_plugin_example

[GoogleCodeExporter]

b.t.w.  Meant to be "enhancement" not "defect", but I never saw that option 
when I entered the ticket.

Original comment by sienkiew...@gtempaccount.com on 16 Sep 2011 at 5:58

[GoogleCodeExporter]

I noticed recently that there is an oddity in the patch.  The line that says

+  if [ "$1" == "--plugin" ]

should have only a single = in it.

(For reasons too dumb to go in to, I also have to work on a lot of csh scripts. 
 This mistaked happened to get through because it works in bash -- which is all 
I have on most of the machines here.)

Original comment by sienkiew...@gtempaccount.com on 21 Dec 2011 at 9:48

[GoogleCodeExporter]

Such a feature would be a nice addition. As the recommended way to use shunit 
is to source it, it would be nice if the output plugin can be selected via ENV 
var. Then one can just add SHUNIT_OUTPUT="tap" to TEST_ENVIRONMEN in e.g. 
Makefile.am.

Original comment by st.k...@gmail.com on 24 Mar 2014 at 10:36