batterseapower / test-framework

Framework for running and organising QuickCheck test properties and HUnit test cases
http://bsp.lighthouseapp.com/projects/15661-hs-test-framework
83 stars 45 forks source link

Support for additional test reporters #39

Open fatuhoku opened 11 years ago

fatuhoku commented 11 years ago

The default output format is great; but it's not suitable for consumption by build management, reporting, or CI systems.

To make test-framework become a viable part of a software development pipleine, it's useful to support other formats of test result outputs.

For instance, Mocha supports a vast range of variably useful reporters:

$ mocha --reporters

    dot - dot matrix
    doc - html documentation
    spec - hierarchical spec list
    json - single json object
    progress - progress bar
    list - spec-style listing
    tap - test-anything-protocol
    landing - unicode landing strip
    xunit - xunit reportert
    teamcity - teamcity ci support  ***
    html-cov - HTML test coverage
    json-cov - JSON test coverage
    min - minimal reporter (great with --watch)
    json-stream - newline delimited json events
    markdown - markdown documentation (github flavour)
    nyan - nyan cat!

For instance, Mocha's spec reporter looks like this: Mocha's spec reporter

This issue requires refactoring that would abstract from the way test progress and information is printed onto standard out at the moment into something more versatile.

Test reporters can have

Some part of this is necessary for #38 to be implemented. It's questionnable whether this issue can become a part of the #38 altogether.

fatuhoku commented 11 years ago

Much of the existing test runner/reporter lives in Run.hs. We must tear apart logic that creates Doc from putDoc (which outputs it directly onto standard output). This would provide several extension points where Docs can be output. The need for this is most evident in the amount of functions in the IO monad in that source file.

UnkindPartition commented 11 years ago

Is there more to this than alternative runners? test-framework already supports XML output as an alternative runner, for example.