TAP is great. Simplicity is its strength. Adding complexity should trigger caution. That said, I would suggest extending the directives capability.
The objective is to allow test harnesses to adjust their testing suites to their purposes without reinventing the wheel. The need arose for us when we wanted to integrate our TAP based testing in our devops workflow, binding a test run to a software release. All our projects are hosted on Github (whether open source or enterprise) and we wanted to keep track of associated issues, projects, milestones and tags.
All this could be achieved by simply adding custom directives to our existing GNU Autotools based build process and qualifying the build : nightly build, latest stable, release x.y.z, ...
Likewise, by binding our tests to specific issues, we can assert that bugs have been fixed, and that new features have been integrated; and that everything has tested.
Here are my suggestions:
Directive tags are no longer case sensitive and must be uppercase
Directives are introduced by a hash symbol followed by whitespace
As per TAP13, directives can be followed by a textual description, provided that string is preceded by at least one whitespace character
Directives can be added to the test line (as per TAP13) or immediately follow a test line
Each test line can have multiple associated directives, one per line
Directives are not pre-defined in the TAP specifications (except perhaps, historical TODO and SKIP)
Harnesses only process directives they know about and ignore others (à la HTML/SGML)
This allows, in particular, to adjust test suites to devops workflows. In the example below we introduce a PATCH directive which is synonymous to TODO on next patch release, and a FEATURE directive which is synonymous to SKIP unless preparing next major/minor release. The ISSUE directive binds a given test line to a reported issue, for instance on Github.
not ok 21 # PATCH patch-856 Git branch for patch
# ISSUE 123456789 Memory leak
# ISSUE 123456798 Invalid linked array
not ok 22 # FEATURE 12.1 New feature for release 12.1
# ISSUE 123456987 Command line option
# - ISSUE 123456987 This is a comment and not a directive
On could argue that such information can easily be integrated in an associated YAML block. I would disagree, for two major reasons:
Directives are part of the test harness as they bring light on the true/false status of a test, categorising it and eventually inverting the result. A given test harness will not operate optimally without its specific directives, whereas it can probably live without YAML blocks.
But more importantly, YAML blocks incur significant parsing overhead, while parsing directives, thanks to the TAP syntax, is far easier -- almost a one liner in Korn shell.
TAP is great. Simplicity is its strength. Adding complexity should trigger caution. That said, I would suggest extending the directives capability.
The objective is to allow test harnesses to adjust their testing suites to their purposes without reinventing the wheel. The need arose for us when we wanted to integrate our TAP based testing in our devops workflow, binding a test run to a software release. All our projects are hosted on Github (whether open source or enterprise) and we wanted to keep track of associated issues, projects, milestones and tags.
All this could be achieved by simply adding custom directives to our existing GNU Autotools based build process and qualifying the build : nightly build, latest stable, release x.y.z, ...
Likewise, by binding our tests to specific issues, we can assert that bugs have been fixed, and that new features have been integrated; and that everything has tested.
Here are my suggestions:
This allows, in particular, to adjust test suites to devops workflows. In the example below we introduce a
PATCH
directive which is synonymous toTODO
on next patch release, and aFEATURE
directive which is synonymous toSKIP
unless preparing next major/minor release. TheISSUE
directive binds a given test line to a reported issue, for instance on Github.On could argue that such information can easily be integrated in an associated YAML block. I would disagree, for two major reasons:
Directives are part of the test harness as they bring light on the true/false status of a test, categorising it and eventually inverting the result. A given test harness will not operate optimally without its specific directives, whereas it can probably live without YAML blocks.
But more importantly, YAML blocks incur significant parsing overhead, while parsing directives, thanks to the TAP syntax, is far easier -- almost a one liner in Korn shell.