RFC: Extending directives

[marcastel]

TAP is great. Simplicity is its strength. Adding complexity should trigger caution. That said, I would suggest extending the directives capability.

The objective is to allow test harnesses to adjust their testing suites to their purposes without reinventing the wheel. The need arose for us when we wanted to integrate our TAP based testing in our devops workflow, binding a test run to a software release. All our projects are hosted on Github (whether open source or enterprise) and we wanted to keep track of associated issues, projects, milestones and tags.

All this could be achieved by simply adding custom directives to our existing GNU Autotools based build process and qualifying the build : nightly build, latest stable, release x.y.z, ...

Likewise, by binding our tests to specific issues, we can assert that bugs have been fixed, and that new features have been integrated; and that everything has tested.

Here are my suggestions:

• Directive tags are no longer case sensitive and must be uppercase
• Directives are introduced by a hash symbol followed by whitespace
• As per TAP13, directives can be followed by a textual description, provided that string is preceded by at least one whitespace character
• Directives can be added to the test line (as per TAP13) or immediately follow a test line
• Each test line can have multiple associated directives, one per line
• Directives are not pre-defined in the TAP specifications (except perhaps, historical TODO and SKIP)
• Harnesses only process directives they know about and ignore others (à la HTML/SGML)

This allows, in particular, to adjust test suites to devops workflows. In the example below we introduce a PATCH directive which is synonymous to TODO on next patch release, and a FEATURE directive which is synonymous to SKIP unless preparing next major/minor release. The ISSUE directive binds a given test line to a reported issue, for instance on Github.

not ok 21 # PATCH patch-856 Git branch for patch
# ISSUE 123456789 Memory leak
# ISSUE 123456798 Invalid linked array

not ok 22 # FEATURE 12.1 New feature for release 12.1
# ISSUE 123456987 Command line option
# - ISSUE 123456987 This is a comment and not a directive

On could argue that such information can easily be integrated in an associated YAML block. I would disagree, for two major reasons:

1. Directives are part of the test harness as they bring light on the true/false status of a test, categorising it and eventually inverting the result. A given test harness will not operate optimally without its specific directives, whereas it can probably live without YAML blocks.

2. But more importantly, YAML blocks incur significant parsing overhead, while parsing directives, thanks to the TAP syntax, is far easier -- almost a one liner in Korn shell.

[marcastel]

Oops. Mea culpa. Added this RFC to the website repository. Not the right place. Where should I relocated it ? Cheers.