format of TAP output for skipped tests is incorrect

[harschware]

The current TAP output of BATS is incorrect TAP format for skipped tests. e.g. bats -t fixtures/bats/skipped.bats 1..2 ok 1 # skip a skipped test ok 2 # skip (a reason) a skipped test with a reason

The problem is that the description should precede the comment, and everything after 'skip' is the reason for skipping the test. TAP consumers then do not get the description when using this format.

Here is a screenshot (http://screencast.com/t/qpqbMbk0jFLJ) of Jenkins Tap Plugin results for the following TAP input: 1..2 ok 1 # skip for a really good reason ok 2 skip test with a reason # skip for a really good reason

The following Gist shows what the correct format is using perl's TAP::Parser module: https://gist.github.com/harschware/baec4c05de8e700924ad

[ztombol]

@harschware Could you link the relevant section of the TAP specification?

[harschware]

There isn't an example in the TAP specification of a test with description that is also skipped, which is why I showed two TAP consumers that now work as expected when the TAP output is corrected. But as you can see here: http://screencast.com/t/uXzdoKpoP the general format shows clearly that a test output contains a description followed by a directive. There is an example just below that where it shows a TODO directive that does it correctly (TODO and skip are both directives per the specification). For whatever reason for all the examples of skip they remove the description, which is allowable per the specification. In the case of BATS, it was a false assumption that the description should be moved into the comment section and doing so causes TAP compliant consumers (see my two citations above) to not find a valid description.

[harschware]

Is there anyone who can merge the PR. It is correct, and it fixes the stated issue.

[ztombol]

@harschware I've been planning to try your patch. Hopefully, I can get around doing it sometime next week.

[ztombol]

@harschware You are right. The output does not conform to the specification.

The TAP specification does actually describe the format clearly. Here are the relevant parts.

• Description

Any text after the test number but before a # is the description of the test point.

ok 42 this is the description of the test

Descriptions should not begin with a digit so that they are not confused with the test point number. The harness may do whatever it wants with the description.

• Directive

The test point may include a directive, following a hash on the test line. There are currently two directives allowed: TODO and SKIP. These are discussed below.

And:

Directives

Directives are special notes that follow a # on the test line. Only two are currently defined: TODO and SKIP. Note that these two keywords are not case-sensitive.

[ORESoftware]

zoom