search

freebsd / kyua

Port/Package build and test system
https://github.com/freebsd/kyua/wiki
BSD 3-Clause "New" or "Revised" License
149 stars 42 forks source link

kyua is too strict about TAP SKIP commands #159

Closed brooksdavis closed 8 years ago

brooksdavis commented 8 years ago

The TAP specification (http://testanything.org/tap-specification.html) states any comment in the output that begins with "skip" is a SKIP command and explicitly includes this example:

1..0 # Skipped: WWW::Mechanize not installed

FreeBSD's usr.bin/sed/tests/inplace_race_test.sh produces such a line when running on a non-SMP system, but kyua rejects it with:

inplace_race_test:main  ->  broken: Test plan is reversed  [0.502s]

Changing "Skipped:" to "SKIP" causes the test to be skipped properly.

inplace_race_test:main  ->  skipped: : not an SMP system  [0.489s]

A few other tests suggest it is looking for an upper case skip. For example, making the line start with SKIPPED: yields:

inplace_race_test:main  ->  skipped: PED: not an SMP system  [0.509s]
jmmv commented 8 years ago

Grr, that TAP "specification" is so poorly specified that it doesn't deserve the name. The mention that directives are case insensitive is buried in a random place in the document, and the fact that "Skipped:" is a valid directive only appears in a sample line and is not explained anywhere. May this be a mistake on their side?

Anyway, changed the parser to accept these and other cases. Thanks!