A test running addendum?

[Leont]

Currently we're only describing the TAP stream and nothing around it. I think it may be useful to also (probably separate from the main document?) to write something about running the programs that output TAP. In particular

• Interpreting the return value of the program.
• Using an environmental variable to hint to the TAP producer what versions of TAP the consumer wants to consume (e.g. TAP_VERSIONS=14,13,12)

[ferdnyc]

@Leont

• Using an environmental variable to hint to the TAP producer what versions of TAP the consumer wants to consume (e.g. TAP_VERSIONS=14,13,12)

That sounds to me like an implementation detail and an unspecifiable requirement at the same time.

• Producers aren't required to support outputting any version of TAP other than the one they output (as indicated by the version line), so in terms of practical requirements there's no reason for them to support that envvar either.

• For producers that do support configurable output, does it really benefit anyone to constrain them as to how? They may accept command-line args, they may take a config file, they may respect an envvar. Any of those can work, and depending on the testing scenario each approach can be a better fit than the others. Experience says most tools end up doing "all of the above", if not more.

(Envvars are great in a CI environment or similar, but they suck if you're distributing tests with a public project. Now you have to try and find a way to ensure that everyone who runs the tests has the right variables set in their environment — which won't happen. Whereas, if you can check in a config file that the producer will automatically pick up, problem solved.)

For exit codes, the spec already says (In §Harness Behavior):

A harness should treat a test program as a failed test if:

• The TAP output lines indicate test failure, or
• The TAP output of the process is invalid in a way that is not recoverable, or
• The exit code of the test program is not 0 (including test programs killed by a fatal Unix signal).

If one or more test programs are considered failures, then a TAP Harness should indicate failure to the user in whatever means are appropriate. For example, a command-line test runner might exit with a non-zero status code, while a web-based continuous integration system might put a red "FAILED" notice in the html output.

Which I think is appropriately vague, in the same "different approaches for different scenarios" way.