Make test.h use TAP formatted output

[xnox]

This branch changes output:

    Testing nih_main_write_pidfile()
    ...with successful write
    ...with overwrite of existing pid
    ...with failure to write to temporary file

To:

ok 21 - nih_main_write_pidfile() 
ok 22 - nih_main_write_pidfile() with successful write
ok 23 - nih_main_write_pidfile() with overwrite of existing pid
ok 24 - nih_main_write_pidfile() with failure to write to temporary file

Which is in TAP format (Test Anything Protocol).

This results in automake build logs change from:

...
PASS: test_command
PASS: test_config
PASS: test_logging
PASS: test_error
...
============================================================================
Testsuite summary for libnih 1.0.4
============================================================================
# TOTAL: 17
# PASS:  17
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================

To:

PASS: test_error 1 - nih_error_raise()
PASS: test_error 2 - nih_error_raise_printf()
PASS: test_error 3 - nih_error_raise_system()
PASS: test_error 4 - nih_error_raise_no_memory()
PASS: test_error 5 - nih_error_raise_error()
PASS: test_error 6 - nih_error_raise_error() with no current error
PASS: test_error 7 - nih_error_raise_error() with unhandled error
PASS: test_error 8 - nih_return_error()
PASS: test_error 9 - nih_return_system_error()
PASS: test_error 10 - nih_return_no_memory_error()
PASS: test_error 11 - nih_error_steal()
PASS: test_error 12 - nih_error_steal() with same context
PASS: test_error 13 - nih_error_steal() with different contexts
PASS: test_error 14 - nih_error_push_context()
PASS: test_error 15 - nih_error_pop_context()
PASS: test_error 16 - nih_error_pop_context() with unhandled error in context
PASS: test_error 17 - nih_error_pop_context() with unhandled error beneath context
make[3]: Entering directory `/home/xnox/canonical/libnih/libnih/nih'
make[3]: Nothing to be done for `all'.
make[3]: Leaving directory `/home/xnox/canonical/libnih/libnih/nih'
============================================================================
Testsuite summary for libnih 1.0.4
============================================================================
# TOTAL: 839
# PASS:  839
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================

Which is a more accurate and useful build-log output. Furthermore it will enable: skipping tests, marking them expected fail, marking them TODO (unimplemented features), whilst accurately reporting each one of them.

There is no break in testing API and this implementation works across all libnih tests. There is no dependency on executing the test-suite under the TAP runner.

There is one change of behaviour: instead of aborting (bailing) on test failure, a FAIL is reported and the individual test binary will "keep going". I believe this is acceptable as the overall build will fail and is accurately reported. I added a define "TEST_TAP_BAIL_ON_FAIL" which will report a TAP "bail" command and abort as it was done previously. In case one wants previous behaviour.

[keybuk]

With this format, you don't need the "function"/"feature" test banners

[xnox]

Yeah, I'm undecided about how the banners should be done.

From testing point of view, current NIH style tests are straight forward - say what you want to test, do silently or die loud.

TAP feels backwards: do everything, say if did or died, announce what it tried to do.

But indeed function/feature test banners are redundant. Especially when there is no test associated with it.

Also this implementation is very rough, and shredded ARM toolchain into generating hundred thousands of conditional branches out of range assembler errors. But not on Intel / PowerPC. I must be writing really bad code.

[keybuk]

The other problem is that libnih tests aren't intended to be "crash safe", a crash is just a way of failing.

So to be truly TAP-safe you'd have to run each test in a child process and check exit statuses

Otherwise you'll randomly miss out tests if one crashes

On Thursday, November 7, 2013, xnox wrote:

Yeah, I'm undecided about how the banners should be done.

From testing point of view, current NIH style tests are straight forward - say what you want to test, do silently or die loud.

TAP feels backwards: do everything, say if did or died, announce what it tried to do.

But indeed function/feature test banners are redundant. Especially when there is no test associated with it.

Also this implementation is very rough, and shredded ARM toolchain into generating hundred thousands of conditional branches out of range assembler errors. But not on Intel / PowerPC. I must be writing really bad code.

— Reply to this email directly or view it on GitHubhttps://github.com/keybuk/libnih/pull/8#issuecomment-28019488 .

Have you ever, ever felt like this? Had strange things happen? Are you going round the twist?

[xnox]

At the moment the api is fully compatible with existing one, but to introduce skip (e.g. TEST_SKIP_UNLESS) and expect failure (e.g. TEST_FAILED_UNLESS), I would ideally want to wrap the tests, and then potentially run them as child processes.

In practice though, TAP test-suite depends on a TAP test-runner. And as such the automake test-runner does notice many issues:

• when tests arrive out of order
• when duplicate test results arrive
• when no test-plan was announced (or diverted to wrong output...)
• when tests die, abort, or their children are killed by signals
• overall non-zero exit status of individual test binary

So at the moment using test-suite runner as a safety net is good enough, and it does what it is suppose to do. And I believe it is important to preserve current API, to enable tap output for legacy tests with no modifications to tests themselves.

So I will probably add running tests as childs, but in the tap api only (not compatible with existing TEST_#TYPE ("foo bar"))