TAP xfail fails

[matthiasclasen]

I have a tap test that is failing. I mark it as should_fail: true. meson continues to report ERROR for it and fails my testsuite run.

This is with meson 0.55.3.

The test that gives me trouble is added here:

https://gitlab.gnome.org/GNOME/gtk/-/blob/0f5b9d8058cc5f6ae17b82dea0fb5b30a39c0867/testsuite/gtk/meson.build#L180

Looking at the tap parser code, this is not too suprising:

       if returncode != 0:
            res = TestResult.ERROR
            stde += '\n(test program exited with status code {})'.format(returncode,)

        if res is None:
            # Now determine the overall result of the test based on the outcome of the subcases
            if all(t is TestResult.SKIP for t in results):
                # This includes the case where num_tests is zero
                res = TestResult.SKIP
            elif test.should_fail:
                res = TestResult.EXPECTEDFAIL if failed else TestResult.UNEXPECTEDPASS
            else:
                res = TestResult.FAIL if failed else TestResult.OK

It never looks at should_fail if returncode is != 0.

Given that the tap protocol explicitly doesn't define the exit code, that seems wrong.

[bonzini]

Why is the test returning a non-zero exit code? Generally, TAP producers return zero for the exit code even if there are failures, for example:

$ cat f
#! /bin/sh
echo 1..1
echo ok 1
exit 1
$ chmod +x f
$ prove ./f
./f ..  Dubious, test returned 1 (wstat 256, 0x100)
All 1 subtests passed 

Test Summary Report
-------------------
./f (Wstat: 256 Tests: 1 Failed: 0)
  Non-zero exit status: 1
Files=1, Tests=1,  0 wallclock secs ( 0.02 usr +  0.01 sys =  0.03 CPU)
Result: FAIL

[bonzini]

Historically, nonzero exit codes were used instead of "bail out" lines, so this is a GLib or more likely GTK+ bug in my opinion (it's hard to change GLib at this time). The workaround is simply to call g_test_run() instead of return g_test_run().

[bonzini]

Also, an even better way to fix it is to mark the test as XFAIL in the C source rather than in Meson using g_test_incomplete.

[matthiasclasen]

It is not that easy to mark tests as incomplete - they may contain dozens of assertions, and I would have to replace each of them with an equivalent if (!condition) g_test_incomplete()

[matthiasclasen]

Why is the test returning a non-zero exit code? Generally, TAP producers return zero for the exit code even if there are failures, for example

Here is what the tap spec says:

A harness should treat a test program as a failed test if:

    The TAP output lines indicate test failure, or
    The TAP output of the process is invalid in a way that is not recoverable, or
    The exit code of the test program is not 0 (including test programs killed by a fatal Unix signal).

I don't think you can make assumptions about TAP producers beyond what the protocol prescribes if you want to support existing producers.

As it is, the meson xfail support is useless with code using GLib test support.