https://testanything.org/tap-specification.html#todo-tests says "Should a todo
test point begin succeeding, the harness should report it as a bonus." Meson
chooses to treat that as a form of failure. The Perl modules in which TAP
originated, however, treat it as success. To see this, run
prove --verbose unexpected_pass.pl, with unexpected_pass.pl containing:
unexpected_pass.pl ..
ok 1 - not TODO
ok 2 - UNEXPECTEDPASS # TODO demo
not ok 3 - EXPECTEDFAIL # TODO demo
# Failed (TODO) test 'EXPECTEDFAIL'
# at unexpected_pass.pl line 6.
1..3
ok
All tests successful.
Test Summary Report
-------------------
unexpected_pass.pl (Wstat: 0 Tests: 3 Failed: 0)
TODO passed: 2
Files=1, Tests=3, 0 wallclock secs ( 0.01 usr 0.00 sys + 0.02 cusr 0.00 csys = 0.03 CPU)
Result: PASS
https://testanything.org/tap-specification.html#todo-tests says "Should a todo test point begin succeeding, the harness should report it as a bonus." Meson chooses to treat that as a form of failure. The Perl modules in which TAP originated, however, treat it as success. To see this, run
prove --verbose unexpected_pass.pl
, withunexpected_pass.pl
containing:The return code is zero, and the output is:
I see Meson chose intentionally (search for "unexpected" and "todo"): https://github.com/mesonbuild/meson/pull/4832 https://github.com/mesonbuild/meson/issues/2923 https://github.com/mesonbuild/meson/pull/12362
I'm filing this issue mostly to help other people find it. That said, we could change the documentation or eliminate the difference.