mesonbuild / meson

The Meson Build System
http://mesonbuild.com
Apache License 2.0
5.3k stars 1.52k forks source link

Meson and Perl disagree on UNEXPECTEDPASS TAP test point failing the test file #13183

Open nmisch opened 2 weeks ago

nmisch commented 2 weeks ago

https://testanything.org/tap-specification.html#todo-tests says "Should a todo test point begin succeeding, the harness should report it as a bonus." Meson chooses to treat that as a form of failure. The Perl modules in which TAP originated, however, treat it as success. To see this, run prove --verbose unexpected_pass.pl, with unexpected_pass.pl containing:

use Test::More;
ok(1, 'not TODO');
Test::More->builder->todo_start('demo');
ok(1, 'UNEXPECTEDPASS');
ok(0, 'EXPECTEDFAIL');
Test::More->builder->todo_end;
done_testing();

The return code is zero, and the output is:

unexpected_pass.pl .. 
ok 1 - not TODO
ok 2 - UNEXPECTEDPASS # TODO demo
not ok 3 - EXPECTEDFAIL # TODO demo
#   Failed (TODO) test 'EXPECTEDFAIL'
#   at unexpected_pass.pl line 6.
1..3
ok
All tests successful.

Test Summary Report
-------------------
unexpected_pass.pl (Wstat: 0 Tests: 3 Failed: 0)
  TODO passed:   2
Files=1, Tests=3,  0 wallclock secs ( 0.01 usr  0.00 sys +  0.02 cusr  0.00 csys =  0.03 CPU)
Result: PASS

I see Meson chose intentionally (search for "unexpected" and "todo"): https://github.com/mesonbuild/meson/pull/4832 https://github.com/mesonbuild/meson/issues/2923 https://github.com/mesonbuild/meson/pull/12362

I'm filing this issue mostly to help other people find it. That said, we could change the documentation or eliminate the difference.