p5pRT
commented
11 years ago From @jkeenan
Since the St Louis Perl Hackathon in November\, I and others have been writing descriptions for tests in the Perl 5 core distribution's test suite which currently lack them. I have encountered a case where adding descriptions to two tests (a) in one case changes the outcome of the test from PASS to FAIL; and (b) where in both cases the descriptions fail to print -- as if they had never been written in the first place!
Consider tests 4 and 5 in t/op/not.t:
##### # test not(..) and ! is(! 1\, not 1); is(! 0\, not 0); #####
Currently\, none of the tests in t/op/not.t have descriptions. Hence\, the output of running these two tests is as opaque as that of any other test in that file:
##### $ ./perl t/op/not.t 1..16 ok 1 ... ok 4 ok 5 ... ok 16 #####
Now\, suppose we add descriptions to all tests in this file. Here I display only the first five:
##### pass("logical negation of empty list") if not(); is(not()\, 1\, "logical negation of empty list in numeric comparison"); is(not()\, not(0)\, "logical negation of empty list compared with logical negation of false value");
is(! 1\, not 1\, q{description 1}); is(! 0\, not 0\, q{description 0}); #####
The output I get is:
##### $ ./perl t/op/xnot.t 1..5 ok 1 - logical negation of empty list ok 2 - logical negation of empty list in numeric comparison ok 3 - logical negation of empty list compared with logical negation of false value ok 4 not ok 5 # Failed test 5 - at t/op/xnot.t line 19 # got "1" # expected "" #####
Where did the descriptions for tests 4 and 5 go? And why is test 5 now failing?
I posed this question on the St Louis and Toronto Perlmongers mailing lists. Thanks to Nathan Nutter\, Scott Smith\, Liam R E Quin\, Olaf Alders\, Uri Guttman\, Tom Legrady\, Fulko Hew for participating in the discussion.
Adapting a diagnostic suggested by Fulko Hew\, let's set aside the "testing" aspect of the problem and define a function which prints out its arguments:
##### sub examine {print "arg: \<$_>\n" for @_;} #####
Let's first pass to this function the 3 arguments provided to is() above\, but without the 'not ' in the second argument:
##### examine(! 1\, 1\, q{description 1}); examine(! 0\, 0\, q{description 0}); #####
The output:
##### arg: \<> arg: \<1> arg: \<description 1> arg: \<1> arg: \<0> arg: \<description 0> #####
The high-precedence '!' operator binds to the number following it and (apparently) immediately forces that argument to be evaluated. '! 1' evaluates to Perl-false -- hence the empty string between the angle-brackets in the first line of output. '! 0' evaluates to Perl-true -- hence the '1' in the fourth line of output. As we would intuitively expect\, examine() reports 3 arguments in each call. And why do we intuitively expect that? Because the commas inside the function call are functioning as item separators within a list.
Now let's add the string 'not ' to the second argument in each call:
##### print "Prepending 'not ' to second argument\n"; examine(! 1\, not 1\, q{description 1}); examine(! 0\, not 0\, q{description 0}); #####
This time\, our overall output is: ##### arg: \<> arg: \<1> arg: \<description 1> arg: \<1> arg: \<0> arg: \<description 0> Prepending 'not ' to second argument arg: \<> arg: \<> arg: \<1> arg: \<> #####
Here we're seeing the apparently anomalous results we saw when we tried to add a description to tests 4 and 5 in t/op/not.t. We note:
1. The high-precedence '!' operator is continuing to bind to the number following it in each call\, forcing immediate evaluation. The first and third lines following "Prepending" correspond to the first and fourth lines above "Prepending".
2. Each call\, however\, is only reporting two "arguments" -- and each "argument" is being evaluated to an empty string (second and fourth lines following "Prepending").
In the discussion on the Perlmonger mailing lists\, Uri Guttman observed that 'not' isn't just a lower precedence version of !. It is also a named function with a prototype.
##### ./perl -le 'print prototype "CORE::not"' $; #####
Since 'not' as a function is prototyped to expect a scalar argument\, the comma to the right of each of 'not 1' and 'not 0' is no longer functioning as an item separator in list context. Rather (to quote 'perlop')\, "[i]n scalar context it evaluates its left argument\, throws that value away\, then evaluates its right argument and returns that value." The "right argument"\, in these cases\, are q{description 1} and q{description 0}. When evaluated by 'not' -- now serving as a unary negation operator -- the expressions evaluate to Perl-false. Hence the empty strings between the angle brackets in the second and fourth lines after "Prepending".
If you adapt the diagnostic suggested by Fulko to use either Test::More::is() or the is() from 't/test.pl'\, you should get similar results.
This suggests that tests in 4 and 5 in t/op/not.t are not conducting a precise test of what you would at first think they are testing. As written\, you would think they're testing that '! 1' and 'not 1' "do the same thing" and that '! 0' and 'not 0' "do the same thing" as well. But the fact that adding a description to the test affects the outcome of the test when run in a testing context means that the tests are not spelled sufficiently precisely.
Hence\, I propose we apply the two patches attached. The first merely puts parentheses around the second argument in each of tests 4 and 5. The second adds descriptions to those two tests and to every other test in the file as well.
Please review.
Thank you very much. Jim Keenan
Migrated from rt.perl.org#116242 (status was 'resolved')
Searchable as RT116242$