search

Perl-Critic / PPI

53 stars 44 forks source link

t/21_exhaustive.t: random failures (PPI-1.262) #239

Closed eserte closed 5 years ago

eserte commented 5 years ago

It seems that random failures in t/21_exhaustive.t are possible. One example:

# input:  "c\${&^!bs%r?;^0wm.1._,q\n*r^]Xg:~|\t<&'_Vs;mr|/\n}=\n-#9!>x9>='_<#,'#^]>9{qmg^%'g( /qytc.?,\n ~f&>0+&gy&V,\@&fg>/f {!+-bVs9?9c%1\"1^{(_r)w`/%~qb>*|#,f,-1]~<c\$`g#19\"\t]s;*=,f_:V`f<;\"%\@\\V0\$^;W~bz;?!]9."
# output: "c\${&^!bs%r?;^0wm.1.,q\n*r^]Xg:~|\t<&'_Vs;mr|/\n}=\n-#9!>x9>='_<#,'#^]>9{qmg^%'g( /qytc.?,\n ~f&>0+&gy&V,\@&fg>/f {!+-bVs9?9c%1\"1^{(_r)w`/%~qb>*|#,f,-1]~<c\$`g#19\"\t]s;*=,f_:V`f<;\"%\@\\V0\$^;W~bz;?!]9."
# shorted failing substring: ""

#   Failed test 'round trip successful'
#   at t/21_exhaustive.t line 112.
# Looks like you failed 1 test of 1114.
t/21_exhaustive.t ................ 
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1114 subtests

Another example:

Use of uninitialized value $round_tripped in string eq at t/21_exhaustive.t line 139.
# input:  "_+#8r\tt`-[\n<+0cXV!=.?c!c0=)<<~')19_-b+?WW'[z q-y?#m\@Wt>.'.xg=}|*w|:+[|9}\"mW:Xf()?(cx9*<\t\nVx\$*=z09 z&VX`~1/^\\/,+f}\$_[%q{VW,/b_`9wxx\n\n:;gg\")~\n~x{)f1<'~y1qsf.-b\@V- `;9t\\>{`9-'\$[\@zt;x>%_xt=(\t!\@0"
Use of uninitialized value $quotable in substitution (s///) at t/lib/PPI/Test.pm line 30.
Use of uninitialized value $quotable in substitution (s///) at t/lib/PPI/Test.pm line 31.
Use of uninitialized value $quotable in substitution (s///) at t/lib/PPI/Test.pm line 32.
Use of uninitialized value $quotable in substitution (s///) at t/lib/PPI/Test.pm line 33.
Use of uninitialized value $quotable in substitution (s///) at t/lib/PPI/Test.pm line 34.
Use of uninitialized value $quotable in substitution (s///) at t/lib/PPI/Test.pm line 35.
Use of uninitialized value $round_tripped_quoted in concatenation (.) or string at t/21_exhaustive.t line 144.
# output: ""
# shorted failing substring: "_+#8r\tt`-[\n<+0cXV!=.?c!c0=)<<~')19_-b+?WW'[z q-y?#m\@Wt>.'.xg=}|*w|:+[|9}\"mW:Xf()?(cx9*<\t\nVx\$*=z09 z&VX`~1/^\\/,+f}\$_[%q{VW,/b_`9wxx\n\n:;gg\")~\n~x{)f1<'~y1qsf.-b\@V- `;9t\\>{`9-'\$[\@zt;x>%_xt=(\t!\@0"

#   Failed test 'round trip successful'
#   at t/21_exhaustive.t line 112.
# Looks like you failed 1 test of 1114.
t/21_exhaustive.t ................
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1114 subtests
wchristian commented 5 years ago

Okay, yeah, cool. I broke that. Thanks. :)

wchristian commented 5 years ago

Fixed in https://metacpan.org/release/MITHALDU/PPI-1.264 Thanks a lot. :D

eserte commented 5 years ago

It seems that t/21_exhaustive.t can still fail with PPI-1.264. A sample fail report already on CPAN Testers: http://www.cpantesters.org/cpan/report/bcd8f890-6bd5-11e9-9761-74b8c862b317

And a failure on one of my smokers:

Use of uninitialized value $round_tripped in string eq at t/21_exhaustive.t line 139.
# input:  "<<~` (y0<\"wsxV?[\tV q`+\"<X)[/b&g+.{\\`b/)z]\\%=?s\@ x,`\@1q,^{Vs0)gtm_^0*;>{/*9's,\t!\\*\"r\n%*:>?qzqs|0#m0!-\"'=\t\@^{\$8z&ww[(WX-Vb:[%(y0\${g[ttWzX_*=r_=9==&c|8W\"wq1|%[_\\[/)8<+%x8&s+W+8X:gz?b%q1xft&x&)f"
Use of uninitialized value $quotable in substitution (s///) at t/lib/PPI/Test.pm line 30.
Use of uninitialized value $quotable in substitution (s///) at t/lib/PPI/Test.pm line 31.
Use of uninitialized value $quotable in substitution (s///) at t/lib/PPI/Test.pm line 32.
Use of uninitialized value $quotable in substitution (s///) at t/lib/PPI/Test.pm line 33.
Use of uninitialized value $quotable in substitution (s///) at t/lib/PPI/Test.pm line 34.
Use of uninitialized value $quotable in substitution (s///) at t/lib/PPI/Test.pm line 35.
Use of uninitialized value $round_tripped_quoted in concatenation (.) or string at t/21_exhaustive.t line 144.
# output: ""
# shorted failing substring: ""

#   Failed test 'round trip successful'
#   at t/21_exhaustive.t line 112.
# Looks like you failed 1 test of 1114.
t/21_exhaustive.t ................ 
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1114 subtests
eserte commented 5 years ago

Sample reports:

wchristian commented 5 years ago

All of those appeared to be related to unescaped parens sneaking into regexes. I've improved the tooling around those those tests, fixed the code, and added your examples as regression tests. Released as https://metacpan.org/release/MITHALDU/PPI-1.266

Thanks again. :)