Tests have started to fail with recent versions of Test2-Suite

[andk]

Sample fail report: http://www.cpantesters.org/cpan/report/88742816-c420-11ed-8c5a-962118a2e227

Statistical analysis of the results of my local smokers:

****************************************************************
Regression 'mod:Test2::V0'
****************************************************************
Name                   Theta          StdErr     T-stat
[0='const']           1.0000          0.0000    11402894181785682.00
[1='eq_0.000145']            -0.0000          0.0000      -6.29
[2='eq_0.000147']            -1.0000          0.0000    -8063063801093288.00
[3='eq_0.000148']            -1.0000          0.0000    -9875196038092174.00

R^2= 1.000, N= 93, K= 4
****************************************************************

suggests that Test2::V0 0.000147 is probably a blame spot. @exodist, you may be interested!

[exodist]

Hmm, going to need more info on how to reproduce the failure. I just installed the latest Test2::Suite, then Net::Google::Drive::Simple and it installed fine.

exodist@abydos main $ cpanm Test2::Suite                                                                                                                                                              [/tmp/S0BWS7LNT3]
--> Working on Test2::Suite
Fetching http://www.cpan.org/authors/id/E/EX/EXODIST/Test2-Suite-0.000148.tar.gz ... OK
Configuring Test2-Suite-0.000148 ... OK
Building and testing Test2-Suite-0.000148 ... OK
Successfully installed Test2-Suite-0.000148 (upgraded from 0.000145)
1 distribution installed

--> Working on Net::Google::Drive::Simple
Fetching http://www.cpan.org/authors/id/A/AT/ATOOMIC/Net-Google-Drive-Simple-3.02.tar.gz ... OK
Configuring Net-Google-Drive-Simple-3.02 ... OK
==> Found dependencies: JSON, Log::Log4perl, Sysadm::Install, File::MMagic, OAuth::Cmdline, Test::MockModule
[...]
Building and testing Net-Google-Drive-Simple-3.02 ... OK
Successfully installed Net-Google-Drive-Simple-3.02
10 distributions installed

The version you mention as the cause added 2 new modules and a couple of exports in Test2::V0. That does not mean it is impossible that it caused the problem, but it does seem unlikely as these tests do not appear to have any conflicts with the new exports (refcount*), and they do use the completely new modules. Nothing else significant changed.

I am available and happy to help track this down, specially if it does turn out to be a Test2::Suite problem, but at the moment I cannot reproduce the issue and need more/reliable reproduction steps.

[andk]

Thanks, @exodist, for the quick reply; meanwhile I have produced first passes with Test2-Suite-0.000148; the statistics now have no indication anymore what might be to blame; sorry for the noise, we will have to dig deeper.

[andk]

@exodist: can you answer the question what needs to be done to have the following line appear in the tests:

# Seeded srand with seed '20230319' from local date.

? If you can, you may have the answer how to reproduce the fail. My smokers have meanwhile produced 52 fail reports that you should soon all find via the matrix.cpantesters.org. All of the 52 fail reports produced two such "Seeded..." lines. None of the pass reports has such a line. Maybe this is a key to understanding the problem at hand.

[andk]

@exodist: I just discovered that all the perls that generated a fail had Devel::MAT::Dumper installed.

[andk]

@exodist: between the following two reports I installed Devel::MAT::Dumper; first is a pass, second is a fail:

• http://www.cpantesters.org/cpan/report/3b6608a8-c6f2-11ed-99a7-8f09ee7f97a0
• http://www.cpantesters.org/cpan/report/68d9b220-c6f3-11ed-9dc5-fa0eee7f97a0

[exodist]

ok, the problem seems ot be that if that module is installed and we load it, it produces warnings:

not ok 1 - Unexpected warning: Name "Devel::MAT::Dumper::HELPER_PER_MAGIC" used only once: possible typo at /home/exodist/perl5/perlbrew/perls/main/lib/5.32.1/XSLoader.pm line 111.
not ok 2 - Unexpected warning: Name "Devel::MAT::Dumper::HELPER_PER_PACKAGE" used only once: possible typo at /home/exodist/perl5/perlbrew/perls/main/lib/5.32.1/XSLoader.pm line 111.

[exodist]

This is something that has to be fixed in Devel::MAT::Dumper. As far as I can tell there is no way to hide or suppress these warnings from inside Test2::Suite.

I tried this:

 29 use constant HAVE_DEVEL_MAT_DUMPER => defined eval {
 30     local $SIG{__WARN__} = sub {
 31         return if $_[0] =~ m/Devel::MAT::Dumper::HELPER_PER_(MAGIC|PACKAGE) used only once/;
 32         warn @_;
 33     };
 34     require Devel::MAT::Dumper;
 35 };

But I still get the warnings. no warnings once is also ineffective here as the scope is wrong for that to work.

The only way to "fix" this in Test2::Suite would be to remove the feature, but it is already depended on.

[andk]

@exodist: thanks for getting back to this. I've forwarded this to the issue tracker of Devel::MAT::Dumper at https://rt.cpan.org/Ticket/Display.html?id=147233

[leonerd]

Those warnings shouldn't appear unless someone is globally enabling warnings (ie. the -w flag). Is that the case here?

Compare:

leo@shy:~/src/perl/Devel-MAT-Dumper [bzr]
$ perl -MDevel::MAT::Dumper -e1

leo@shy:~/src/perl/Devel-MAT-Dumper [bzr]
$ perl -w -MDevel::MAT::Dumper -e1
Name "Devel::MAT::Dumper::HELPER_PER_PACKAGE" used only once: possible typo at /usr/lib/x86_64-linux-gnu/perl-base/XSLoader.pm line 112.
Name "Devel::MAT::Dumper::HELPER_PER_MAGIC" used only once: possible typo at /usr/lib/x86_64-linux-gnu/perl-base/XSLoader.pm line 112.

[leonerd]

In any case, a simple workaround fix for now is to simply

our %Devel::MAT::Dumper::HELPER_PER_PACKAGE; ...

somewhere in your loading code. But a better fix would be to observe that those variables are all old legacy support for older modules and probably none of those modules exist any more so I could just delete all the associated code anyway.

But also you should find out where your -w argument comes from and remove it. :)

[exodist]

I did what leonerd suggested and released Test2-Suite version v0.000149 to fix the warnings.