Test-More / test-more

Test2, Test::More, Test::Simple and Test::Builder Perl modules for writing tests
Other
140 stars 89 forks source link

100 candidates having bad experiences with 1.005000_002 #249

Closed andk closed 10 years ago

andk commented 12 years ago

The following list is compiled from results of the regression testing done on analysis.cpantesters.org. In column 2 there is the R^2 denoting the quality of the finding. The closer it is to 1 the less disturbing influences have been identified, and the more likely it is that the measured effect is relevant. In column 3 there is the Theta, the direction and amplitude of the influence. I've filtered column three for values < 0.75 and column 2 to values > 0.35. Arbitrary decisions. 103 lines seemed more than enough to me and indeed the last items in the list are already lots of influences mixed in.

Hope this is interesting to you.

XML-LibXML-SAX-ChunkParser-0.00004            1.000        -1.0000
URI-Escape-XS-0.08                            1.000        -1.0000
UNIVERSAL-DOES-0.004                          1.000        -1.0000
Test-Script-Shebang-0.02                      1.000        -1.0000
Test-More-Prefix-0.001                        1.000        -1.0000
Test-Group-0.19_01                            1.000        -1.0000
Test-Flatten-0.07                             1.000        -1.0000
Test-File-1.28_01                             1.000        -1.0000
Test-Classy-0.09                              1.000        -1.0000
Test-Able-0.11                                1.000        -1.0000
RedisDB-0.24                                  1.000        -1.0000
Proc-Lite-0.09                                1.000        -1.0000
Proc-Lite-0.09_01                             1.000        -1.0000
Proc-Guard-0.06                               1.000        -1.0000
Plack-App-Proxy-Selective-0.10                1.000        -1.0000
Net-Google-DataAPI-0.25                       1.000        -1.0000
Log-Sigil-0.15                                1.000        -1.0000
Log-Dispatch-Config-TestLog-0.02              1.000        -1.0000
Locale-Maketext-1.21                          1.000        -1.0000
IO-Prompt-Simple-0.04                         1.000        -1.0000
HTML-LinkFilter-0.03                          1.000        -1.0000
HTML-Barcode-Code93-0.10                      1.000        -1.0000
Goose-0.004                                   1.000        -1.0000
Game-TextPacMonster-0.03                      1.000        -1.0000
DBM-Deep-2.0004                               1.000        -1.0000
DBIx-Simple-Inject-0.03                       1.000        -1.0000
DBI-Easy-0.23                                 1.000        -1.0000
Date-Holidays-PAK-0.06                        1.000        -1.0000
Data-Monad-0.02                               1.000        -1.0000
CGI-Emulate-PSGI-0.14                         1.000        -1.0000
Carp-1.23                                     1.000        -1.0000
App-mgen-0.16                                 1.000        -1.0000
AnyEvent-Twitter-0.62                         1.000        -1.0000
AnyEvent-Groonga-0.07                         1.000        -1.0000
Algorithm-Kmeanspp-0.03                       1.000        -1.0000
Algorithm-KernelKMeans-0.03_02                1.000        -1.0000
Algorithm-FuzzyCmeans-0.02                    1.000        -1.0000
POE-Component-ResourcePool-0.04               0.969        -1.0000
Test-SFTP-1.10                                0.957        -1.0000
Test-Dependencies-0.12                        0.942        -1.0000
Aspect-1.01                                   0.873        -1.0000
IO-BindHandles-0.005                          0.872        -1.0000
Module-Release-2.06_01                        0.870        -0.9946
Role-Basic-0.12                               0.865        -1.0000
WebService-8tracks-0.01                       0.854        -1.0000
Teng-0.14_04                                  0.853        -0.9953
ShipIt-Step-ChangeAllVersions-0.007           0.853        -1.0000
Data-ModeMerge-0.27                           0.810        -1.0000
WWW-Search-MSN-0.0106                         0.789        -1.0000
DBIx-CouchLike-0.16                           0.781        -1.0000
Data-YUID-0.06                                0.771        -1.0000
POEx-Role-SessionInstantiation-1.102610       0.759        -1.0000
Teng-0.14_03                                  0.747        -1.0000
Devel-Cover-Report-Clover-0.31                0.742        -1.0000
DBIx-Class-0.08196                            0.742        -1.0000
Bison-0.05                                    0.729        -1.0000
Makefile-DOM-0.006                            0.723        -1.0000
Backed_Objects-1.0                            0.713        -1.0000
CGI-Session-ID-crypt_openssl-1.01             0.711        -1.0000
Barcode-DataMatrix-0.03                       0.693        -1.0000
HTML-GUI-0.04                                 0.680        -0.9545
Moose-2.0401                                  0.665        -1.0000
XSLoader-0.15                                 0.664        -1.0000
Parallel-ForkControl-0.5                      0.663        -1.0000
Test-Dist-0.01                                0.641        -1.0000
KiokuDB-Backend-Files-0.06                    0.630        -1.0000
Plack-0.9985                                  0.621        -1.0000
Furl-0.38                                     0.610        -1.0000
Module-Build-Bundle-0.07                      0.600        -1.0000
Spawn-Safe-2.001                              0.596        -1.0000
MooX-Option-0.3                               0.593        -0.8889
Parse-HP-ACU-0.03                             0.579        -1.0000
DBIx-Skinny-Pager-0.11                        0.571        -1.0000
Alien-SVN-v1.6.12.1                           0.556        -1.0000
Parse-CSV-Colnames-0.04                       0.544        -1.0000
DBIx-Skinny-0.0742                            0.543        -1.0000
IPC-Cmd-0.72                                  0.528        -1.0000
Catalyst-Action-REST-0.94                     0.521        -0.9759
ASP4x-Captcha-Imager-0.003                    0.507        -1.0000
Ledger-Parser-0.03                            0.493        -0.8571
WiX3-0.011                                    0.455        -1.0000
signatures-0.06                               0.455        -1.0000
MediaWiki-Bot-v3.4.2                          0.448        -1.0000
WebService-FreeDB-0.78                        0.445        -1.0000
Pod-HTMLEmbed-0.04                            0.445        -1.0000
Test-Story-0.07                               0.440        -1.0000
Acme-Perl-VM-0.006                            0.437        -1.0000
Data-ObjectDriver-0.09                        0.431        -1.0000
Data-MessagePack-0.39                         0.426        -0.9653
Event-ExecFlow-0.64                           0.421        -1.0000
Log-Any-Adapter-Mojo-0.04                     0.418        -1.0000
Encode-JP-Mobile-0.27                         0.410        -1.0000
Net-API-Gett-0.01                             0.391        -0.7812
CPAN-Mini-Extract-1.21                        0.391        -1.0000
BSON-0.06                                     0.390        -1.0000
forks-0.34                                    0.386        -1.0000
Catalyst-View-JavaScript-0.995                0.382        -1.0000
App-Cpan-1.57                                 0.373        -1.0000
Mojolicious-Plugin-HamlRenderer-1.0           0.360        -1.0000
PLP-3.23                                      0.358        -1.0000
MooseX-Daemonize-0.13                         0.358        -1.0000
Ukigumo-0.01                                  0.356        -1.0000
IO-Async-Loop-AnyEvent-0.03                   0.356        -1.0000

Oh, and splitting into individual tickets is left as an exercise:)

andk commented 12 years ago

I've filtered column three for values < 0.75

s/0.75/-0.75/

schwern commented 12 years ago

Thanks! Uhh... wow! Yes, this is very interesting and useful.

I was hoping they'd mostly be the fault of broken Test::* dependencies. Nope... they all seem to be a bit different. Poop.

doherty commented 12 years ago

MediaWiki-Bot-v3.4.2 is almost certainly not failing because of TB2. It is almost certainly failing because I have bugs in my test suite and/or the live site we test against has gone funky again.

petdance commented 12 years ago

Suggestion/request: Rerun it with our CPAN IDs on it. Make it easy for authors to find their names and problems without having to scan the listing of 100.

shlomif commented 12 years ago

I agree with petdance - the CPAN IDs would be useful.

schwern commented 12 years ago

URI::Escape::XS appears to be a false positive. I can't get it to fail on 1.5.0a2.

schwern commented 12 years ago

UNIVERSAL::DOES has been fixed by 78f0c977ca3b79a7a33592557b77b4f14ca61f32