Test-* distributions broken by Test::Builder 1.5

[schwernbot]

From: @schwern Date: Thursday Apr 11, 2013 at 00:58 GMT Orig: https://github.com/Test-More/test-more/issues/352

As of 63606c6f28e15d48a801a9572576fef0beea46e8 this is the list of Test-* distributions whose tests have been broken by Test::Builder 1.5. 104 out of 722 distributions are broken.

A test is "broken" if it passes with stable and fails with the latest Test::Builder 1.5.

They mostly fall into these categories...

• Failures in the test suite due to small changes in the TAP format
• Using private methods of Test::Builder which no longer exist
• Using Test::Builder->details() which is no longer populated by default
• Doing something complicated to the Test::Builder internals

There's a wiki page with more information.

To help...

• Download the module from CPAN
• Run it with Test::More 0.98 to verify it works
• Run it with Test::More from the Test-Builder1.5 branch

Running against the Test-Builder1.5 branch...

• clone the repository
• git checkout Test-Builder1.5
• export PERL5LIB=~/devel/Test-Simple/lib/
• Test a module

At that point you can...

• Patch the problem
• Send the patch to the module
• Open an issue stating you've sent on a patch
• Indicate whether your patch was only to the test suite or required fixing runtime code
• If the runtime had to be patched (as opposed to just the tests) mark it as a conflict.

Or if it works now...

• Open an issue and let us know the distribution and version which works.

Or if the problem is over your head...

• Open an issue with the distribution, version and test failure output.

Acme::Test::Weather
Apache::Test
Bundle::Test::SQL::Statement
CatalystX::Test::Recorder
Kwiki::Test
PITA::Test::Dummy::Perl5::Deps
Plack::Test::ExternalServer
Socialtext::WikiTest
Statistics::Multtest
Statistics::Test::WilcoxonRankSum
Test::A8N
Test::Able
Test::Aggregate
Test::Apache::RewriteRules
Test::AutoBuild
Test::AutomationFramework
Test::BDD::Cucumber
Test::Base
Test::Builder::Clutch
Test::Chimps
Test::Chimps::Anna
Test::Chimps::Client
Test::Class
Test::Class::Filter::Tags
Test::Class::Moose
Test::Class::Most
Test::Class::Sugar
Test::ClassAPI
Test::Classy
Test::Compile
Test::Continuous
Test::DBIx::Class::Schema
Test::Daily
Test::Deep::Teng::Row
Test::Dependencies
Test::DependentModules
Test::Depends
Test::Differences
Test::Dist
Test::Dist::VersionSync
Test::Effects
Test::Exports
Test::FITesque
Test::FTP::Server
Test::Fake::HTTPD
Test::Fixture::DBI
Test::Flatten
Test::GreaterVersion
Test::Groonga
Test::HTML::Spelling
Test::Harness::Straps
Test::HasVersion
Test::Httpd::Apache2
Test::ISBN
Test::Kit
Test::LWP::UserAgent
Test::LatestPrereqs
Test::Legacy
Test::Memory::Usage
Test::Mimic
Test::Mimic::Generator
Test::Mock::LWP::Conditional
Test::MockClass
Test::MonitorSites
Test::More::Fork
Test::More::Strict
Test::Most
Test::NoXS
Test::Parallel
Test::ParallelSubtest
Test::Pcuke
Test::Perl::Dist
Test::Perl::Metrics::Simple
Test::Pockito
Test::PostgreSQL
Test::Pretty
Test::Random
Test::RedisServer
Test::Reporter::Transport::Outlook
Test::SFTP
Test::SMTP
Test::STF::MockServer
Test::SharedFork
Test::Sims
Test::Smoke::Database
Test::Story
Test::TAPv13
Test::TCP
Test::TCP::Multi
Test::Trivial
Test::Type
Test::Unit
Test::WWW::Mechanize::Maypole
Test::WWW::Simple
Test::WWW::WebKit
Test::Warnings
Test::Wrapper
Test::XML::RPC::Catalyst
Test::XML::Simple
Test::mysqld
Test::postgresql
Test::t
Wetware::Test
Wetware::Test::CreateTestSuite

[schwernbot]

From: @schwern Date: Friday Apr 12, 2013 at 09:30 GMT Orig: https://github.com/Test-More/test-more/issues/352#issuecomment-16283895

While at the QA Hackathon we'll be coordinating using this Etherpad. http://openetherpad.org/8IrZW54mHd

[schwernbot]

From: @karenetheridge Date: Friday Apr 12, 2013 at 22:04 GMT Orig: https://github.com/Test-More/test-more/issues/352#issuecomment-16320117

I'm guessing you tried to install all Test-* distributions in CPAN? That's going to miss any Test::* modules in other distributions... for example I just discovered that Test::Moose fails (due to its use of ->details). I've added that to the etherpad list (and removed a false negative, Test::LWP::UserAgent, that isn't a test module on its own but fails due to its use of Test::TempDir).

i.e. dist/test_dists should test not just Test-* distributions, but also do something like what Test::DependentModules does, and test all reverse-dependencies of Test::Builder.

[schwernbot]

From: @karenetheridge Date: Tuesday Apr 16, 2013 at 19:36 GMT Orig: https://github.com/Test-More/test-more/issues/352#issuecomment-16466841

https://metacpan.org/requires/distribution/Test-Tester?sort=[[2,1]] is a good list of other distributions that ought to be tested.

[schwernbot]

From: @schwern Date: Wednesday Apr 17, 2013 at 11:26 GMT Orig: https://github.com/Test-More/test-more/issues/352#issuecomment-16500226

@karenetheridge For now we're concentrating on the Test distributions to firm up the trunk of the Test::Builder dependency tree before moving up. Many distributions fail because important Test distributions aren't working. The results are very polluted and take up time just to find out Plack isn't working because Test::TCP isn't working because Test::SharedFork isn't working which we already knew. Test::Most, Test::Class and Test::SharedFork not working knocks out a big chunk of the tree.

So keep ideas for testing the whole dependency tree on hold for now, unless you can come up with a clever way of determining where the error in a test failure lies which would be super awesome!

Checking Test-Tester deps and also Test::Builder::Tester is a good idea at this stage.

[schwernbot]

From: @karenetheridge Date: Wednesday Apr 17, 2013 at 17:50 GMT Orig: https://github.com/Test-More/test-more/issues/352#issuecomment-16521512

That's quite reasonable. My concern is mainly that there are some Test::* modules in other distributions, that might also be breaking that we don't know about. (I noticed this while trying to install Test::TempDir, which uses Moose, and Moose contains Test::Moose, which was broken.)