Roll back the default formatter to match Test::Builder 0.9x?

[schwernbot]

From: @schwern Date: Tuesday Apr 09, 2013 at 22:43 GMT Orig: https://github.com/Test-More/test-more/issues/351

One of the bigger issues blocking release is that about 100 of the 1000 Test:: modules on CPAN are broken with Test::Builder. Most of these are due to minor formatting changes in the TAP output in modules that don't use Test::Builder::Tester or Test::Tester (and some that do).

The new TAP v13 formatter has a few minor changes compared to the existing formatter. The one which has the largest effect is a TAP version is output before the plan. This confuses a lot of Test::* tests which expect the plan to be the first thing.

@chromatic suggested a fix is to make the default formatter match what Test::Builder currently outputs and worry about introducing formatting changes later. Fortunately we already have a formatter which simulates the quirks of the existing Test::Builder.

Here's the choices I'd like to hear people's opinions on.

1. Keep the new formatter, fix the Test::* tests.
2. Make the TB1 formatter default, leave the Test::* tests alone.

To be clear none of the formatter changes break Test:: modules at runtime. That is, if you have Test::Foo 1.23 installed right now it will still work with the new formatter no matter what we do right now. This is *only about Test::'s own tests*. There will be no need for a user to update their install of Test::Foo for Test::Builder 1.5 because of the formatter change.

The work involved in the first is fixing about 50-100 Test module tests. This can either be a straight hack or converting them to Test::Builder::Tester or Test::Tester. Most of this is fairly straight forward for somebody who understands Perl and tests, it's just a lot of work and pushing back to authors and getting releases made. I can be done at the Perl QA Hackathon.

The work involved in the second is fixing Test::Builder 1.5's own tests. Unfortunately, I wasn't very good at developing and using a formatted agnostic self-test suite and suffer from exactly the same problem as the Test::* modules, just in the other direction. I'd guestimate the patching will be more difficult than patching the Test::* modules, but that's only a guess.

Opinions?

PS This effects the upcoming Perl QA Hackathon and I need to make a decision on this before Friday.

[schwernbot]

From: @karenetheridge Date: Wednesday Apr 10, 2013 at 03:36 GMT Orig: https://github.com/Test-More/test-more/issues/351#issuecomment-16153619

Which Test::* modules are broken? And next, how many other modules in the CPAN use these tests? (Without presuming you have the answers now -- in which case the question becomes "how can we find this out?")

[schwernbot]

From: @schwern Date: Wednesday Apr 10, 2013 at 05:51 GMT Orig: https://github.com/Test-More/test-more/issues/351#issuecomment-16156769

Good question!

I have the means to find that out and I'm running it now. I'll post it once it's done.

There's a program in dist/test_dists which runs the tests of every /^Test-/ dist against both your installed Test::More and against one installed in /tmp/tb2/lib/perl5 for comparison. If you'd like to run it yourself...

git checkout Test-Builder1.5
perl Makefile.PL INSTALL_BASE=/tmp/tb2
make test
make install
perl dist/test_dists

It takes some time, there are 722 Test- dists. I have an old list kicking around that tested 388 of them. These are just distributions which fail, I don't know why.

[schwernbot]

From: @schwern Date: Thursday Apr 11, 2013 at 00:59 GMT Orig: https://github.com/Test-More/test-more/issues/351#issuecomment-16211619

I've run nearly a full scan and made the list available as #352. There are at least 104 failing distributions out of 722.

[schwernbot]

From: @schwern Date: Friday Apr 12, 2013 at 12:03 GMT Orig: https://github.com/Test-More/test-more/issues/351#issuecomment-16289236

I'm running the test comparison with the old formatter to see how much better that works. We have the handy TB2_FORMATTER_CLASS environment variable to control the default formatter.

TB2_FORMATTER_CLASS=TB2::Formatter::TAP::TB1 perl dist/test_dists --mirror file:///var/local/CPAN_mirror --lib=/tmp/TB2/lib/perl5

[schwernbot]

From: @schwern Date: Friday Apr 12, 2013 at 14:28 GMT Orig: https://github.com/Test-More/test-more/issues/351#issuecomment-16295848

I ran 215 tests, shuffled, before the test script blew out. 44 failed which gives a 20% failure rate. This is worse than the 13-14% failure rate with the new formatter

Turns out this is a bug in TB2::CanLoad. See #356

[schwernbot]

From: @colinnewell Date: Saturday Apr 13, 2013 at 15:12 GMT Orig: https://github.com/Test-More/test-more/issues/351#issuecomment-16334761

After analysing a sample of those failures as part of the work for issue 352 it doesn't appear that there are many distributions that are broken solely by TAP format issues. I'd suggest we don't gain much by rolling back.

[schwernbot]

From: @schwern Date: Sunday Apr 14, 2013 at 13:51 GMT Orig: https://github.com/Test-More/test-more/issues/351#issuecomment-16351301

Yeah, the event storage change seems to be a much bigger problem. Maybe we should talk about changing that default.

I'm going to close this up.