Test plans not handled properly

[zaucker]

Test::Pretty doesn't seem to handle test plans as shown in the following examples:

Example 1

The following script erroneously reports a mismatch of tests planned and tests run.

#!/usr/bin/perl -w

use strict;
use warnings;
use Test::More tests => 3;
use Test::Pretty;

### Test::Pretty doesn't handle test plans properly.
###
### This file with Test::Pretty gives
###
###   1..3
###   o  Plain test
###     Subtest 1
###       o  Test 1-1
###       o  Test 1-2
###     Subtest 2
###       o  Test 2-1
###       o  Test 2-2
###     # Bad plan: 5 != 7
###
### while running this test without Test::Pretty shows
### that all plans planned are run:
###
###    1..3
###    ok 1 - Plain test
###        1..2
###        ok 1 - Test 1-1
###        ok 2 - Test 1-2
###    ok 2 - Subtest 1
###        1..2
###        ok 1 - Test 2-1
###        ok 2 - Test 2-2
###    ok 3 - Subtest 2

ok(1, "Plain test");
subtest "Subtest 1" => sub {
    plan tests => 2;

    ok(1, "Test 1-1");
    ok(1, "Test 1-2");
};

subtest "Subtest 2" => sub {
    plan tests => 2;

    ok(1, "Test 2-1");
    ok(1, "Test 2-2");
};
exit;

Example 2

The following script erroneously reports a mismatch of tests planned and tests run and in addition does not complain about the additional test run in subtest 1.

#!/usr/bin/perl -w

use strict;
use warnings;
use Test::More tests => 3;
use Test::Pretty;

### Test::Pretty doesn't handle test plans properly.
###
### This file with Test::Pretty gives
###
###   1..3
###   o  Plain test
###     Subtest 1
###       o  Test 1-1
###       o  Test 1-2
###     Subtest 2
###       o  Test 2-1
###       o  Test 2-2
###     # Bad plan: 5 != 6
###
### while running this test without Test::Pretty shows
### that all plans planned are run:
###
###    1..3
###    ok 1 - Plain test
###        1..1
###        ok 1 - Test 1-1
###        ok 2 - Test 1-2
###        # Looks like you planned 1 test but ran 2.
###    not ok 2 - Subtest 1
###    #   Failed test 'Subtest 1'
###    #   at ./Pretty2.t line 46.
###        1..2
###        ok 1 - Test 2-1
###        ok 2 - Test 2-2
###    ok 3 - Subtest 2
###    # Looks like you failed 1 test of 3.

ok(1, "Plain test");
subtest "Subtest 1" => sub {
    plan tests => 1;

    ok(1, "Test 1-1");
    ok(1, "Test 1-2");
};

subtest "Subtest 2" => sub {
    plan tests => 2;

    ok(1, "Test 2-1");
    ok(1, "Test 2-2");
};
exit;

[tokuhirom]

Could you send me a patch?

tokuhirom

On Thu, Jun 26, 2014 at 4:00 PM, Fritz Zaucker notifications@github.com wrote:

Test::Pretty doesn't seem to handle test plans as shown in the following examples: Example 1

The following script erroneously reports a mismatch of tests planned and tests run.

!/usr/bin/perl -w

use strict;use warnings;use Test::More tests => 3;use Test::Pretty;

Test::Pretty doesn't handle test plans properly.###### This file with Test::Pretty gives###### 1..3### o Plain test### Subtest 1### o Test 1-1### o Test 1-2### Subtest 2### o Test 2-1### o Test 2-2### # Bad plan: 5 != 7###### while running this test without Test::Pretty shows### that all plans planned are run:###### 1..3### ok 1 - Plain test### 1..2### ok 1 - Test 1-1### ok 2 - Test 1-2### ok 2 - Subtest 1### 1..2### ok 1 - Test 2-1### ok 2 - Test 2-2### ok 3 - Subtest 2

ok(1, "Plain test");subtest "Subtest 1" => sub { plan tests => 2;

ok(1, "Test 1-1");
ok(1, "Test 1-2");};

subtest "Subtest 2" => sub { plan tests => 2;

ok(1, "Test 2-1");
ok(1, "Test 2-2");};exit;

Example 2

The following script erroneously reports a mismatch of tests planned and tests run and in addition does not complain about the additional test run in subtest 1.

!/usr/bin/perl -w

use strict;use warnings;use Test::More tests => 3;use Test::Pretty;

Test::Pretty doesn't handle test plans properly.###### This file with Test::Pretty gives###### 1..3### o Plain test### Subtest 1### o Test 1-1### o Test 1-2### Subtest 2### o Test 2-1### o Test 2-2### # Bad plan: 5 != 6###### while running this test without Test::Pretty shows### that all plans planned are run:###### 1..3### ok 1 - Plain test### 1..1### ok 1 - Test 1-1### ok 2 - Test 1-2### # Looks like you planned 1 test but ran 2.### not ok 2 - Subtest 1### # Failed test 'Subtest 1'### # at ./Pretty2.t line 46.### 1..2### ok 1 - Test 2-1### ok 2 - Test 2-2### ok 3 - Subtest 2### # Looks like you failed 1 test of 3.

ok(1, "Plain test");subtest "Subtest 1" => sub { plan tests => 1;

ok(1, "Test 1-1");
ok(1, "Test 1-2");};

subtest "Subtest 2" => sub { plan tests => 2;

ok(1, "Test 2-1");
ok(1, "Test 2-2");};exit;

— Reply to this email directly or view it on GitHub https://github.com/tokuhirom/Test-Pretty/issues/21.

[zaucker]

Unfortunately, I have no idea what the cause of the problem is. Looking at Pretty.pm I'd expect it to have something to do with __plan_tests() or _expected_tests(), but this is your code, so I'd have to re-engineer what you are doing to fix it ...

Cheers, Fritz

On Thu, 26 Jun 2014, Tokuhiro Matsuno wrote:

Could you send me a patch?

tokuhirom

On Thu, Jun 26, 2014 at 4:00 PM, Fritz Zaucker notifications@github.com wrote:

Test::Pretty doesn't seem to handle test plans as shown in the following examples: Example 1

The following script erroneously reports a mismatch of tests planned and tests run.

!/usr/bin/perl -w

use strict;use warnings;use Test::More tests => 3;use Test::Pretty;

Test::Pretty doesn't handle test plans properly.###### This file with Test::Pretty gives###### 1..3### o Plain test### Subtest 1### o Test 1-1### o Test 1-2### Subtest 2### o Test 2-1### o Test 2-2### # Bad plan: 5 != 7###### while running this test without Test::Pretty shows### that all plans planned are run:###### 1..3### ok 1 - Plain test### 1..2### ok 1 - Test 1-1### ok 2 - Test 1-2### ok 2 - Subtest 1### 1..2### ok 1 - Test 2-1### ok 2 - Test 2-2### ok 3 - Subtest 2

ok(1, "Plain test");subtest "Subtest 1" => sub { plan tests => 2;

ok(1, "Test 1-1");
ok(1, "Test 1-2");};

subtest "Subtest 2" => sub { plan tests => 2;

ok(1, "Test 2-1");
ok(1, "Test 2-2");};exit;

Example 2

The following script erroneously reports a mismatch of tests planned and tests run and in addition does not complain about the additional test run in subtest 1.

!/usr/bin/perl -w

use strict;use warnings;use Test::More tests => 3;use Test::Pretty;

Test::Pretty doesn't handle test plans properly.###### This file with Test::Pretty gives###### 1..3### o Plain test### Subtest 1### o Test 1-1### o Test 1-2### Subtest 2### o Test 2-1### o Test 2-2### # Bad plan: 5 != 6###### while running this test without Test::Pretty shows### that all plans planned are run:###### 1..3### ok 1 - Plain test### 1..1### ok 1 - Test 1-1### ok 2 - Test 1-2### # Looks like you planned 1 test but ran 2.### not ok 2 - Subtest 1### # Failed test 'Subtest 1'### # at ./Pretty2.t line 46.### 1..2### ok 1 - Test 2-1### ok 2 - Test 2-2### ok 3 - Subtest 2### # Looks like you failed 1 test of 3.

ok(1, "Plain test");subtest "Subtest 1" => sub { plan tests => 1;

ok(1, "Test 1-1");
ok(1, "Test 1-2");};

subtest "Subtest 2" => sub { plan tests => 2;

ok(1, "Test 2-1");
ok(1, "Test 2-2");};exit;

? Reply to this email directly or view it on GitHub https://github.com/tokuhirom/Test-Pretty/issues/21.

---

Reply to this email directly or view it on GitHub: https://github.com/tokuhirom/Test-Pretty/issues/21#issuecomment-47194865

Oetiker+Partner AG tel: +41 62 775 9903 (direct) Fritz Zaucker +41 62 775 9900 (switch board) Aarweg 15 +41 79 675 0630 (mobile) CH-4600 Olten fax: +41 62 775 9905 Schweiz web: www.oetiker.ch

[preaction]

Looks like I found another case for this:

use strict;
use warnings;
use Test::More;

subtest 'foo' => sub {
    subtest 'foobar' => sub {
        pass "a";
    };
    subtest 'foobaz' => sub {
        plan skip_all => "b is not finished";
        fail "b";
    };
    subtest 'foobuzz' => sub {
        pass "c";
    };
    pass "d";
};
pass "e";
done_testing;

When skip_all is encountered, the tests simply stop completely. No further tests appear to be run.

I'm going to look into this once I get this current thing off my plate.

[preaction]

Looks like in fixing this, I'd probably have to copy and paste a bunch of code from Test::Builder, but I'm not sure I like that solution very well.

I like where you replaced Test::Builder methods with your own methods that eventually call the Test::Builder methods (https://github.com/tokuhirom/Test-Pretty/blob/master/lib/Test/Pretty.pm#L94), but that only happens in a certain case. I can't figure out why there are two cases. Could you explain? https://github.com/tokuhirom/Test-Pretty/blob/master/lib/Test/Pretty.pm#L47

I think it would be better if the wrapping of Test::Builder methods happened all the time, in order to ensure that we get all the information we need to make things pretty, but also to ensure that Test::Builder works exactly as it should. Then we might be able to work with the Test::Builder project to help make it so we would not need to replace their methods, so that Test::Pretty does not break if Test::Builder's API changes.

I wrote a plugin that formats TAP into something that vim's quickfix window supports, but that required the use of Test::More::Diagnostic. It sounds like the same thing could be done here, so if Test::More::Diagnostic was integrated into Test::More, we could make pretty output in a safe and easy way.

[skaji]

I encountered the following issue:

# test.t
use strict;
use warnings;
use Test::More;

subtest skip1 => sub {
    plan skip_all => "skip1";
};
subtest skip2 => sub {
    plan skip_all => "skip2";
};

done_testing;

> prove -v -PPretty test.t; echo "exit status $?"

  skip1
1..0 # SKIP skip1
skip skip1
  skip2
1..0 # SKIP skip2
skip skip2
skipped: skip1

exit status 1

I expected exit status 0, but got exit status 1. I'm using Test::Pretty 0.30, Test::More 1.001009, perl 5.20.1 on OSX 10.10.1.