search

Test-More / TB2

Test::Builder version 2, the next generation of building testing modules in Perl
Other
1 stars 0 forks source link

Test::Class failure on Test-Builder2 #205

Closed schwernbot closed 10 years ago

schwernbot commented 10 years ago

From: @apeiron Date: Saturday Mar 31, 2012 at 11:17 GMT Orig: https://github.com/Test-More/test-more/issues/276

Building and testing Test-Class-0.36 ... Building Test-Class
t/00-load.t .................... 1/? # Testing Test::Class::Load 0.35, Perl 5.014002, /Users/apeiron/perl5/perlbrew/perls/testbuilder2/bin/perl
t/00-load.t .................... ok
t/20-load-classes.t ............ ok
t/21-load-subclassed.t ......... ok
t/Tests.t ...................... ok
t/_new.t ....................... ok
t/bad-autoloads.t .............. skipped: need Contextual::Return
t/bailout.t .................... ok
t/builder.t .................... ok
t/compile.t .................... ok
t/current_method.t ............. ok
t/diag_on_failure.t ............ 1/?
#   Failed test 'we show the test class and method name on test failure'
#   at t/diag_on_failure.t line 24.
# STDOUT is:
#1..2
# not ok 1 - failing test
# ok 2 - passing test
#
# not:
# not ok 1 - failing test
#
# ok 2 - passing test
#
# as expected
#1 test of 1 failed.
t/diag_on_failure.t ............ Failed 1/1 subtests
t/die_before_plan.t ............ 1/?
#   Failed test 'die before plan'
#   at t/die_before_plan.t line 30.
# STDOUT is:
#1..1
# not ok 1 - setup (for test method 'test') died (died before plan set)
# ok 2 - test just here to get setup method run
#
# not:
# not ok 1 - setup (for test method 'test') died (died before plan set)
#
# ok 2 - test just here to get setup method run
#
# as expected
#1 test of 1 failed.
t/die_before_plan.t ............ Failed 1/1 subtests
t/expected_tests.t ............. ok
t/fail1.t ...................... 1/?
#   Failed test 'FAIL_ALL with plan'
#   at t/fail1.t line 18.
# STDOUT is:
#1..1
# not ok 1 - failing
#
# not:
# not ok 1 - failing
#
# not ok 2 - failing
#
# as expected
# STDERR is:
# #   Failed test 'failing'
# #   at t/fail1.t line 15.
#
# not:
# /#\s+Failed\ test.*?\n?.*?at\ t\/fail1\.t line 15.*\n?/
#
# /#\s+Failed\ test.*?\n?.*?at\ t\/fail1\.t line 15.*\n?/
#
# as expected

#   Failed test 'exit value okay'
#   at t/fail1.t line 19.
#          got: '1'
#     expected: '2'
#2 tests of 2 failed.
t/fail1.t ...................... Failed 2/2 subtests
t/fail2.t ...................... 1/?
#   Failed test 'fail2'
#   at t/fail2.t line 38.
# STDOUT is:
#1..3
# not ok 1 - The thing isa Object
# not ok 2 - cannot create Objects
# not ok 3 - cannot create Objects
#
# not:
# not ok 1 - The thing isa Object
#
# not ok 2 - cannot create Objects
#
# as expected
# STDERR is:
# #   Failed test 'The thing isa Object'
# #   at t/fail2.t line 18.
# #   (in Object::Test->_test_new)
# #     The thing isn't defined
# #   Failed test 'cannot create Objects'
# #   at t/fail2.t line 18.
# #   (in Object::Test->_test_new)
# #   Failed test 'cannot create Objects'
# #   at t/fail2.t line 18.
# #   (in Object::Test->_test_new)
#
# not:
# /#\s+Failed\ test.*?\n?.*?at\ t\/fail2\.t line 18.*\n?/
#
# #   (in Object::Test->_test_new)
#
# (?^:#\s+The thing isn't defined\n)
# /#\s+Failed\ test.*?\n?.*?at\ t\/fail2\.t line 18.*\n?/
#
# #   (in Object::Test->_test_new)
#
# as expected

#   Failed test 'exit value okay'
#   at t/fail2.t line 39.
#          got: '3'
#     expected: '2'
#2 tests of 2 failed.
t/fail2.t ...................... Failed 2/2 subtests
t/fail3.t ...................... All 2 subtests passed
t/filter.t ..................... ok
t/filter_bad_filter.t .......... ok
t/filter_fixtures.t ............ ok
t/filter_fixtures_only.t ....... ok
t/filter_multiple.t ............ ok
t/header.t ..................... 1/?
#   Failed test 'no duplicate headers'
#   at t/header.t line 21.
# STDOUT is:
#1..1
# ok 1 - test in Test::Class
#
# not:
# ok 1 - test in Test::Class
#
# as expected
#1 test of 1 failed.
t/header.t ..................... Failed 1/1 subtests
t/late_header.t ................ ok
t/methodinfo.t ................. ok
t/named_test.t ................. 1/?
#   Failed test 'test names set to method name by default'
#   at t/named_test.t line 20.
# STDOUT is:
#1..1
# ok 1 - default name
#
# not:
# ok 1 - default name
#
# as expected
#1 test of 1 failed.
t/named_test.t ................. Failed 1/1 subtests
t/num_method_tests.t ........... ok
t/num_tests.t .................. ok
t/override.t ................... ok
t/rt15870.t .................... ok
t/rt17264.t .................... ok
t/run_all_tests.t .............. ok
t/runtests.t ................... ok
t/runtests_die.t ............... 1/?
#   Failed test 'early die handled'
#   at t/runtests_die.t line 34.
# STDOUT is:
#1..2
# not ok 1 - The thing isa Object
# not ok 2 - test_object died (could not create object)
#
# not:
# not ok 1 - The thing isa Object
#
# not ok 2 - test_object died (could not create object)
#
# as expected
#1 test of 1 failed.
t/runtests_die.t ............... Failed 1/1 subtests
t/runtests_extra.t ............. 1/?
#   Failed test 'extra test detected'
#   at t/runtests_extra.t line 23.
# STDOUT is:
#1..1
# ok 1 - expected test
# ok 2 - extra test
#
# not:
# ok 1 - expected test
#
# ok 2 - extra test
#
# as expected
#1 test of 1 failed.
t/runtests_extra.t ............. Failed 1/1 subtests
t/runtests_noplan.t ............ ok
t/runtests_of.t ................ ok
t/runtests_result.t ............ 1/?
#   Failed test 'single failure ran okay'
#   at t/runtests_result.t line 42.
# STDOUT is:
#1..2
# not ok 1 - fails
# ok 2 - passes
#
# not:
# not ok 1 - fails
#
# ok 2 - passes
#
# as expected
#1 test of 4 failed.
t/runtests_result.t ............ Failed 1/4 subtests
t/runtests_return.t ............ 1/?
#   Failed test 'early return handled'
#   at t/runtests_return.t line 21.
# STDOUT is:
#1..1
# ok 1 # skip darwin only test
#
# not:
# ok 1 # skip darwin only test
#
# as expected
#1 test of 1 failed.
t/runtests_return.t ............ Failed 1/1 subtests
t/runtests_trailing.t .......... 1/?
#   Failed test 'trailing expection detected'
#   at t/runtests_trailing.t line 25.
# STDOUT is:
#1..1
# ok 1 - successful test
# not ok 2 - trailing_exception died (died)
#
# not:
# ok 1 - successful test
#
# not ok 2 - trailing_exception died (died)
#
# as expected
#1 test of 1 failed.
t/runtests_trailing.t .......... Failed 1/1 subtests
t/runtests_with_wrong_class.t .. ok
t/show_plan_in_shutdown.t ...... ok
t/skip1.t ...................... Failed 1/1 subtests
t/skip2.t ...................... 1/6 # Test 1 got: "TAP version 13" (t/skip2.t at line 37)
#   Expected: "1..4"
#  t/skip2.t line 37 is:                ok($actual, $expected);
# Test 2 got: "1..4" (t/skip2.t at line 37 fail #2)
#   Expected: "ok 1 - test"
# Test 3 got: "ok 1 - test" (t/skip2.t at line 37 fail #3)
#   Expected: "ok 2 # skip skippy"
# Test 4 got: "ok 2 # SKIP skippy" (t/skip2.t at line 37 fail #4)
#   Expected: "ok 3 # skip skippy"
# Test 5 got: "ok 3 # SKIP skippy" (t/skip2.t at line 37 fail #5)
#   Expected: "ok 4 # skip skippy"
Use of uninitialized value $expected in scalar chomp at t/skip2.t line 36, <DATA> line 5.
# Test 6 got: "ok 4 # SKIP skippy" (t/skip2.t at line 37 fail #6)
#   Expected: <UNDEF>
t/skip2.t ...................... Failed 6/6 subtests
t/skip_class_reason.t .......... 1/3 # Test 1 got: "TAP version 13" (t/skip_class_reason.t at line 45)
#   Expected: "1..1"
#  t/skip_class_reason.t line 45 is:            ok($actual, $expected);
# Test 2 got: "1..1" (t/skip_class_reason.t at line 45 fail #2)
#   Expected: "ok 1 # skip because SKIP_CLASS returned a string"
Use of uninitialized value $expected in scalar chomp at t/skip_class_reason.t line 44, <DATA> line 2.
# Test 3 got: "ok 1 # SKIP because SKIP_CLASS returned a string" (t/skip_class_reason.t at line 45 fail #3)
#   Expected: <UNDEF>
t/skip_class_reason.t .......... Failed 3/3 subtests
t/spaces.t ..................... 1/?
#   Failed test 'can have spaces around attributes'
#   at t/spaces.t line 20.
# STDOUT is:
#1..1
# ok 1 - it works
#
# not:
# ok 1 - it works
#
# as expected
#1 test of 1 failed.
t/spaces.t ..................... Failed 1/1 subtests
t/startup.t .................... ok
t/startup_that_dies.t .......... ok
t/test_classes.t ............... ok
t/test_deep.t .................. ok
t/test_method.t ................ 1/?
#   Failed test 'no TEST_METHOD runs all tests'
#   at t/test_method.t line 30.
# STDOUT is:
#1..11
# ok 1 - startup
# ok 2 - setup
# ok 3 - test1
# ok 4 - teardown
# ok 5 - setup
# ok 6 - test2
# ok 7 - teardown
# ok 8 - setup
# ok 9 - test3
# ok 10 - teardown
# ok 11 - shutdown
#
# not:
# ok 1 - startup
#
# ok 2 - setup
#
# ok 3 - test1
#
# ok 4 - teardown
#
# ok 5 - setup
#
# ok 6 - test2
#
# ok 7 - teardown
#
# ok 8 - setup
#
# ok 9 - test3
#
# ok 10 - teardown
#
# ok 11 - shutdown
#
# as expected

#   Failed test 'single match just runs one test'
#   at t/test_method.t line 34.
# STDOUT is:
#1..5
# ok 1 - startup
# ok 2 - setup
# ok 3 - test1
# ok 4 - teardown
# ok 5 - shutdown
#
# not:
# ok 1 - startup
#
# ok 2 - setup
#
# ok 3 - test1
#
# ok 4 - teardown
#
# ok 5 - shutdown
#
# as expected

#   Failed test 'two matches run both tests'
#   at t/test_method.t line 38.
# STDOUT is:
#1..8
# ok 1 - startup
# ok 2 - setup
# ok 3 - test1
# ok 4 - teardown
# ok 5 - setup
# ok 6 - test3
# ok 7 - teardown
# ok 8 - shutdown
#
# not:
# ok 1 - startup
#
# ok 2 - setup
#
# ok 3 - test1
#
# ok 4 - teardown
#
# ok 5 - setup
#
# ok 6 - test3
#
# ok 7 - teardown
#
# ok 8 - shutdown
#
# as expected
#3 tests of 4 failed.
t/test_method.t ................ Failed 3/4 subtests
t/test_verbose.t ............... 1/?
#   Failed test 'TEST_VERBOSE outputs method diagnostic'
#   at t/test_verbose.t line 24.
# STDOUT is:
#1..2
# ok 1 - test1
# ok 2 - test2
#
# not:
# ok 1 - test1
#
# ok 2 - test2
#
# as expected
#1 test of 1 failed.
t/test_verbose.t ............... Failed 1/1 subtests
t/todo.t ....................... # Test::More: 1.005000004
t/todo.t ....................... 1/?
#   Failed test 'todo tests work'
#   at t/todo.t line 43.
# STDOUT is:
#1..1
# not ok 1 - object live # TODO unimplemented
# #   Failed (TODO) test 'object live'
# #   at t/todo.t line 16.
# #   (in Foo->todo_test)
#
# not:
# not ok 1 - object live # TODO unimplemented
#
# /#\s+Failed\ \(TODO\)\ test.*?\n?.*?at\ t\/todo\.t line 16.*\n?/
#
# #   (in Foo->todo_test)
#
# as expected
#1 test of 2 failed.
t/todo.t ....................... Failed 1/2 subtests

Test Summary Report
-------------------
t/diag_on_failure.t          (Wstat: 0 Tests: 1 Failed: 1)
  Failed test:  1
t/die_before_plan.t          (Wstat: 0 Tests: 1 Failed: 1)
  Failed test:  1
t/fail1.t                    (Wstat: 0 Tests: 2 Failed: 2)
  Failed tests:  1-2
t/fail2.t                    (Wstat: 0 Tests: 2 Failed: 2)
  Failed tests:  1-2
t/fail3.t                    (Wstat: 0 Tests: 2 Failed: 0)
  Parse errors: More than one plan found in TAP output
t/header.t                   (Wstat: 0 Tests: 1 Failed: 1)
  Failed test:  1
  Parse errors: More than one plan found in TAP output
t/named_test.t               (Wstat: 0 Tests: 1 Failed: 1)
  Failed test:  1
  Parse errors: More than one plan found in TAP output
t/runtests_die.t             (Wstat: 0 Tests: 1 Failed: 1)
  Failed test:  1
t/runtests_extra.t           (Wstat: 0 Tests: 1 Failed: 1)
  Failed test:  1
t/runtests_result.t          (Wstat: 0 Tests: 4 Failed: 1)
  Failed test:  1
t/runtests_return.t          (Wstat: 0 Tests: 1 Failed: 1)
  Failed test:  1
t/runtests_trailing.t        (Wstat: 0 Tests: 1 Failed: 1)
  Failed test:  1
t/skip1.t                    (Wstat: 0 Tests: 1 Failed: 1)
  Failed test:  1
t/skip2.t                    (Wstat: 0 Tests: 7 Failed: 7)
  Failed tests:  1-7
  Parse errors: Bad plan.  You planned 6 tests but ran 7.
t/skip_class_reason.t        (Wstat: 0 Tests: 4 Failed: 4)
  Failed tests:  1-4
  Parse errors: Bad plan.  You planned 3 tests but ran 4.
t/spaces.t                   (Wstat: 0 Tests: 1 Failed: 1)
  Failed test:  1
  Parse errors: More than one plan found in TAP output
t/test_method.t              (Wstat: 0 Tests: 4 Failed: 3)
  Failed tests:  2-4
t/test_verbose.t             (Wstat: 0 Tests: 1 Failed: 1)
  Failed test:  1
t/todo.t                     (Wstat: 0 Tests: 2 Failed: 1)
  Failed test:  1
Files=52, Tests=181,  6 wallclock secs ( 0.22 usr  0.12 sys +  5.76 cusr  0.43 csys =  6.53 CPU)
Result: FAIL
Failed 19/52 test programs. 31/181 subtests failed.
FAIL
Testing Test-Class-0.36 failed.
schwernbot commented 10 years ago

From: @schwern Date: Friday Apr 13, 2012 at 03:52 GMT Orig: https://github.com/Test-More/test-more/issues/276#issuecomment-5108041

prove -br t works far better failing only t/skip1.t and t/test_method.t. This is very strange. It's acting like Test::Builder::Tester can't see the HARNESS_ACTIVE environment variable or something.

It's something to do with the fact that the scripts have taint mode set.

schwernbot commented 10 years ago

From: @schwern Date: Saturday Aug 04, 2012 at 23:44 GMT Orig: https://github.com/Test-More/test-more/issues/276#issuecomment-7506415

A combination of better emulating TBT 0.9x quirks 072ee05ddd6eae941f693f11ee9c099709e2fdf7 and 00d4609f79fef0ad0db954ba9d9982026eb755e0 and small patches to Test::Class as well as it's tests fixes this.

Pull request sent upstream. https://github.com/adrianh/test-class/pull/2

schwernbot commented 10 years ago

From: @schwern Date: Sunday Apr 14, 2013 at 14:47 GMT Orig: https://github.com/Test-More/test-more/issues/276#issuecomment-16352138

Test::Class 0.39 fixed this, yay!