search

tape-testing / tape

tap-producing test harness for node and browsers
MIT License
5.77k stars 307 forks source link

Failure due to `plan != count` causes all other tests to fail as 'exited without ending' #550

Open SH4R1K7 opened 3 years ago

SH4R1K7 commented 3 years ago

Howdy, this is a peculiar one:

test = require('tape');
test('failed plan', function(t){
  t.plan(1);
});

test('not even called', function(t){
  t.comment('------------ not called');
  t.plan(1);
  t.pass();
  // t.end();
});

Expected the 2nd test to pass but it doesn't even get called. Whether the 2nd test uses t.plan or t.end makes no difference. Additionally, all tests in other files also fail in the same fashion when run as tape test/*.js (the above is the minimal necessary to reproduce the issue).

Output below:

TAP version 13
# failed plan
not ok 1 plan != count
  ---
    operator: fail
    expected: 1
    actual:   0
    at: process.<anonymous> (/home/pechkin/data/kb/node_modules/tape/index.js:85:19)
    stack: |-
      Error: plan != count
          at Test.assert [as _assert] (/home/pechkin/data/kb/node_modules/tape/lib/test.js:310:54)
          at Test.bound [as _assert] (/home/pechkin/data/kb/node_modules/tape/lib/test.js:91:32)
          at Test.fail (/home/pechkin/data/kb/node_modules/tape/lib/test.js:404:10)
          at Test.bound [as fail] (/home/pechkin/data/kb/node_modules/tape/lib/test.js:91:32)
          at Test._exit (/home/pechkin/data/kb/node_modules/tape/lib/test.js:261:14)
          at Test.bound [as _exit] (/home/pechkin/data/kb/node_modules/tape/lib/test.js:91:32)
          at process.<anonymous> (/home/pechkin/data/kb/node_modules/tape/index.js:85:19)
          at process.emit (node:events:376:20)
  ...
not ok 2 test exited without ending: not even called
  ---
    operator: fail
    at: process.<anonymous> (/home/pechkin/data/kb/node_modules/tape/index.js:85:19)
    stack: |-
      Error: test exited without ending: not even called
          at Test.assert [as _assert] (/home/pechkin/data/kb/node_modules/tape/lib/test.js:310:54)
          at Test.bound [as _assert] (/home/pechkin/data/kb/node_modules/tape/lib/test.js:91:32)
          at Test.fail (/home/pechkin/data/kb/node_modules/tape/lib/test.js:404:10)
          at Test.bound [as fail] (/home/pechkin/data/kb/node_modules/tape/lib/test.js:91:32)
          at Test._exit (/home/pechkin/data/kb/node_modules/tape/lib/test.js:267:14)
          at Test.bound [as _exit] (/home/pechkin/data/kb/node_modules/tape/lib/test.js:91:32)
          at process.<anonymous> (/home/pechkin/data/kb/node_modules/tape/index.js:85:19)
          at process.emit (node:events:376:20)
  ...

1..2
# tests 2
# pass  0
# fail  2

The event emitted at the bottom of the stack is exit from the async trampoline.

Reproduced here. Note that the codepen doesn't capture the same output as when running the same code locally.

Tested against tape v5.1.0 and v5.2.0.

ljharb commented 3 years ago

I assume this happens the same in tape v4 - I've noticed this many times in the past and it didn't register as an inherent problem, but it's reasonable to expect that the second test functions normally as long as the first test doesn't terminate the process.

Raynos commented 3 years ago

I've also noticed this many times that when tape fails sometimes it fails more loudly then other times.

I always ignore the noise and fix the root cause without thinking of removing the noise.

SH4R1K7 commented 3 years ago

but it's reasonable to expect that the second test functions normally as long as the first test doesn't terminate the process.

Yep, I should've made that a bit clearer I guess - I have lots of tests that pass but having added a new test that fails the plan causes all of my passing tests to 'fail' as well; I say 'fail' because they're not actually executed but are reported as having failed.

I ignored it for some time as well but I'm at a point where the only way to fix the failing test is to fix a few other ones first which is made very difficult - the only way out is to t.skip every test that causes this issue, then unskip them, round and round.

ljharb commented 3 months ago

Looking back at this, uncommenting the t.end() in your OP's example has the proper output.

I've added a failing test case in https://github.com/tape-testing/tape/compare/plan-failure; notably if i swap the tap test cases (right now i have "with end" followed by "without end") then they both fail, so i think the issue is that there's something that's wrongly stateful.