tape-testing / tape

tap-producing test harness for node and browsers
MIT License
5.77k stars 307 forks source link

Problem with using stream events #465

Closed r0mflip closed 5 years ago

r0mflip commented 5 years ago

The stream created by tape.createStream() is closed on process exit. Until the process exits the close event on the stream won't fire. This is causing issues with tap consumer programs like tap-spec and tap-out where they cannot hook into the end or close event of the stream.

Consider this where add pases and multiply fails:

const test = require('tape');
const tapSpec = require('tap-spec');
const {add, divide} = require('../math.js');

// piping programatically
test.createStream()
  .pipe(tapSpec())
  .pipe(process.stdout);

test('should add', t => {
  t.equal(add(3, 4), 7);
  t.end();
});

test('should divide', t => {
  t.equal(divide(8, 2), 4);
  t.end();
});

I get the output:

> node test\test.js

  should add

    √ should be equal

  should divide

    × should be equal
    ------------------
      operator: equal
      expected: 4
      actual:   6
      at: Test.t (D:\dev\projects\testing\tap-spec-test\test\test.js:15:5)
      stack: |-
  Error: should be equal
  at Test.assert [as _assert] ...

Whereas when I use tap-spec from command line (removing programatic pipe) node test/test.js | tap-spec I get:

> node test/test.js | tap-spec

  should add

    √ should be equal

  should divide

    × should be equal
    ------------------
      operator: equal
      expected: 4
      actual:   6
      at: Test.t (D:\dev\projects\testing\tap-spec-test\test\test.js:15:5)
      stack: |-
  Error: should be equal
  at Test.assert [as _assert] ...

  Failed Tests: There was 1 failure

    should divide

      × should be equal

  total:     2
  passing:   1
  failing:   1
  duration:  176ms

The results in the bottom are obtained by tap-out on the close event of the stream. When run the close event on the stream won't fire until before the whole process exits, at which point tap-out stops working. In command line usage the stream close event fires as it is from stdin and still the tap-out or tap-spec process would be alive.

Is there a solution to this problem or should I be just pipe my tap output to other programs.

ljharb commented 5 years ago

Is there perhaps something the command line code for tap-spec is doing that your programmatic version isn’t?

r0mflip commented 5 years ago

I've checked and experimented, its totally the same. The only thing is when and how the stream ends.

tap-spec uses tap-out as a parser. tap-out listens for the end of the stream and emits an output event. That event tap-spec uses to print the count of tests, passed failed etc.

I've debudded it using breakpoints. The event is never fired, tap-out just waits for the stream to end and meanwhile the whole process exits. The process was just exiting. When I debugged tap-spec as a js file and passing input to it through stdin, it worked correctly.

tap-spec taking in input is the same as programatic one but the input is from stdin.

P.S.: I hardly understand whats happening inside tape, but came to infer that the stream is closed on process exit from this

ljharb commented 5 years ago

The streams stuff is the part of tape I'm least familiar with, unfortunately.

I'd be happy to review a PR - even just one containing failing tests would be helpful.

r0mflip commented 5 years ago

I'll see what I can do. I have a few ideas in mind but that might end up changing the API a bit.

r0mflip commented 5 years ago

@ljharb Made PR #466. Sorry for the delay, I had exams 😁