Async test cases that fail end the entire test suite

[m90]

I ran into some weird behavior on Mochify's side just now. When running the following test suite:

const assert = require('assert')

describe('some tests', function() {
  it('sync fail', function() {
    assert(false, 'false')
  })

  it('async fail', function(done) {
    setTimeout(function() {
      assert(0, 'zero')
      done()
    }, 1500)
  })

  it('sync ok', function() {
    assert(true, 'yes')
  })

  it('async ok', function(done) {
    setTimeout(function() {
      assert(1, 'one')
      done()
    }, 1500)
  })
})

Mochify will stop at the second test case:

# chromium:

  some tests
    1) sync fail
    2) async fail
AssertionError
Error: Exit 1

whereas Mocha would still continue:

 some tests
    1) sync fail
    2) async fail
    ✓ sync ok
    ✓ async ok (1502ms)

  2 passing (3s)
  2 failing

  1) some tests
       sync fail:

      AssertionError [ERR_ASSERTION]: false
      + expected - actual

      -false
      +true

      at Context.<anonymous> (test.js:5:5)

  2) some tests
       async fail:
     Uncaught AssertionError [ERR_ASSERTION]: zero
      at Timeout._onTimeout (test.js:10:7)

This behavior seems to have been introduced in version 5.6.1 as handling a pageerror event will now immediately exit because of this: https://github.com/mantoni/mochify.js/commit/547119aa917298429634da7d21c03b839ddc0ad6#diff-9352d7c807debf0acf1801f179b52ebeR119

I don't know exactly why this has been introduced and how Puppeteer, Mocha and Mochify should play together in case of an uncaught exception, so I am hesitant to PR with just removing the error emitting, although that did restore the previous (expected too?) behavior.

[m90]

I can also confirm that this is not a change in puppeteer causing this as it will also happen when running mochify@5.6.1 against puppeteer@1.0.0.

[m90]

:birthday: Fixed via #192 and released in v6.0.4 :birthday: