mmanela / chutzpah

Chutzpah is an open source JavaScript test runner which enables you to run unit tests using QUnit, Jasmine, Mocha and TypeScript.
http://mmanela.github.io/chutzpah/
Apache License 2.0
550 stars 143 forks source link

Chutzpah stops running QUNIT tests if one test times out #805

Open skelker opened 3 years ago

skelker commented 3 years ago

For this html file with 3 simple QUnit tests, the second test will timeout, and Chutzpah will then report an error and not run the 3rd test. Sorry about the formatting below. The error I see is:

Error: Unknown error occurred when executing test file. Received exit code of 2 While Running:D:\chutzpah\test.html

It does run ok in the browser. This is a simple example of the issue. I have a much bigger test file with hundreds of tests, but this timeout issue occurs with the 76th test and Chutzpah will not finish running the tests once that test times out. I did try it with the /debug argument, and I don't see anything enlightening other than a bunch of these:

#_#Log#_# {"type":"Log","log":{"message":"!!_!! isTestingDone"}}
<!doctype html>
<meta charset="utf-8">
<title>Test Suite</title>
<link rel="stylesheet" href="https://code.jquery.com/qunit/qunit-1.23.1.css">
<body>
  <div id="qunit"></div>
  <div id="qunit-fixture"></div>
  <script src="https://code.jquery.com/qunit/qunit-1.23.1.js"></script>
  <script>
    QUnit.config.autostart = false
    QUnit.config.testTimeout = 15000;
    QUnit.module('Timeout Module'); 
    QUnit.test("AAA This first test works", function (assert) {
        assert.ok( 1+1 == 2);
        });

    QUnit.test("BBB This second test will timeout", function (assert) {
        var done = assert.async();
        setTimeout(function() {
        assert.ok(1 == 1);
        done();
        }, 17000);
        });

    QUnit.test("CCC This this third test doesn't run with Chutzpah", function (assert) {
        assert.ok( 2+2 == 4);
        });
  </script>
</body>
D:\chutzpah>c:/ProgramData/chocolatey/lib/Chutzpah/chutzpah.console.exe test.html /engine Chrome
Chutzpah console test runner  (64-bit .NET 4.0.30319.42000)
Version 4.4.8.0
Copyright (C) 2018 Matthew Manela (http://matthewmanela.com).

Error: Unknown error occurred when executing test file. Received exit code of 2
While Running:D:\chutzpah\test.html

=== 1 total, 0 failed, took 11.31 seconds ===

If I modify the test to timeout at 9 seconds (and run the test longer than 9 seconds), then Chutzpah will run all 3 tests and fail the second test. A timeout higher than 9 seconds stops Chutzpah from running all the tests.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.