🐛 Bug: Timeout of 0 (no timeout) for tests that never resolve closes Mocha with always successful exit-code without finishing reporter output

[131]

Timeout 0 and un-resolved async function lead nowhere. No throw, no error, no next test running, nothing.

describe("should never end", function(){
  this.timeout(0);

  it("should never end", function(done){ });

  it("is here now", function(){
    console.log("Allgood");
  });
});

Current result

R:\>node d:dvp\node_modules\mocha\bin\_mocha test.js
  should never end
R:\>

Expected result

R:\>node d:dvp\node_modules\mocha\bin\_mocha test.js
  should never end
  (???? something ?? waiting undefinitively ?) , but NOT nothing
R:\>

[boneskull]

Thanks for the bug report. This is intended behavior, as this.timeout(0) will disable timeouts completely.

[131]

"disabling timeout" mean that because of a faulty test, my test suite app is broken (tests are not failing, nor succedding, code execution just stop and nodejs ends). I cannot understand how you can allow this setting.

[ScottFreeCode]

@boneskull Disabling timeouts means it will wait until the test completes no matter what (i.e. indefinitely in this case since the test never calls done), right? If so, I can confirm that this is instead stopping execution rather than waiting.

Not only that, but I've discovered something even odder: it doesn't even finish reporting. As in, if you put failing tests before the one that doesn't complete, their error details won't be printed at the end, and if you use a reporter that's supposed to list total count of tests passed/failed/pending it won't print that either. (That would presumably be a bug even if disabling the timeout isn't supposed to wait indefinitely.) But the return code is 0, so it's not crashing Node or anything like that...

[boneskull]

@scottfreecode I'm not really sure what you're asking. If your test doesn't complete, and you've disabled timeouts, how can Mocha exit except via a signal? I must not be understanding.

It's possible to, say, intercept the signal and tell the reporter to print a summary before exiting, but I can't tell if that's a good idea or a bad one.

But Mocha executes runnables in serial. If a runnable never ends, nothing else happens. So that may seem like stopping execution, but execution will continue if the runnable eventually completes.

So, you can disable timeouts. Use cases include stuff like debuggers. @131 if you don't want the behavior, don't use timeout(0).

[ScottFreeCode]

@boneskull That answers my question, thanks -- the behavior you describe is what I expected.

Here's a simpler and clearer demonstration of the issue:

test.js:

it("found the test file", function() {})
it("never ends", function(done) {})

commandline:

node_modules/.bin/mocha --timeout=0 test.js && echo "Mocha completed!"

√ found the test file Mocha completed!

[ScottFreeCode]

So, I ran into this at work the other day, and the weird thing there is my tests actually weren't -- as far as I'm aware -- failing to ever resolve, just taking too long (and I had set timeout(0) until I found out how long they take). This prompted me to look into the issue again.

Strangely, I couldn't reproduce it with a test that uses setTimeout or setInterval to resolve only after some time, like the tests at my job that prompted me to revisit it. I am going to look at those tests again and try to figure out why they were hitting this. ([edit] Pulled up my work test code and commented out the timeouts and added timeout of 0, and it didn't reproduce; something else about those tests must have been updated since I saw the timeout issue.) In the meantime...

...The fact I couldn't reproduce it with setTimeout leads me to suspect a possible cause: no work is scheduled on the NodeJS event loop, so Node shuts down. E.g.:

var start = Date.now()
process.on("beforeExit", function() { console.log("closing after " + (Date.now() - start) + " milliseconds") })

it("should never end", function(done) {})

mocha --timeout 0

closing after 31 milliseconds

[edit] Just tested a possible fix for this (not necessarily applied in the right place however -- it should be in the runner or something, I tested it in bin/_mocha): var keepalive = setInterval(function () {}, 30 * 1000); before running tests and clearInterval(keepAlive); after completion (so if using --no-exit Mocha still closes)... [edit again] This seems to work! [edit again again] But we can do so much better: https://github.com/mochajs/mocha/pull/2954#issuecomment-321837609

[ScottFreeCode]

Came up with a better way to handle this: If Node would close due to the test having no more actions scheduled and not having called done or resolved a returned Promise, then Mocha will fail the test immediately. (In the browser, I haven't found a way to get it to do anything other than hang, so it's still a potential issue for projects that only test in the browser and not Node that also use timeout of 0 and similarly disable Karma timeouts or whatever they're using to run Mocha in the browser...) See #2959 for proposed solution.

[jordantogether]

I still have this issue to this day. If I walk through each test, the time for execution depends, and if I leave the timeout, it's either too long or too short, so I set this.timeout(0) so each test has enough time and use done() to tell it it's done. My tests do eventually complete, just with rather long times to execute because they do large operations.

If I put breakpoints on the code, and step through each, mocha outputs the time in milliseconds after each test. However, if I just run this without any breakpoints, mocha randomly seems to give some tests times in ms, and others just don't seem to wait for the result - it still seems like there is a hard-coded timeout regardless of what I put, because mocha marks them as done, but I'm sure they're not. As soon as it's stepped through in the debugger, it works.

In this instance, I'm not using Promises, but Observables, and listening to the observable in the test and calling done() when the observable provides response, mocha provides an inconsistent output as to whether a test has completed satisfactorily and usually it concerns me that it doesn't have an execution time, which means mocha pretty much just gave up on it from what I can see.

Is there any understanding as to what causes this? Currently running latest version of mocha with NYC.

[ScottFreeCode]

If Mocha's marking the tests as complete and outputting the summary at the end, then it's not the same issue as this one (which is Mocha closing altogether, no more reporting anything)... Think you could put together a minimal code example that causes it, for us to take a look at? If so, a fresh issue for tests that shouldn't complete but eventually pass when this.timeout(0) would be great.

(Tangentially, the slow setting controls when Mocha prints the time a test took, or at least it's supposed to. It'll print in red if the slow time is passed or yellow if half the slow time is passed, if I recall correctly. You can fiddle with that to get it to print time more consistently if you want to see it, or to ensure it doesn't print time unless something's out of the ordinary for a test if you normally don't want to see it, or make it at least twice the timeout to have it never print time at all.)

[akc42]

This issue still seems to be outstanding from 2 1/2 years ago which seems surprising as I have just run into it again to day. How does everyone else handle it?

I have a series of tests for a set of code which was working a while ago. However I have been doing development and let the tests fall into disrepair. So today I have decided to re-run them and fix any errors. I am using Node 10.15.3

With a normal run a few of my tests are failing with a timeout failure. To and try and debug them I am running the very same tests with the --no-timeouts flag (I started with actually this.timeout(0) in the tests with the same result, but realised I had the command line flag option and removed the this.timeout(0) from the tests).

With timeouts in place all the test run, those that fail are printed out in red and I get the summary of x passing y failing, and then a list of each test and why it failed (because of timeout).

As soon as I run with --no-timeouts set, I get success messages for all tests prior to the first failure and then mocha exits back to the command line, not printing anything more.

As indicated above I think its because there is nothing more left in the node event loop (running in the Visual Studio debugger, that appears to be what is happening).