Q: proper way to create a test case in the suite that's already running?

I am implementing a tool for combinatorial reachability testing of asynchronous code -- https://github.com/MaxMotovilov/cort. It operates by running a test case repeatedly to execute all valid permutations of its explicitly declared asynchronous steps. Because of that, individual runs of the same case have to be registered with the reporting platform dynamically, not knowing in advance how many will there be (a while loop, not a for loop).

The trouble I run into when trying to integrate it with Mocha is that at a point where test case is running, list of tests to be executed within the suite is already inaccessible. The best solution I came up with so far is a fairly nasty monkey patch: replace suite.tests.slice with a function that would return an in-place window into suite.tests so that the execution loop can be extended while it's still running. Simplified sample code is below; I would very much like to know of a better way to achieve the same end if it exists.

class InPlaceSlice extends Array {
    constructor( from ) {
        super( ...from );
    }

    slice( from, to ) { return new Window( this, from, to ) }
}

describe( "Dynamic test creation", function() { 

    this.tests = new InPlaceSlice( this.tests );

    var total = 3, suite = this;

    it( testName(), function myTest( done ) {
        setImmediate( () => {
            if( --total > 0 ) {
                suite.addTest( new this.test.constructor( testName(), myTest ) );
            }
            done();
        } );
    } );

    function testName() {
        return "Should create " + (total-1) + " more tests asynchronously"  
    }
} );

class Window {
    constructor( array, from, to ) {
        this.array = array;
        this.from = from || 0;
        this.to = to;
    }

    shift() {
        return this.array[ this.from++ ]
    }

    get length() {
        return this.end - this.from
    }

    get end() {
        return this.to == null ? this.array.length : this.to < 0 ? this.array.length + this.to : this.to
    }
}

Altering the set of tests to run is unsupported and unlikely to work entirely as expected no matter how it is attempted. A variety of factors in Mocha's design necessitate running in two phases:

define tests
execute tests

(The first phase can be performed asynchronously if necessary, although it isn't always intuitive or pretty.)

You will have to separate the logic for determining what the tests are from the actual testing steps, e.g. instead of something like this:

it("generates more tests", function() {
  var testableList = getArrayDynamically()
  assert.stuffAbout(testableList)
  testableList.forEach(function(entry) {
    doSomethingToAddTestThatRuns(function() {
      assert.otherStuffAbout(entry)
    })
  })
})

...instead do something like this:

var testableList = getArrayDynamically()
it("just the test portion of what was the generator", function() {
  assert.stuffAbout(testableList)
})
testableList.forEach(function(entry) {
  it("test for " + entry, function() {
    assert.otherStuffAbout(entry)
  })
})

Unfortunately, the value of the tool is in making the process of exhausting permutations transparent to the user (i.e. the test creator) -- other than using the step designation API (later()), the rest of the test code follows guidelines of the testing platform it is written for (e.g. Mocha, nodeunit...). Moreover, the search tree (and therefore the total number of possible permutations) cannot be built by static analysis: the test code has to be run repeatedly; it is not impossible that certain permutation of the steps may result in additional steps being executed!

As to unlikely to work entirely as expected no matter how it is attempted -- well, that's why I'm asking. The "solution" above came from a couple hours spent investigating Mocha code with debugger; I am already aware of at least one issue with it (no support for retry()). It is usually possible to support at least a large subset of functionality with intricate kludges of this sort (thankfully JS provides extensive introspection facilities!) but dependency on specific implementation details is the bigger problem that I would like to minimize if possible.

I've looked at Cort a bit to try to understand what you were saying about the use case, and it seems to me you've got three options to work within Mocha's constraint about not defining more tests while tests are already running:

Just use cort-unit inside an it:
```
const cort = require("cort-unit")
```

it("combinatorial whatever", cort.bind(null, (later, done) => { later(() => console.error("step 1")) later(() => console.error("step 2")) .later(() => (console.error("step 3"), done())) }))


- Immediately determine all possible orders in which to run the `later`s when a Mocha-cort function is called; for each possible order of `later`s, call `it(meta.name, <function that runs that order of laters and calls Mocha's done callback with the individual result>)`; no additional "permutations" may be determined based on the results of actually running any one "permutation". (You may also need to make sure that this Mocha-cort function gets the current `it` function and doesn't just rely on it being truly global; I'm not sure what all Mocha's doing behind the scenes when it sets up its globals for the test files.)
- Run all the possible "permutations" immediately but, rather than handing off the single complete result to an `it`, instead hand off each "permutation"'s individual result to `it(meta.name, done => done(<result from Cort>))` (This has the same caveat about the current `it`. It has an additional caveat that, since Cort won't know what `it`s to call till after each asynchronous permutation runs, you will need to work Mocha's [`--delay` and `run`](https://mochajs.org/#delayed-root-suite) into it also -- calling Mocha's `run` only after all the permutations are finished and their `it` calls made.)

Just use cort-unit inside an it

Which means making one test out of multiple runs of the same test. Lots of reasons why it's a bad idea; the biggest one being that all the setup/teardown hooks will not be executed for each run of the test but only once before the first run and after the last one. Basically this means giving up most of the Mocha facilities.

Immediately determine all possible orders in which to run the laters when a Mocha-cort function is called

You mean, before executing it() for this test even once? Same issue with hooks, plus I'd need to somehow record all assertions. Also, I don't think it generally possible to determine all possible orders: certain code paths in the test may not be activated depending on the order.

Run all the possible "permutations" immediately

About the same thing as the one before: test code has to be run outside the context of Mocha test?

The way things are looking, my original option "just kludge through the dynamic definition and see what breaks" is almost attractive enough to follow up on :-) I guess I'll try it and see what happens. The runner loop in Runner.prototype.runTests() looked encouraging enough...

Same issue with hooks, plus I'd need to somehow record all assertions.

What?

The second option is (roughly) equivalent to:

it(<first permutation>, function() {
  firstStep()
  secondStep()
  thirdStep()
})

it(<second permutation>, function() {
  secondStep()
  firstStep()
  thirdStep()
})

it(<third permutation>, function() {
  secondStep()
  thirdStep()
  firstStep()
})

That should work just fine with Mocha's hooks, both the ones that run once per suite and the ones that run once per test (whichever is appropriate for a given use case); just put them before or after the Cort process that generates the permutations. (I'm also not sure how saving assertions comes into play in this scenario. Maybe we're talking past each other about design here for lack of more detailed examples?)

Also, I don't think it generally possible to determine all possible orders: certain code paths in the test may not be activated depending on the order.

I'm having a hard time offering any further advice on this front without a more concrete example. Everything I see in the readme looks like the possible orders in which to run the later callbacks can be determined once all the later callbacks are known. Are we talking here about creating more later callbacks inside one of the running later callbacks? Or about the idea that if a later callback that calls done() runs before one of the others then the remaining later callbacks shouldn't end up being run? Or some other mechanism not yet described in the readme?

About the same thing as the one before: test code has to be run outside the context of Mocha test?

Not sure if you're asking if that's the one thing that's different or asking if they're the same in that regard, but to answer either question: that's how they're different, the second runs the later callbacks inside the Mocha tests (one test per permutation), the third runs them right away and as each permutation finishes generates a Mocha test that merely uses the result of that permutation. Again, it's (roughly, I'm trying to describe high-level logic and not things like the asynchronicity or how try-catching errors would work) equivalent to:

var passed

passed = firstStep()
  && secondStep()
  && thirdStep()
it(<first permutation>, function() { if (!passed) { throw new Error(<first permutation failure>) } })

passed = secondStep()
  && firstStep()
  && thirdStep()
it(<second permutation>, function() { if (!passed) { throw new Error(<second permutation failure>) } })

passed = secondStep()
  && thirdStep()
  && firstStep()
it(<third permutation>, function() { if (!passed) { throw new Error(<third permutation failure>) } })

Unlike the second option, this pretty clearly rules out using Mocha's hooks (even moreso than the first option in fact, since the first option at least could use before[All] and after[All] and this can't use any), although (also unlike the second option) it would be able to handle finding out the possible permutations not-ahead-of-time.

The second option is (roughly) equivalent to [snipped]

Not really: "immediately determine all possible orders" means "run the test body at least once before any of the it() bodies are executed" -- static code analysis not being generally feasible, and a monumental task in any case -- hence 1st run is not made inside an it(). I was playing with a variation of this idea where 1st run is made in a suite separate from (and preceding) the one where new tests would be injected but (a) it would still violate the rule of 2 separate stages and (b) there is ultimately no guarantee all code paths are even entered during the first run so you can't determine all possible orders from it anyway. Dynamic insertion of suites appears (from looking at Mocha code) to have the same exact problems as dynamic insertion of tests.

the third runs them right away and as each permutation finishes generates a Mocha test that merely uses the result of that permutation

Ok, I understand now. This definitely means forgoing most if not all Mocha facilities :-(

Hey @MaxMotovilov, sorry for the late reply -- I wrote up the following comment, then went to double-check the examples on the repo and forgot to come back here and post it. I figure it's worth letting you know what my thoughts were, but feel free to ignore any questions here if they're adequately answered by the readme now -- I need to get back over there and check out your updates! (What can I say, my life is a bit hectic.)

Hmm. So this might be where I've been misunderstanding something and/or having trouble figuring out how this works. The initial callback that calls later is run once per permutation rather than just once to get the possible permutations? How exactly does Cort know when all the possible permutations have been found and that the callback wouldn't have done something different on the next run after the one Cort considers the last?

Instinctively, I'd be more inclined to do something like:

beforeEach for setup that has to happen every time
define all the possible laters once
have a callback that is called once per permutation at the end that can sanity-check the results

(Maybe that last one is the onComplete callback? I was under the impression that that runs just once for the whole run rather than once per permutation, but I still haven't got as thorough a handle on how Cort's intended to work as I'd like.)

How exactly does Cort know when all the possible permutations have been found and that the callback wouldn't have done something different on the next run after the one Cort considers the last?

It maintains a "search tree" throughout the runs. Each tree node contains all available events to choose from at this stage: there're two kinds in 0.2.0 -- a "scheduled" step (i.e. call to later() has executed but the callback hasn't) and a "running" step (i.e. the callback has executed, it had the ready parameter -- new since last time, see the doc, the notifying call to ready() has been made but the user's callback has not yet been called or promise resolved). The edges leading away from the node are events already tried in previous runs. Cort keeps growing this tree until something breaks in the test, it encounters non-deterministic behavior (the event should be available but isn't) or it runs out of search space and calls it a success. Hence, it is quite conceivable that the entirety of search space is not discovered in any specific number of runs. It is also quite conceivable that the search space proves infinite, thus I provide a hard limit option maxRuns.

I think you might understand it better now if you go through the new readme -- it now has two reasonably real-life examples with runnable Mocha test code. Note that beforeEach() / afterEach() hooks actually work, and so does retry(); I am really curious as to what Mocha functionality I do break with my ham fisted kludges if anything...

mochajs / mocha

Q: proper way to create a test case in the suite that's already running? #2854