Newman does not wait until very end of execution to extract test results

[Tiller]

Newman Version: 3.9.3 OS: Various unix based systems Using: newman as a node library

Hello, I'm running into an issue when executing a custom test in newman, and the same test works just as expected into the postman client.

What code am I running? Basically:

var done = false;

SwaggerParser.validate(body)
  .then(function(api) {
    tests["Definition file is valid"] = true;
    done = true;
  })
  .catch(function(err) {
    tests["Definition file is valid"] = false;
    done = true;
    console.error(err);
  });

// I KNOW, but postman won't wait for the swagger parser execution otherwise
var waits = function() {
    if (!done) {
        setTimeout(waits, 100);
    }
};

waits();

And here's the catch: the swagger parser library I'm using is asynchronous and returns a Promise. If I just execute the validate method as is, the execution stops even before the validation has started.

So the only trick I found is to use setTimeouts to block the engine from going on until the asynchronous function is completed. And that part - as far as I can see - works fine in both postman and newman.

Where the two softs differ is in the "Definition file is valid" test. Postman take into account the said test and include it into the final result where newman just don't care. The test is not logged in the output, and it doesn't care whether it's false or not. Might be note worthy: when using the "cli" reporter and putting a 10s delay on the setTimeout, my other tests (synchronous, in the same execution) are printed instantly into the console, then the console hangs for 10s before continuing the run

So in summary: newman properly pause execution when setTimeouts are used, but do not extract tests that are added within these timeouts.

[Tiller]

After further testing this morning, it appears to work fine with newman 3.8.3

→ Recursively Test Doc Api GET... [200 OK, 7.1KB, 44ms] ✓ Status code is 200 ✓ Response time is less than 800ms ✓ Disabled perm ... is not present in document ✓ Definition file is valid

Where with newman 3.9.3:

→ Recursively Test Doc Api GET ... [200 OK, 7.1KB, 48ms] ✓ Status code is 200 ✓ Response time is less than 800ms ✓ Disabled perm ... is not present in document

[kunagpal]

@Tiller Does this still happen with Newman v3.9.4?

[kunagpal]

@Tiller Closing issue due to inactivity, feel free to revert if you have any further questions 😄