Newman sends report summary and custom report after 180 sec and continues to run.

[bbartoni]

1. Newman Version (can be found via newman -v):3.9.x
2. OS details (type, version, and architecture):confirmed on Linux Mint and Mac
3. Are you using Newman as a library, or via the CLI? library
4. Did you encounter this recently, or has this bug always been there: recently
5. Expected behaviour: newman should report after the run is completed
6. Command / script used to run Newman: node script
7. Sample collection, and auxiliary files (minus the sensitive details):
8. Screenshots (if applicable):

When executing a script which runs newman and interprets the newman run, then sends a custom report to Testrail, we found that during the test run (180 sec into the run) newman triggers the cli reporter, which triggers our custom reporter, but the newman run continues to execute in CLI. This results in incomplete reports. Our collection is quite big with up to 13 different iterations and up to 2K assertions so it takes way longer than 180 sec to complete all iterations.

I found that this behavior is present in 3.9.x newman versions.

Rolling back to v3.8.3 fixed our issue!

Placed this issue here in hope that someone will find this to be the solution for their issue.

[kunagpal]

@bbartoni Thanks for reporting this issue. Could you also share the following details?

1. The node and npm versions being used.
2. The script used to invoke newman.run.
3. Details on the custom reporter would also be useful here.

[bbartoni]

node v9.2.1. npm v5.6.0. more info to come, but like I said, it's working fine now that we rolled back the newman version.

here's the runNewman function:

function runNewman(callback) {
  var testRunList = []
  var aggregatedResult = null

  async.map([null], function(folder, result) {
    // Get the parameters from env variables
    currentOptions = {
      reporters: config.options.reporters,
      environment: path.join(__dirname, 'environments/'+config.options.environment), //your env
      iterationCount: parseInt(program.number),
        insecure: config.options.insecure
    };

    if (program.collection) {
      currentOptions.collection = path.join(__dirname, 'sanity-collections/'+program.collection)
    } else {
      currentOptions.collection = path.join(__dirname, 'sanity-collections/'+config.options.collection)
    }

    if (program.globalVar) {
      currentOptions.globals = getGlobals(program.globalVar)
    } else {
      currentOptions.globals = getGlobals(null)
    }
    callback(newman.run(currentOptions, callback))
  })
}

custom reporter sends a bulk report to Testrail after DONE state is achieved on the newman process.

[kunagpal]

@bbartoni The figure of 180 seconds (3 minutes) here is interesting as it is exactly equal to the default global timeout used by postman-runtime (the underlying library to unify collection run behaviour across Postman monitors, the app, and Newman.). Here, the global timeout is the total time for the entire collection run to complete. This could well be the reason for the collection runs having a blip in behaviour at the 3-minute mark.

As for the perceived continuation in the collection run, this could be caused by a workaround we had to apply in order to force collection run output to be dumped to the console in case of errors. Essentially, the pending content in the output buffer is forcefully flushed right before the process exits. This could lead to the misleading conclusion that the collection run is continuing when it actually isn't.

Could you try explicitly setting timeout, timeoutScript, and timeoutRequest to 0 in your collection run options object and let me know if that helps? Thanks!

A (somewhat) similar discussion has happened here: https://community.getpostman.com/t/error-callback-timed-out-why/1508/17

[kunagpal]

In addition, a fix (of sorts) has been added in https://github.com/postmanlabs/newman/pull/1630, and will be available with Newman v4.

[shamasis]

@kunagpal - could it be that runs are forcing through even after the outermost timeout has been fired? If that is the case then we would need to check if abort signal is being properly sent from outside after the outermost setTimeout is fried.

[kunagpal]

@shamasis Yes, that is also a possibility. We need to confirm the exact underlying cause.

[bbartoni]

This is not an issue for me any longer, @kunagpal . after temporary roll-back to v3.8.3, we recently added explicit timeout: 120000 (which suites our needs, but still limits the exec), I then updated to newman -v 3.9.3 and been running without any problems to this day.