search

postmanlabs / newman

Newman is a command-line collection runner for Postman
https://www.postman.com
Apache License 2.0
6.86k stars 1.16k forks source link

Incorrect summary.run.executions information while running asynchronous requests #1882

Open groov1kk opened 5 years ago

groov1kk commented 5 years ago

  1. Newman Version (can be found via newman -v): 4.3.1

  2. OS details (type, version, and architecture): Windows 7

  3. Are you using Newman as a library, or via the CLI? Both

  4. Did you encounter this recently, or has this bug always been there: Always

  5. Expected behaviour: summary.run.executions array on "beforeDone" event has to contains proper information about current executions

  6. Command / script used to run Newman: newman run ./collections/some_collection.json

  7. Sample collection, and auxiliary files (minus the sensitive details):

  8. Screenshots (if applicable):

groov1kk commented 5 years ago

Steps to reproduce:

  1. Create simple postman project
  2. Add several async requests to test-script section (pm.sendRequest());
  3. Run collection programmatically.

  4. Check executions array in 'beforeDone' event. For example:

    // Finishes all test steps and tests emitter.on('beforeDone', (err, o) => {

    // Test run objects
const executions = o.summary.run.executions;

    }

Expected behaviour: this array has to contain proper information for all sent requests. Each execution object has to contain unique id and information about request/response status.

Actual behaviour: All executions objects have the same id of one of async request. If one of async request failed - all objects will have the same error messages.

codenirvana commented 5 years ago

Duplicate of #1868

shamasis commented 5 years ago

Closing as duplicate. Track the other issue for progress.

vikyd commented 4 years ago

I don't think this is duplicate of #1868 .

In the Tests of my Request :

pm.sendRequest('http://otherdomain', function (err, res) {
   // ...
});

And then in my newman program:

  .on('done', function(err, summary) {
    let resps = summary.run.executions
    // ...
  })

Expected behaviour: The response result of the Request itself

Actual behaviour: All executions(includes sendRequest) objects have the same id and response text .

Christopher-Kane commented 3 years ago

@shamasis @codenirvana Can we get some clarification on this issue based on what @vikyd stated? I'm tracking to track down an issue that is causing an issue in Report Portal and this or #1868 seems to be the issue.

shamasis commented 3 years ago

Let me have a look at it again. I'm reopening this just in case.

Christopher-Kane commented 3 years ago

I appreciate it @shamasis. If I can speak to my issue a little bit more. This issue reported here: https://github.com/reportportal/agent-js-postman/issues/8 the people responsible for that open source project believe that the issue that they are experiencing is caused by: https://github.com/postmanlabs/newman/issues/1868 which is still open. From what I can gather it seems that using pm.sendRequest or sending requests to authenticate with an OAuth2 resource result in the information coming back from newman not being correct in some way causing report portal to fail. I have tested the exact same setup using a service that does not use OAuth2 but rather basic authentication with a username and password and that worked as expected since there was no pm.sendRequest involved in fact even in the case with the OAuth2 resource, if I include a single request to get a token before I execute everything else in the collection instead of a pre-request script, it will still fail with seemingly the same error. Please let me know what information I can provide you with. Thanks again!