search

postmanlabs / newman-reporter-html

Apache License 2.0
81 stars 86 forks source link

summary.run.executions contains each request only once, even if executed multiple times #189

Open mcserbak opened 6 years ago

mcserbak commented 6 years ago

  1. Newman Version (can be found via newman -v): 4.0.2

  2. OS details (type, version, and architecture): Linux (Ubuntu, 18.04.1, x64)

  3. Are you using Newman as a library, or via the CLI? Via the CLI

  4. Did you encounter this recently, or has this bug always been there: Encountered recently, but checking the git repo, it has always been there

  5. Expected behaviour: I'd expect summary.run.executions to contain one item per request executed, even if the request is the same one looped over and over again. The problem is observable by checking newman.report.html, where the Get Timestamp request turns up only once, with a response body not fulfilling the assertion, however the Test Script on the request is set up in a way, to re-run the same request while a condition holds. My suspicion is that the response body is saved from the first execution of the request only. Suspected code: https://github.com/postmanlabs/newman/blob/8426e61ade45c76ee6dfde23ce25235c7c5c1648/lib/run/summary.js#L143

  6. Command / script used to run Newman: newman run NewmanHtmlReporterMultiRequestIssue.postman_collection.json --reporters cli,html --reporter-html-export newman.report.html --reporter-html-template template-withreqrespcontents.hbs

  7. Sample collection, and auxiliary files (minus the sensitive details): NewmanHtmlReporterMultiRequestIssue.postman_collection.json.txt <- please remove .txt extension template-withreqrespcontents.hbs.txt <- please remove .txt extension

  8. Screenshots (if applicable): N/A

Steps to reproduce the problem:

  1. Run newman command in point 5.
  2. Check resulting newman.report.html file
kunagpal commented 6 years ago

@mcserbak This is intentional, as the HTML reporter aggregates run statistics across multiple iterations. Quite a few users have been caught off guard by this behaviour, and we're looking at ways to improve this to become more usable.

mcserbak commented 6 years ago

@kunagpal I totally see your point. Between iterations, even the number of repeats could differ.

Let me rephrase my request in a way, that would be valid even if I had more iterations: As a tester, I absolutely need to have those request and response bodies logged, that made any of the assertions fail, so that I can show them to the dev team in my bug reports.

My first preference would be to have it in the junit report, but I can live with any other solution (having it logged in the HTML report, or even in console).

Thanks!

jhardin-aptos commented 5 years ago

To rephrase @mcserbak's rephrasing: if the loop changes GET route URL members or query parameters, you won't have a clear indication of which request had a problem if a test failed on only some requests.

I worked around this by adding the relevant GET route info into the test description, but that's somewhat clumsy:

image

It would be better if it the HTML reporter output was closer to the default CLI report:

→ Basic data request - data driven
  ┌
  │ 'INFO: Loop index = 27 of 50 customer number 9999910337'
  └
  GET https://localhost:44393/CRMAPI/v1/customer/9999910337/email [200 OK, 1.59KB, 40ms]
  √  Customer 9999910337: Response is OK
  √  Customer 9999910337: Response must have a body
  √  Customer 9999910337: Returned an array of contacts
  √  Customer 9999910337: Returned desired contacts

Attempting to set next request to Basic data request - data driven

→ Basic data request - data driven
  ┌
  │ 'INFO: Loop index = 28 of 50 customer number 9999910352'
  └
  GET https://localhost:44393/CRMAPI/v1/customer/9999910352/email [200 OK, 1.6KB, 49ms]
  √  Customer 9999910352: Response is OK
  √  Customer 9999910352: Response must have a body
  √  Customer 9999910352: Returned an array of contacts
  √  Customer 9999910352: Returned desired contacts

Attempting to set next request to Basic data request - data driven

→ Basic data request - data driven
  ┌
  │ 'INFO: Loop index = 29 of 50 customer number 9999910358'
  └
  GET https://localhost:44393/CRMAPI/v1/customer/9999910358/email [200 OK, 2.03KB, 32ms]
  √  Customer 9999910358: Response is OK
  √  Customer 9999910358: Response must have a body
  √  Customer 9999910358: Returned an array of contacts
  √  Customer 9999910358: Returned desired contacts

Attempting to set next request to Basic data request - data driven

Still present in Newman 4.3.1, CLI