search

DannyDainton / newman-reporter-htmlextra

A HTML reporter for Postman's Command Line Runner, Newman. Includes Non Aggregated Runs broken down by Iterations, Skipped Tests, Console Logs and the handlebars helpers module for better custom templates.
Apache License 2.0
357 stars 139 forks source link

Show "Total Failed Requests" instead of "Total Failed Assertions" in newman report #321

Closed simkat closed 3 years ago

simkat commented 3 years ago

image I have copied request.failed from the stats table session listed below in the report but it is giving blank value.

DannyDainton commented 3 years ago

I wouldn't add the .stats part to the {{#with summary}} block as that's going to mess with the rest of the values that's are being used in that section of the report.

Does adding {{stats.requests.failed}} not work for you?

simkat commented 3 years ago

Thanks so much for your quick reply. That does seem to have worked although there seems to be bug that if there is only 1 failed request. The {{stats.requests.failed}} returns as 0. Please let me know if anything can be done to fix this. Many many thanks for the quick turn around!

DannyDainton commented 3 years ago

Need more details to know what's happening here.

All depends what you mean by failed requests, why is it failing? What's making it 'failed'?

DannyDainton commented 3 years ago

I'm going to close this as it's not really an issue and you have the correct reference now.

We can still carry on the conversation in the thread though.

simkat commented 3 years ago

Thanks. By failed request I meant if ”only” one api call (request) in the postman collection suit fails due to failed assertions (one or more) within that request. Then the expected value for {{stats.requests.failed}} is 1 But Actual value is 0

This works fine for 2 (or more) failed api request (call) due to failed assertions (one or more) within those requests.

So for 2 failed api request {{stats.requests.failed}} returns 2 and so on.

Please let me know if you need more information. Thanks for helping out!!

DannyDainton commented 3 years ago

It feels like that's not going to work as you expect it to, a request can fail without assertions.

image

A failed assertion wouldn't cause the request to fail.

Can you provide an example of these things, talking about them is always going to be worst than showing what you mean?

All the data is coming from Newman and the reporter is just displaying the information that it gets.

As a quick way to show what's going to be in the final report, you can use the CLI reporter which shows that data in the summary table.

simkat commented 3 years ago

ah alright, thanks for the clarification. Basically, I am looking to report on number of api requests failed rather than assertions. So, in the example collection attached - there are 2 api requests and within them there are 5 assertions. 3 assertions are set to fail and 2 are set to passed. So, in the summary report on the top, we want to see "Total Tests Failed" as 2 rather than 2 assertions failed as 3. But, it appears may not be possible.

Please rename the file below to *.json to open in postman (as I couldn't attach json here so renamed to txt). test2.postman_collection.txt

DannyDainton commented 3 years ago

Failing Requests are not the same a Failing Assertions so what you require is never going to be completely correct.

The Tests are the Assertions, that's what is telling you failed in the Request. The Request failing is something different.

github-actions[bot] commented 2 years ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.