Reporter is unable to find tests for the call that resulted in 500

[avostretsov]

Is There An Existing Issue

• [X] I have searched the issue tracker for similar problems.

What Are You Seeing

When request results in 500, reporter is unable to find tests for this call and gives false positive result.

Steps To Reproduce The Issue

1. Run newman from the command line using postman collection you know would have one of the calls result in 500 (use mock server if needed).
2. From browser open html extra report > Total Requests tab > click on the call that resulted in 500

Expected: Tests for the call should have been executed

Actual: NO TESTS FOR THIS REQUEST - see screenshot attached

Full Newman Command Or Node Script

./node_modules/newman/bin/newman.js run https://api.postman.com/collections/15351868-40bd488b-2524-4280-91c3-b501d5db5ad4?access_key=XXXXXXX -d secrets-stage.json --env-var strive_host=https://api.uat.gpn.strivegaming.dev --env-var strive_base_path=/api/v1 --env-var rgs_host=https://rgs.stage.bspot.com --env-var rgs_base_path=/rgs/api/v1 --env-var rgs_game_base_path=/gapi/api/v1.3 --env-var baseUrl=https://d8dc28cd-10d3-40b9-a44c-173c528f2df5.mock.pstmn.io -r htmlextra,cli

HTMLEXTRA Version

1.22.11

Newman Version

5.3.2

Additional Context

No response

[github-actions[bot]]

Hey @avostretsov: 👋 - Thanks for helping to make the reporter better by raising this ticket. I will take a look as soon as I can and get back to you. Cheers, Danny.

[avostretsov]

For comparison, I am attaching a screenshot of the true positive test result for the same call when status is not 500

[DannyDainton]

Hey @avostretsov,

Do you have the cli console output for that 500 request? Can you recreate it consistently?

I've tried with a basic GET https://postman-echo.com/status/500 using the Postman Echo service - I can see tests being run as part of this 500 Status Code request.

I'm unsure what that request returned on your side so I can't really do much to be honest, the reporter just spits out what Newman feeds it so if there was an issue with the data at that point, it wouldn't have come through to the final HTML report.

The only thing that I can think of, that might cause that output, is if you had added the tests but didn't save the request before running the Collection with the Postman API. The changes may not have been sync at that point so i might look like there are no tests for the request.

[avostretsov]

I found my problem which is in the first two lines of the test. Since the call was unsuccessful, location var is undefined. Postman then goes to the next line and performs split() method on undefined which triggers an error in the log: "TypeError: Cannot read properties of undefined (reading 'split')". Postman then bails out from the tests of this call. So, no tests were executed below the second line which resulted in the false positive. Moving call's status assertion to the top fixes my issue.

I have couple of suggestions.

1. Is it possible for tests to still be executed if there are errors like mine?
2. Would it make more sense to neutrally color-code requests with no or ignored tests, say grey?

[DannyDainton]

It's nothing really that the reporter, this one or other ones, can do about that situation.

If the data isn't in the Newman Summary Object at the end of the run, I can't show anything. So if it's telling me there are no tests, that's all I can show.

There could be a million and one reasons why that might be the case but I can't write logic to cover them all 😅

I'm not sure I understand what you mean by point 2, can you provide an visual example and expand on how that situation might occur.

[avostretsov]

For point 1, I understand that this is not a reporter issue. I guess what I meant was that it could be useful to bring this issue up to the postman dev team so they can consider addressing it.

For point 2 - requests with failed tests under Total Requests are color-coded with Red and all other requests - with Green. I am trying to make a point that having requests with no tests marked Green can be misleading. Neutral color like grey would make more sense.

[DannyDainton]

Free free to raise an issue on the public tracker for the team to triage and set a priority against. It's not something I'll be handling. I like to keep this reporter separate to my other work.

A request doesn't need a test within it to be 'good' or 'bad' so changing the colour wouldn't be something I'll be doing. The report is only a template file which can be copied and the colour can be change by anyone using the reporter. You could basically do anything you like and present the data in the way that suits your own usecase.

I'm going to close this issue and it isn't really a bug with the reporter and more on the Postman/Newman side of things.

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.