search

postmanlabs / newman

Newman is a command-line collection runner for Postman
https://www.postman.com
Apache License 2.0
6.89k stars 1.17k forks source link

Newman CLI results to indicate the source (i.e. folder or collection) for the particular test in the results for better debugging #2022

Open rafkhan opened 5 years ago

rafkhan commented 5 years ago

I started by writing the failing test case, checking if my endpoint 401s at the right time. Despite not adding a 200 check, newman seems to be automatically inserting that into every test.

↳ Fail to get Client AT in Authorization Body w/ Bad Key and Secret
  POST http://localhost:5001/api/v1/accessToken [401 Unauthorized, 350B, 16ms]
  6. Status code is 200 // THIS IS THE FAILING TEST
  ✓  Body should be Unauthorized
  ✓  Status code is 401

Is there a feature on the horizon for writing tests to ensure endpoints fail correctly?

  1. Newman Version (can be found via newman -v):
  2. OS details (type, version, and architecture):
  3. Are you using Newman as a library, or via the CLI?
  4. Did you encounter this recently, or has this bug always been there:
  5. Expected behaviour:
  6. Command / script used to run Newman:
  7. Sample collection, and auxiliary files (minus the sensitive details):
  8. Screenshots (if applicable):
shamasis commented 5 years ago

Can you double check your collection that tests are not added in collection or folder level scripts?

@codenirvana can we add hints to indicate that the tests are from folder or collection scripts?

rafkhan commented 5 years ago

@shamasis You called it! That was exactly my problem, thanks so much.

I forked this collection and hadn't noticed that they setup those specific cases in a collection-level test. But yes! That would be a great idea, probably would have been more clear to know where the test was originating from.

rafkhan commented 5 years ago

@kunagpal what would something like this look like? Would you want the runtime to report the source location of any test run? I'd definitely be open to make this contribution with a little design help.

narayanpai1 commented 4 years ago

@codenirvana Can I work on this? If the file is local, just report the location of the collection and if its a link, report the link address right?

narayanpai1 commented 4 years ago

@codenirvana I am able to show the location of the test as shown below: gsoc_pic I had to add a function in https://github.com/postmanlabs/newman/blob/develop/lib/util.js to get the test location, given (PostmanItem, script-id) and reference it from https://github.com/postmanlabs/newman/blob/develop/lib/reporters/cli/index.js. Let me know if the feature is as required and if it is okay to modify the util.js file.

narayanpai1 commented 4 years ago

@shamasis Is this what you were expecting? Can i proceed with a pull request?

shamasis commented 4 years ago

The output is in right direction. This is fantastic. I’m worried that we made things look very busy in CLI output and that in most cases this is not the most important information to always display in brackets.

Ideas? Perhaps only show an indicator to only indicate that this test was run from an inherited script and not show expanded details until --debug is set?

The premise is that this info would not be needed for majority of scenarios and we shouldn’t consume valuable character real estate and add cognitive load while reading test info for everyone.

csk1827 commented 4 years ago

@shamasis Yes, the detailed CLI output is not needed in most of the cases. It would be better if these details are provided to the user on demand i.e. on setting -debug or some other relevant new tag.

narayanpai1 commented 4 years ago

@shamasis I agree. So I added just a symbol adhering to their meaning in the context ie.

  1. symbols.root for a collection-based test
  2. symbols.folder for a folder-based test

It looks like this now: image

Does this work?

And for the debug option, I am ready to work on it. Let me know if adding an option to the command is feasible.

shamasis commented 4 years ago

We can use --verbose. It already exists. 😊

narayanpai1 commented 4 years ago

I have completed both of the tasks. But the test-library is failing after adding my code for decreasing the branch coverage below the threshold. This is because my function is defensive to argument errors(just like the other functions) and there is no way to cover those branches in any situation in an integration test. I had decided to add unit tests after you check my code. But now, I am stuck at this point. Can you please help me out @shamasis?

Abhirup-99 commented 4 years ago

@narayanpai1 is this issue still up? I would like to work on this.

narayanpai1 commented 4 years ago

Hi @Abhirup-99, my PR already adds the feature and I have been told it will be reviewed shortly.

Abhirup-99 commented 4 years ago

Thank you for the clarification.

cgaswin commented 3 years ago

Is this issue fixed or should I be working on this issue?