search

postmanlabs / newman

Newman is a command-line collection runner for Postman
https://www.postman.com
Apache License 2.0
6.88k stars 1.17k forks source link

Limit CLI output to only failed requests #2736

Open mschechter-bellese opened 3 years ago

mschechter-bellese commented 3 years ago

Version Information:

  1. Newman Version (can be found via newman -v):
    • 5.0.1
  2. OS details (type, version, and architecture):
    • Docker via newman:5.0.1-alpine
  3. Are you using Newman as a library, or via the CLI?
    • CLI
  4. Did you encounter this recently, or has this bug always been there:
    • As far as I know, this has always been there
  5. Expected behaviour:
    • Requesting a behavior change
  6. Command / script used to run Newman:
    • run <collection> -e <environment> -d <data> --bail --silent
  7. Sample collection, and auxiliary files (minus the sensitive details):
    • Data not available / relevant
  8. Screenshots (if applicable):
    • N/A

Description of desired functionality:

It would be very useful to have a switch for the CLI reporter (and possibly the JSON reporter as well) that would filter the output so that only failed test cases were included.

This is different from the resolution proposed for #136, in that the output should only include failures. Having separation within the JSON file is one thing - when running as part of a CI pipeline, we want to limit the logging/output as much as possible, and the CLI reporter currently does not support this.

This will reduce cycle time and simplify troubleshooting, as we'd be able to see exactly which test cases failed and with what results without having to go through hundreds or thousands of success cases to find the failures.

This could be useful for the JSON reporter as well, as it would limit the file size and reduce noise.

elit-altum commented 3 years ago

Hey @mschechter-bellese ! Would using this reporter by @DannyDainton with the --reporter-htmlextra-showOnlyFails fill your usecase ?

mschechter-bellese commented 3 years ago

@elit-altum,

Thanks for the information!

I am using Newman via Docker as part of my CI pipeline; I need something that will fail the build if the tests do not complete as expected. If this will still generate a non-zero return code, it's a potential option.

However, this will double the size of the Docker image I need to download (108.46 MB vs. 48.47 MB); as I'm using virtualized builds, the download has to occur on every build across a large number of applications. This could impact the timeliness of build status results.

I think that given the existing switches on the CLI/JSON reporters (and the switch you recommend on the htmlextra reporter), it would be good to have the option to limit the output to failures only for the Postman-maintained reporters.