postmanlabs / newman

Newman is a command-line collection runner for Postman
https://www.postman.com
Apache License 2.0
6.88k stars 1.16k forks source link

html report with duplicate information for multiple iterations #1049

Closed sophiejacquin closed 7 years ago

sophiejacquin commented 7 years ago
  1. Newman Version (can be found via newman -v): 3.5.2

  2. OS details (type, version, and architecture): ubuntu 16,04 64 bits

  3. Are you using Newman as a library, or via the CLI? CLI

  4. Did you encounter this recently, or has this bug always been there: my first try

  5. Expected behaviour: When I run a collection with n iteration i would like to get a html

    report with a single resume but I got n time the identical one as in the join file. Is it possible to avoid that? pb_report_redondances.pdf

  6. Command / script used to run Newman: newman run general_positive_create_case.postman_collection.json -e test2_superadmin.postman_environment.json --export-globals globals.json -d create_100_tenants.json -n 3 -r html --reporter-html-export Reports.html

czardoz commented 7 years ago

@sophiejacquin Can you elaborate on what the expected behavior is? Newman aggregates results across multiple iterations for the HTML reporter.

sophiejacquin commented 7 years ago

The collection contain 3 test: -get_nb_objects -create -check_object_added

In the report we can see that there are 9 results description blocks, so 3(which is the number of iterations) for each test, but all of them give a resume of what happens accross all the iteration. So it is a bit redundant. I ideally would like to have only one block result for each test that aggregates results across the iterations. So I would like to get only the 3 first description block of the file. Is there a way to do that?

I Hope I am clear enough...

czardoz commented 7 years ago

@sophiejacquin Newman does not do that by default, but you can totally do this by writing your own reporter.

For a simple example of a reporter, check out our Emoji reporter: https://github.com/postmanlabs/newman/blob/develop/lib/reporters/emojitrain.js

I'm closing this issue, but feel free to comment here or open a new one if you have any questions :smile:

sophiejacquin commented 7 years ago

I am just a bit curious on why it is done like that. Are there some cases were it will not be redundant?

czardoz commented 7 years ago

@sophiejacquin Over the last couple of years, we've noticed that a lot of people have very specific reporting needs, someone may want an aggregated report, whereas someone else is interested in the details, which can help them debug issues.

Since it's not possible to allow the huge number of these customizations within Newman core, we've built it to allow people to use their own custom reporters.