postmanlabs / newman

Newman is a command-line collection runner for Postman
https://www.postman.com
Apache License 2.0
6.88k stars 1.16k forks source link

Unable to separate results from different iterations in html report #591

Closed keithmork closed 8 years ago

keithmork commented 8 years ago

When using Newman to do data-driven test, results from different cases mixed together in the summary report... The request name doesn't support variables, like whatever title - {{caseName}} didn't work (in JMeter you can change sampler name through iterations and the result will be displayed separately).

data file: (I want to use 1 data file for 1 API, it contains both success and failure cases. )

[
  {
    "caseName": "Patient",
    "caseDescription": "",
    "statusCode": "200",
    "customCode": "1",
    "mustHave": ["patientId", "access_token", "userId", "telephone", "userType", "status"],
    "mustNotHave": ["doctorNum"],
    "userType": 1,
    "name": "xxx"
  },
  { 
    "caseName": "Doctor",
    "caseDescription": "",
    "statusCode": "200",
    "customCode": "1",
    "mustHave": ["doctorNum", "access_token", "userId", "telephone", "userType", "status"],
    "mustNotHave": ["patientId"],
    "userType": 3,
    "name": ""
  }
]

html report: (it's impossible to tell which iteration failed...)

Name    Pass count  Fail count
Status code is 200  2   0
Response time is less than 15000ms  2   0
Return success  2   0
response body should have: patientId    0   1
response body should have: access_token 2   0
response body should have: userId   2   0
response body should have: telephone    2   0
response body should have: userType 2   0
response body should have: status   2   0
response body should NOT have: doctorNum    1   0
[INFO] mobile: 14581382863  1   0
response body should have: doctorNum    1   0
response body should NOT have: patientId    1   0
[INFO] mobile: 14957437471  1   0
keithmork commented 8 years ago

I just found iteration variable in Postman, now the result will do, but still a bit inconvenient : (

Name    Pass count  Fail count
Loop: 0 [INFO] Title: Patient   1   0
Loop: 0 [INFO] Description: 1   0
Loop: 0 [INFO] mobile: 14295018089  1   0
Loop: 0 Status code is 200  1   0
Loop: 0 Response time is less than 15000ms  1   0
Loop: 0 Return success  1   0
Loop: 0 response body should have: patientId    0   1
Loop: 0 response body should have: access_token 1   0
Loop: 0 response body should have: userId   1   0
Loop: 0 response body should have: telephone    1   0
Loop: 0 response body should have: userType 1   0
Loop: 0 response body should have: status   1   0
Loop: 0 response body should NOT have: doctorNum    1   0
Loop: 1 [INFO] Title: Doctor    1   0
Loop: 1 [INFO] Description: 1   0
Loop: 1 [INFO] mobile: 14837803477  1   0
Loop: 1 Status code is 200  1   0
Loop: 1 Response time is less than 15000ms  1   0
Loop: 1 Return success  1   0
Loop: 1 response body should have: doctorNum    1   0
Loop: 1 response body should have: access_token 1   0
Loop: 1 response body should have: userId   1   0
Loop: 1 response body should have: telephone    1   0
Loop: 1 response body should have: userType 1   0
Loop: 1 response body should have: status   1   0
Loop: 1 response body should NOT have: patientId    1   0
shamasis commented 8 years ago

Having a better way to distinguish iterations will surely go a long way. But I'm afraid that this is not something we are working on currently. There's an option to add your own HTML template and soon we will document how to write your own reporter. That should help for a while till we add more fidelity to the HTML reports.

shamasis commented 8 years ago

This is being internally tracked. Closing this.