eykrehbein / strest

⚡️ CI-ready tests for REST APIs configured in YAML
MIT License
1.74k stars 59 forks source link

Improve the JSON output to hold validation results #142

Open maikeffi opened 5 years ago

maikeffi commented 5 years ago

The Json output generated do not have status on Test Thanks for this awesome tool . When performing a test it is always nice to have a place where the test results , are stored It would be nice have a section in json where status of test as is PASS , FAIL , SKIPPED along with reason is maintained !

version: 2

requests:
    getDummyData:
      request:
        url: https://reqres.in/api/users/2
        method: GET
      validate:
        - jsonpath: status
          expect: 200  
        - jsonpath: content.data.email
          expect: janet.weaver@reqres               
      log: true   

Command used to execute the test strest 0-sampleTest.strest.yml -n -s dummy_test.json

Output Json

{
  "getDummyData": {
    "status": 200,
    "statusText": "OK",
    "headers": {
      "date": "Wed, 26 Jun 2019 15:37:00 GMT",
      "content-type": "application/json; charset=utf-8",
      "content-length": "170",
      "connection": "close",
      "set-cookie": [
        "__cfduid=d7b5472470e700bae60fa035ac616659f1561563420; expires=Thu, 25-Jun-20 15:37:00 GMT; path=/; domain=.reqres.in; HttpOnly; Secure"
      ],
      "x-powered-by": "Express",
      "access-control-allow-origin": "*",
      "etag": "W/\"aa-yZW/45DWGt/1ri05OLnMt/FJ3RY\"",
      "via": "1.1 vegur",
      "cf-cache-status": "HIT",
      "age": "6363",
      "expires": "Wed, 26 Jun 2019 19:37:00 GMT",
      "cache-control": "public, max-age=14400",
      "accept-ranges": "bytes",
      "expect-ct": "max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"",
      "server": "cloudflare",
      "cf-ray": "4ed042942f5f3498-LHR"
    },
    "content": {
      "data": {
        "id": 2,
        "email": "janet.weaver@reqres.in",
        "first_name": "Janet",
        "last_name": "Weaver",
        "avatar": "https://s3.amazonaws.com/uifaces/faces/twitter/josephstein/128.jpg"
      }
    }
  }
}

Solution I'd like The above test is actually a failed test . when run from the console we can see that it has thrown a red text as below .

✖ Testing getDummyData failed (1.234s)
[ Validation ] The JSON response value should have been janet.weaver@reqres but instead it was janet.weaver@reqres.in

But in the future when we go back and take a look at the json it does not provide an insight on if its a success / failed test case !

Example Output: Showcasing only the desired section which can be appended to the existing json Test result for each test

{
"content": {
      "data": {
        "id": 2,
        "email": "janet.weaver@reqres.in",
        "first_name": "Janet",
        "last_name": "Weaver",
        "avatar": "https://s3.amazonaws.com/uifaces/faces/twitter/josephstein/128.jpg"
      }
    },
"result":{
   "status": "failed",
"reason": "The JSON response value should have been janet.weaver@reqres but instead it was janet.weaver@reqres.in"
    }
}

Test summary at the end of a suite below data is fictional !

{
"summary":{
   "total": 10,
   "pass": 5,
   "fail": 3,
  "skipped":2,
 "time": "1.144s"
     }
}