search

microsoft / restler-fuzzer

RESTler is the first stateful REST API fuzzing tool for automatically testing cloud services through their REST APIs and finding security and reliability bugs in these services.
MIT License
2.6k stars 300 forks source link

Add number of requests sorted by HTTP status code in testing summary #204

Open Mafti opened 3 years ago

Mafti commented 3 years ago

Hi,

as a follow-up of #199 I would like to see the total execution + failed in testing summary. Instead of having to sum them up myself and combining the 2 jsons :)

{ "requests_summary": { "executed": 400, //i.e. 200 "failed": 100, //not-200 },

PatGod commented 3 years ago

This question/request is already addressed in #199 - please see that thread for further discussion.

PatGod commented 3 years ago

Note that testing_summary.json already contains the information you want - please see #199 (no need to combine 2 json files).

PatGod commented 3 years ago

[Re-opening and clarifying]

Suggestion: in testing_summary.json, add a new section reporting the number of requests that triggered specific HTTP status codes.

Example:

"total_request_sent_per_status_code": {
  "200": 10,
  "201": 3,
  "400": 100,
  "403": 2,
  "500": 4,
  "503: 1
}

This information is already produced by resultsAnalyzer, but having it (incrementally computed and) included in testing_summary.json may be simpler. (Plus resultsAnalyzer is sometimes flaky and can be expensive for very long RESTler runs.)

marina-p commented 3 years ago

Yep, we should add these.

(Separately, we should clarify in the documentation why results analyzer may in rare cases produce different numbers...)

mbiuki commented 3 years ago

Hi @marina-p and @PatGod,

I have a suggestion here. Is there a possibility that similar to Schemathesis , RESTler could also report discrepancies between the open API spec and the query responses? For example, if the server sends 403 and that is not defined in the spec for a particular endpoint, RESTler should take note of this inconsistency.

(Separately, we should clarify in the documentation why results analyzer may in rare cases produce different numbers...)

This deals with the reproducibility of results. Some of the issues that are reported, but not reproducible later, are the problem of timing with the server, I find.