microsoft / restler-fuzzer

RESTler is the first stateful REST API fuzzing tool for automatically testing cloud services through their REST APIs and finding security and reliability bugs in these services.
MIT License
2.59k stars 296 forks source link

[Feature Request] Report the first failing request in speccov.json #637

Closed mikekistler closed 1 year ago

mikekistler commented 2 years ago

💡 Idea

When attempting to diagnose failures in a RESTler run, I am using the speccov.json file to get a concise list of the failing requests -- like this:

jq '.[] | select(.valid == 0) | .sample_request // empty ' speccov.json | less

It looks to me like this file contains only the last attempt for this endpoint (which has api-version=fuzzstring) and not the first (that uses all the example values). I would much prefer that speccov.json have the first failing request, particularly in the case where this uses example values.

More detail/background in #633.

Design Notes

No response