[Feature Request] Structured log for improved results analysis

[gitmadness]

💡 Idea

We want to enable some of the post run analysis thus we are looking for a structured JSON log which can be configured every instance of request + response where a bug is detected. These bugs could be HTTP result codes where the result codes can be configured, or issues detected by Checkers.

Just to clarify, the log does not need to have all the request + responses. Just the ones where RESTler deemed the response is a bug.

The following data should be included in each request + response in the log.

1. Client Timestamp - Time RESTler evaluated the response and determined there's a bug.
2. Request a. HTTP Version b. Method c. Uri d. Query String e. Headers f. Host g. Body
3. Response a. HTTP Version b. Status Code c. Status Description d. Headers e. Body
4. Bug info a. Source - Name of the Checker if found by a checker or “main”? b. Type of bug (e.g. UseAfterFree, HTTP 500, HTTP 401)

We do not have requirements for the log to be generated while a run is in progress. It can be generated after a run (e.g., a ResultAnalyzer command) and can be enabled from a config file or Restler.exe command line.

We do not have a requirement on whether there are multiple log files (e.g., one file for InvalidValueChecker and another for main) or just one log file as long as the information in items 1 through 4 are included (perhaps item 4 can be inferred from the file name).

Design Notes

Example of what we're looking for:

[
    {
        "request": {
            "version": "HTTP/1.1",
            "method": "POST",
            "uri": {
                "path": [
                    "",
                    "api",
                    "carts"
                ],
                "queryString": {}
            },
            "headers": {
                "Accept": "application/json",
                "Content-Length": "28",
                "Content-Type": "application/json",
                "Host": "localhost:7069",
                "User-Agent": "restler/9.2.2"
            },
            "body": {
                "name": "fuzzstring"
            }
        },
        "response": {
            "version": "HTTP/1.1",
            "statusCode": 201,
            "statusDescription": "Created",
            "headers": {
                "Content-Length": "56",
                "Content-Type": "application/json; charset=utf-8",
                "Date": "Thu, 14 Sep 2023 18:53:07 GMT",
                "Location": "carts/fuzzstring_cart",
                "Server": "Kestrel"
            },
            "body": {
                "identifier": "fuzzstring_cart",
                "userName": "fuzzstring"
            }
        },
        "Bug": {
            "source": "SomeChecker",
            "issueFound": "Cart should already exist. Not expecting 201"
        }
    },
    {
        "request": {
            "version": "HTTP/1.1",
            "method": "POST",
            "uri": {
                "path": [
                    "",
                    "api",
                    "carts"
                ],
                "queryString": {}
            },
            "headers": {
                "Accept": "application/json",
                "Content-Length": "21",
                "Content-Type": "application/json",
                "Host": "localhost:7069",
                "User-Agent": "restler/9.2.2"
            },
            "body": {
                "name": "fuzzstring"
            }
        },
        "response": {
            "version": "HTTP/1.1",
            "statusCode": 500,
            "statusDescription": "Internal Server Error",
            "headers": {
                "Content-Length": "0",
                "Date": "Thu, 14 Sep 2023 18:53:07 GMT",
                "Server": "Kestrel"
            }
        },
        "checker": {
            "name": "main_driver",
            "issueFound" : "HTTP500"
        }
    }
]