microsoft / restler-fuzzer

RESTler is the first stateful REST API fuzzing tool for automatically testing cloud services through their REST APIs and finding security and reliability bugs in these services.
MIT License
2.52k stars 283 forks source link

Enable structured logging for RESTler request sequences #829

Closed marina-p closed 7 months ago

marina-p commented 8 months ago

This change adds structured logging for all of the network traffic that is currently logged to the network.*.txt log. The data is written to a 'trace database'. The default format is newline-delimited json, but custom logging formats are supported via a module specified in the engine settings (similar to checkers).

Below is an example for demo_server:

{"sent_timestamp": "2023-11-02T10:10:59.544824+00:00", "received_timestamp": null, "request": "GET /api/blog/posts?page=1&per_page=1 HTTP/1.1\r\nAccept: application/json\r\nHost: localhost\r\nContent-Length: 0\r\nx-restler-sequence-id: fe2172ab-151c-419f-b918-f3e7483b3230\r\n\r\n", "response": null, "request_json": "null", "response_json": null, "tags": {"request_id": "27f9653431313fdc3fecc4a890b72b80b4ce1e59", "sequence_id": "fe2172ab-151c-419f-b918-f3e7483b3230", "combination_id": "1950cbddab7726489624c3d346d3426561c921ad_1", "hex_definition": "6daf8d22c7a6b3472fc83c9f08f290b3507c3ff3", "origin": "main_driver"}}

{"sent_timestamp": null, "received_timestamp": "2023-11-02T10:10:59.597613+00:00", "request": null, "response": "HTTP/1.1 400 Bad Request\r\ndate: Thu, 02 Nov 2023 10:10:58 GMT\r\nserver: uvicorn\r\ncontent-length: 41\r\ncontent-type: application/json\r\n\r\n{\"detail\":\"per_page must be at least 2.\"}", "request_json": null, "response_json": null, "tags": {"request_id": "27f9653431313fdc3fecc4a890b72b80b4ce1e59", "sequence_id": "fe2172ab-151c-419f-b918-f3e7483b3230", "combination_id": "1950cbddab7726489624c3d346d3426561c921ad_1", "hex_definition": "6daf8d22c7a6b3472fc83c9f08f290b3507c3ff3", "origin": "main_driver"}}

Testing:

marina-p commented 8 months ago

Just fixed a few issues - please get the latest iteration for testing


In reply to: 1711381128

wilbaker commented 8 months ago

Just fixed a few issues - please get the latest iteration for testing

In reply to: 1711381128

Thanks! I will give this a try.

marina-p commented 8 months ago

TODO: test replay mode with trace DB & update replay sequence to generate a unique sequence ID. #Resolved

marina-p commented 7 months ago

@wilbaker updated the PR with the origin and checker changes.


In reply to: 1792953323