Tutorial - "Request coverage (successful / total): 6/6"

[thesammiller]

Description

I am working through the tutorial, and my results are not lining up with the tutorial documentation.

Steps to reproduce

Fresh install of restler-fuzzer. Run the demo server. Follow the tutorial steps.

Expected results

Expect demo results to match the tutorial.

Actual results

When I run the test, I get the output:

Starting task Test...
Using python: 'python3' (Python 3.9.2)
Request coverage (successful / total): 6 / 6
Attempted requests: 6 / 6
No bugs were found.
Task Test succeeded.
Collecting logs...

The first entry in main.txt is the expected invalid request:

Rendering request 0 from scratch

2022-12-05 21:39:38.755: Request 0
2022-12-05 21:39:38.755: Endpoint - /api/blog/posts
2022-12-05 21:39:38.755: Hex Def - 1950cbddab7726489624c3d346d3426561c921ad
2022-12-05 21:39:38.755: Sequence length that satisfies dependencies: 1
2022-12-05 21:39:38.755: Rendering VALID
        - restler_static_string: 'GET '
        - restler_static_string: ''
        - restler_static_string: '/'
        - restler_static_string: 'api'
        - restler_static_string: '/'
        - restler_static_string: 'blog'
        - restler_static_string: '/'
        - restler_static_string: 'posts'
        - restler_static_string: '?'
        - restler_static_string: 'page='
        - restler_fuzzable_int: '1'
        - restler_static_string: '&'
        - restler_static_string: 'per_page='
        - restler_fuzzable_int: '1'
        - restler_static_string: ' HTTP/1.1\r\n'
        - restler_static_string: 'Accept: application/json\r\n'
        - restler_static_string: 'Host: localhost:8888\r\n'
        - restler_refreshable_authentication_token: 'authentication_token_tag'
        - restler_static_string: '\r\n'

The value for restler_fuzzable_int is not 2:

"restler_fuzzable_int": [
    "1"
  ],

Continuing the tutorial, the contents of bug_buckets is also different:

-rw-r--r-- 1 smiller smiller  834 Dec  5 21:44 bug_buckets.json
-rw-r--r-- 1 smiller smiller 3957 Dec  5 21:44 bug_buckets.txt
-rw-r--r-- 1 smiller smiller 1676 Dec  5 21:44 InvalidDynamicObjectChecker_20x_1.txt
-rw-r--r-- 1 smiller smiller 1729 Dec  5 21:44 InvalidDynamicObjectChecker_20x_2.txt
-rw-r--r-- 1 smiller smiller 1247 Dec  5 21:43 InvalidValueChecker_500_1.txt
-rw-r--r-- 1 smiller smiller 1769 Dec  5 21:44 PayloadBodyChecker_500_1.txt
-rw-r--r-- 1 smiller smiller 1722 Dec  5 21:44 PayloadBodyChecker_500_2.txt
-rw-r--r-- 1 smiller smiller 1860 Dec  5 21:44 UseAfterFreeChecker_20x_1.txt

It seems like there is an additional InvalidValueChecker_500: 1 with the following text in InvalidValueChecker_500_1.txt:

################################################################################
 InvalidValueChecker_500

 Hash: InvalidValueChecker_500_5f9bb084cbb3a2529b26bf690142685a65bd355b

 To attempt to reproduce this bug using restler, run restler with the command
 line option of --replay_log <path_to_this_log>.
 If an authentication token is required, you must also specify the token_refresh_cmd.

 This log may contain specific values for IDs or names that were generated
 during fuzzing, using the fuzzing dictionary. Such names will be re-played
 without modification. You must update the replay log manually with any changes
 required to execute the requests in your environment (for example, replacing
 pre-created account, subscription, or other resource IDs, as needed).
################################################################################

-> GET /api/blog/posts?page=1&per_page=371981492432636949 HTTP/1.1\r\nAccept: application/json\r\nHost: localhost:8888\r\n\r\n
! producer_timing_delay 0
! max_async_wait_time 20
PREVIOUS RESPONSE: 'HTTP/1.1 500 Internal Server Error\r\ndate: Tue, 06 Dec 2022 02:43:50 GMT\r\nserver: uvicorn\r\ncontent-length: 34\r\ncontent-type: application/json\r\n\r\n{"detail":"per_page is too large"}'

Environment details

Debian GNU/Linux 11 (bullseye) Python 3.9.2 Dotnet 6.0.403 Restler-Fuzzer main branch

[marina-p]

Hello @thesammiller,

Thank you for opening this issue. The tutorial is out-of-date and the above RESTler results in your description are correct for the current demo_server. We will get the tutorial updated (please do not close this issue).

Just FYI, the differences you noted are both due to the demo_server having been rewritten recently, and new RESTler enhancements (e.g. new invalid value checker).

Thanks,

Marina

[thesammiller]

Thank you! I thought it might be something like some RESTler improvements.

Since this is labeled good first issue, would it be helpful for me to put together a PR which updates the file? I noticed some differences in the fuzz as well, but is there anything else that would need to be updated?

[marina-p]

Yes, that would be great if you could update the tutorial.

There are no other files that need to be updated.
For double checking the results of test/fuzz/fuzz-lean, you can see the hard-coded expected results in this file (this is a test baseline for a test that runs in the RESTler CI): https://github.com/microsoft/restler-fuzzer/blob/main/restler/end_to_end_tests/test_quick_start.py.

Thanks,

Marina