fuzz test: one query parameter request happens 10k times more than others

[dfranklin-at-bg]

My API specification includes a GET request with query parameters (GET /api/v1/endpoint?floor=param). I have set up dict.json with restler_custom_payload and have specified the query parameter with an array of both valid and bogus options. (example "floor": ["floor2", "floor1", "not_a_floor", "floor3"]). I also have a number of other endpoints in the API with similarly-defined query parameters. There are 30 requests in total, some of which have multiple query parameters.

When I run fuzz testing for 30 minutes, there will be approx 40,000 requests for the first parameter (floor2) and 30 requests for each of the other parameters - but only for this one particular endpoint.

Requests for other endpoints are evenly distributed among the various query parameter lists. It's just this one particular endpoint that is unbalanced in terms of requests.

It so happens that this endpoint is the first one alphabetically and is listed at the top of Compile/dependencies_debug.json and Compile/dependencies.json. There is nothing else unique or strange about this endpoint in the API spec; it's like all the others.

If I change the ordering of the query values in dict.json, for instance, put "floor3" first and "floor2" last in the list - then there will be 40K requests for floor3, and 30 for the other values in the list.

[dfranklin-at-bg]

For the first generation all 30 requests are rendered. For the second and subsequent generations, the unbalanced endpoint request for the first parameter gets sent, then all other endpoints are skipped.

Generation-2: Rendering Sequence-5

        Request: 1 (Current combination: 1 / 4)
                - restler_static_string: 'GET '
                - restler_static_string: '/api/v1'
                - restler_static_string: '/'
                - restler_static_string: 'endpoint'
                - restler_static_string: '?'
                - restler_static_string: 'floor='
                + restler_custom_payload: [floor2, floor1, ...]
                - restler_static_string: ' HTTP/1.1\r\n'
                - restler_static_string: 'Accept: application/json\r\n'
                - restler_static_string: '\r\n'

        Request: 2 (Remaining candidate combinations: 36)
        Request hash: 6950648bfcfef17042e92ede3f8dd05f496d05bf

                - restler_static_string: 'POST '
                - restler_static_string: '/api/v1'
                - restler_static_string: '/'
                - restler_static_string: 'elements'
                - restler_static_string: '/'
                + restler_custom_payload: [24001882, 24001132, ...]
                - restler_static_string: '?'
                - restler_static_string: 'floor='
                + restler_custom_payload: [floor2, floor1, ...]
                - restler_static_string: ' HTTP/1.1\r\n'
                - restler_static_string: 'Accept: application/json\r\n'
                - restler_static_string: '\r\n'

2022-06-22 15:23:14.135: Skipping rendering: 0
2022-06-22 15:23:14.136: Skipping rendering: 1
2022-06-22 15:23:14.138: Skipping rendering: 2
2022-06-22 15:23:14.139: Skipping rendering: 3
2022-06-22 15:23:14.140: Skipping rendering: 4
2022-06-22 15:23:14.142: Skipping rendering: 5
2022-06-22 15:23:14.143: Skipping rendering: 6
2022-06-22 15:23:14.145: Skipping rendering: 7
2022-06-22 15:23:14.146: Skipping rendering: 8
2022-06-22 15:23:14.148: Skipping rendering: 9
2022-06-22 15:23:14.149: Skipping rendering: 10
2022-06-22 15:23:14.151: Skipping rendering: 11
2022-06-22 15:23:14.152: Skipping rendering: 12
2022-06-22 15:23:14.153: Skipping rendering: 13
2022-06-22 15:23:14.156: Skipping rendering: 14
2022-06-22 15:23:14.157: Skipping rendering: 15
-- SNIPPED --

[marina-p]

Hello @dfranklin-at-bg,

The strange behavior you are seeing is because RESTler has a default mode of bfs-fast fuzzing mode [1]. (Unfortunately, the documentation was not updated and still says bfs is the default. I will get the docs updated shortly; for reference the code update was in #272).

You may already be aware of this, but just for completeness: in bfs-fast mode, let's say you are in generation-3 (fuzzing sequences of length 3), and let's say the request you are fuzzing does not have any pre-requisites. RESTler will use just ONE request from each of generation-1 and generation-2 to fuzz the request. RESTler does not randomize which requests are chosen in gen-1 and gen-2 - as you observed above, it simply picks the first one every time.

You can override the fuzzing_mode in the engine settings (for example, you can set it to bfs to test all combinations in every generation).

Thanks,

Marina

[dfranklin-at-bg]

Thank you @marina-p it looks like bfs is just what I want.