Timeout Exception Encountered with InvalidValueChecker During Fuzzing

[ImanSharaf]

Description

I've been using Restler for fuzzing tests and have encountered a recurring issue where I receive a timeout exception when applying the InvalidValueChecker. The error message I'm seeing is (restler-fuzzer/Fuzz/EngineStdOut.txt):

Exception 'Exceed Timeout' applying checker <checkers.invalid_value_checker.InvalidValueChecker object at 0x103396b50>
timeout reached

This issue has been consistently reproducible in my testing environment.

Steps to reproduce

I ran it against this repo With this command on a Mac:

bins/restler/Restler fuzz --grammar_file Compile/grammar.py --dictionary_file Compile/dict.json --settings Compile/engine_settings.json --time_budget 0.05 --target_ip 127.0.0.1 --target_port 2503 --no_ssl

[marina-p]

Hello @ImanSharaf,

The invalid value checker may take a long time depending on the specification and service (a large number of parameters, large schema, or long service response times). RESTler currently does not have a way to limit the budget of each checker, it will always try to do the same work regardless of the overall time budget, and will terminate when the time budget is exceeded. You can estimate the time it will take for all checkers to finish by running in 'fuzz-lean' mode, which will run the same checkers as 'fuzz' mode but only once per request type (endpoint+method), i.e. without generating many sequences as in 'fuzz' mode.

Thanks,

Marina

[marcindulak]

I assume the Exception 'Exceed Timeout' in Fuzz/EngineStdOut.txt is an indication of a "global" timeout for InvalidValueChecker, and not an indication that a timeout happened during some of individual HTTP requests.

As OP, I also see this exception

...
2024-01-21 12:30:53.490: Generation: 2 
Invalid value generator random seed: 1705840254.474184
Setting invalid value generator random seed: 12345
Exception 'Exceed Timeout' applying checker <checkers.invalid_value_checker.InvalidValueChecker object 
at 0xffff82603690>
timeout reached
2024-01-21 14:36:57.865: Terminating garbage collection. Waiting for max 300 seconds. 
Done.

and Fuzz/RestlerResults/experiment*/bug_buckets/bug_buckets.txt contains entries like the one below. They are "never" reproducible

InvalidValueChecker_timeout: 5
Total Buckets: 5
-------------
InvalidValueChecker_timeout - Unable to reproduce bug - InvalidValueChecker_timeout_1.replay.txt
Attempted to reproduce 2 time(s); Reproduced 0 time(s)
Hash: InvalidValueChecker_timeout_7a7fb1bfa0824003b891255be84a906876eb7ead
GET /api/v1/cities?fields=fuzzstring HTTP/1.1\r\nAccept: application/json\r\nHost: app\r\nauthenticatio
n_token_tag\r\n

Is there a way to increase the timeout for the individual HTTP requests for InvalidValueChecker_timeout?

[marcindulak]

This is the setting I've asked above: max_request_execution_time https://github.com/microsoft/restler-fuzzer/blob/main/docs/user-guide/SettingsFile.md#max_request_execution_time-float-default-120-max-600, and the default is 120 seconds, so it's probably a good idea to reduce it, and not increase as I initially thought.