random_seed option does not behave as expected

marina-p commented 1 year ago

Description

Currently, the random_seed engine setting is applied inconsistently across different checkers/fuzzing modes. This needs to be fixed, as follows:

1) Test mode Test mode should be deterministic by default, and use the same random seed. If users plug in a different random seed via the engine settings, this should be used consistently everywhere (remove any hard-coded constants).

2) Fuzz modes:

By default, fuzz modes should behave the same as 'Test' mode. This minimizes confusion when first setting up RESTler - a user will by default have deterministic runs to improve coverage in 'Test' mode and get clean on the 'Fuzz' modes.

However, a subsequent use case is to add CI/CD coverage, for which the user may want to vary the random seed on each run to get additional coverage over time. A new option should be provided in the engine settings for this (so the user does not need to generate the new seed themselves every time), for example generate_random_seed: true (this will override the random_seed option if specified in the settings file).

The random seed should be logged to the testing summary so it is easily discoverable.

If a user wants to generate the random seed themselves (for example, to deterministically re-run with a particular seed that found a bug), they could do so by plugging in the random_seed value from the testing summary in the engine settings.

Steps to reproduce

See above.

Expected results

No response

Actual results

No response

Environment details

No response

wilbaker commented 1 year ago

@marina-p thanks for filing this! I've not used the random_seed myself and I have a few questions about how this change would work:

It looks like random_seed currently configured on a per-checker basis in the settings files. Will that continue to be an option, or will all checkers share the same random_seed?
Will there need to be a new place to specify the random_seed that will be used outside of the checkers? I'm not sure where this would fit into the existing settings files. Example from requests.py: https://github.com/microsoft/restler-fuzzer/blob/28e0ec5b90f11a4649b242d7b2cbcf83b4694a28/restler/engine/core/requests.py#L9
Related: could/should this setting be supported as a command-line argument? I'm not familiar with the criteria for allowing a setting to be set on the command-line.

marina-p commented 1 year ago

Hello @wilbaker,

There should not be any changes to how the checkers are configured, only fixes. For example, if a checker random seed is already configurable, this change will not remove this feature. If there are places where a checker is using a static random seed value (like in your second bullet above), this will be modified to use the setting value.
There will be a random_seed property in the engine settings file.
We typically do not add engine settings to the command line, only to the engine settings file. The settings currently exposed on the command line are there for backwards compatibility.

Thanks,

Marina

microsoft / restler-fuzzer