Does fuzz-lightyear follow “Successful Request Sequences Only” strategy?

[AlexB1986]

Hi Team! Thank you for your efforts and open sourcing such interesting tool!

While playing with Fuzz-lightyear we faced with interesting issue – it seems that fuzz-lightyear does not follow “Successful Request Sequences Only” strategy. Could you please check our results?

Detailed information Let’s consider generating 2-length sequences (we used Mozilla Kinto as a target of testing). Before generating 2-length sequences last_result (based on 1-length sequences validation) is: Function _add_request_to_sequence generate a following sequence for validation: Before yield results into validate_sequence result.responses.responses structure is: Then we yield sequence into validate_sequence First request is OK: and we add data to responses.responses: Second request is not OK (404) and we pass to exception block and do not add any data to responses.responses.

When we obtain “result” in line, it looks like:

I.e., there are two requests and only one response (because second response was not successful). But result.is_successful() == true and we add that sequence (that is actually not a successful request sequence) to good_sequences and we will use it to construct 3-length sequences later.

To test our assumption we modify is_successful() implementation in the following way

return bool(self.responses) → return bool(self.responses) and bool(len(self.responses) == len(self.requests))

and made testing of two version (original and modified) against a same target (Mozilla Kinto) of testing with one seed and n=3. The results are:

• original version: 1424 failed, 260 passed, in 74.45 seconds
• modified version: 495 failed, 230 passed, in 36.78 seconds

That modification allows us to reduce running time and increase ratio of passed/(passed+falled).

[domanchi]

Hi @AlexB1986,

What a great idea to use Mozilla Kinto to test this! And thank you for your detailed bug report. I think you might be on to something -- let me verify on our end, and report back with results.

[domanchi]

I was able to reproduce this in a test case. It looks like we do indeed respect "Successful Request Sequences Only" (that is, a failed request will quit the sequence early), however, due to the bug you correctly identified, a failed sequence isn't accurately reflected in FuzzingResult(...).is_successful().

As a result, this will cause incorrect test reporting, as well as redundant sequences generated.

I'll cut a PR out to address this.

[domanchi]

Closing issue, as fix has been merged.