Closed AlexB1986 closed 3 years ago
Hi @AlexB1986,
What a great idea to use Mozilla Kinto to test this! And thank you for your detailed bug report. I think you might be on to something -- let me verify on our end, and report back with results.
I was able to reproduce this in a test case. It looks like we do indeed respect "Successful Request Sequences Only" (that is, a failed request will quit the sequence early), however, due to the bug you correctly identified, a failed sequence isn't accurately reflected in FuzzingResult(...).is_successful()
.
As a result, this will cause incorrect test reporting, as well as redundant sequences generated.
I'll cut a PR out to address this.
Closing issue, as fix has been merged.
Hi Team! Thank you for your efforts and open sourcing such interesting tool!
While playing with Fuzz-lightyear we faced with interesting issue – it seems that fuzz-lightyear does not follow “Successful Request Sequences Only” strategy. Could you please check our results?
Detailed information Let’s consider generating 2-length sequences (we used Mozilla Kinto as a target of testing). Before generating 2-length sequences last_result (based on 1-length sequences validation) is:
Function _add_request_to_sequence generate a following sequence for validation:
Before yield results into validate_sequence result.responses.responses structure is:
Then we yield sequence into validate_sequence
First request is OK:
and we add data to responses.responses:
Second request is not OK (404)
and we pass to exception block and do not add any data to responses.responses.
When we obtain “result” in line, it looks like:![image](https://user-images.githubusercontent.com/22557771/93193309-5b08c580-f74f-11ea-914e-42d5bfa5c219.png)
I.e., there are two requests and only one response (because second response was not successful). But result.is_successful() == true and we add that sequence (that is actually not a successful request sequence) to good_sequences and we will use it to construct 3-length sequences later.
To test our assumption we modify is_successful() implementation in the following way
return bool(self.responses) → return bool(self.responses) and bool(len(self.responses) == len(self.requests))
and made testing of two version (original and modified) against a same target (Mozilla Kinto) of testing with one seed and n=3. The results are:
That modification allows us to reduce running time and increase ratio of passed/(passed+falled).