search

google / clusterfuzz

Scalable fuzzing infrastructure.
https://google.github.io/clusterfuzz
Apache License 2.0
5.29k stars 554 forks source link

Implement grouping test cases by culprit commit #3056

Open mi-ac opened 1 year ago

mi-ac commented 1 year ago

There are currently three grouper options: _group_testcases_with_similar_states(testcase_map) _group_testcases_with_same_issues(testcase_map) _group_testcases_based_on_variants(testcase_map)

Additionally to those, for some projects (with exact revision resolution) like V8, it would be beneficial to group also by regression range (when it points to a single commit). This would prevent bug floods of correctness fuzzing like https://crbug.com/1443672 and might be an alternative to a generic issue-filing limit (https://github.com/google/clusterfuzz/issues/2706).

There might be jobs or crash types though for which such grouping renders a problem? Haven't thought it through yet. Hence, such a feature might need to be "opt-in" per job or have an exclude list for some jobs or crash types. E.g. a commit that enables a new sanitizer option might have several distinct bugs bisecting back to it...

For V8 correctness fuzzing though, it would certainly be a big plus to group by suspected commit.

jonathanmetzman commented 1 year ago

I strongly support this idea, and have had similar thoughts on this for a while.