There have been previous attempts at fuzz testing the Chapel compiler, to some success (see https://github.com/chapel-lang/chapel/issues/13097 and https://github.com/chapel-lang/chapel/issues/9987). These attempts have a caught various internal issues with the compiler. I think we should integrate proper fuzz testing as a part of our testing suite and I could find no other issue on this topic.
What is fuzzing
Fuzzing is an automated way of generating inputs for a system to test for unexpected inputs. For Chapel's use case, this would look like Chapel source code that might cause an internal error.
These codes were generated with a simple rule-set that transformed some basic Chapel design patterns (programs) by inserting or substituting 1- or 2-length repeated sequences of symbol characters (ASCII codes in the range: 0x21-2f, 31-40, 5b-60, 7b-7e). Runtime took several hours to find these cases. The Chapel team could probably do a better job (more exhaustive set of inputs, more patterns); this is more of a feasibility demonstration.
Both of the previous attempts used mutation fuzzing, taking existing good programs and tweaking them. I think we could probably find a third-party fuzzing tool that does a good job of mutation based fuzzing, and just point it to our existing test repo. https://www.fuzzingbook.org/html/MutationFuzzer.html seems to have some good resources for this.
DanilaFe
commented
2 months ago
There have been previous attempts at fuzz testing the Chapel compiler, to some success (see https://github.com/chapel-lang/chapel/issues/13097 and https://github.com/chapel-lang/chapel/issues/9987). These attempts have a caught various internal issues with the compiler. I think we should integrate proper fuzz testing as a part of our testing suite and I could find no other issue on this topic.
What is fuzzing
Fuzzing is an automated way of generating inputs for a system to test for unexpected inputs. For Chapel's use case, this would look like Chapel source code that might cause an internal error.
Previous attempts
In https://github.com/chapel-lang/chapel/issues/9987, no external fuzzing tool was used.
In https://github.com/chapel-lang/chapel/issues/13097, an attempt was made to use AFL, but that was slow. Instead, a distributed Chapel program was used to make small modifications to existing Chapel programs.
Proposals
Both of the previous attempts used mutation fuzzing, taking existing good programs and tweaking them. I think we could probably find a third-party fuzzing tool that does a good job of mutation based fuzzing, and just point it to our existing test repo. https://www.fuzzingbook.org/html/MutationFuzzer.html seems to have some good resources for this.
Another approach would be to use the existing grammar to drive a fuzzer. https://www.fuzzingbook.org/html/Grammars.html has some resources for this.