Open nathanielc opened 5 years ago
Yes, public repository at https://github.com/mark-rushakoff/flux-fuzz.
Anyone who picks this up should plan on doing a short call with me to walk through the idiosyncrasies and arts of fuzzing. It will be a lot easier as a conversation than as a writeup.
I talked with @mark-rushakoff and here's a general outline:
I have made an initial fuzzer that's easy to run using Docker at https://github.com/influxdata/flux-fuzz.
For the additional work, we should spec that out and talk about it at standup. I am running it locally to get some generated tests locally for now. Running it on a server wouldn't be too hard, but it requires a few things:
After using this for a few times while developing the parser, here's a revised todo of what I would find useful and feasible.
While the continuous fuzzer above might be potentially useful, I don't think it would actually help very much. Most of the time when I am working with the fuzzer, I'm looking for crashes. Most of the crashes are pretty straightforward and are found within a few minutes. I've run the fuzzer for 10 minutes and 30 minutes and when the number of crashes is zero, it usually stays zero. In the abstract, running it all the time could be a benefit, but I'm not sure it's practical especially considering that the parser code isn't the most complex.
Instead, here's what I'm thinking.
The process of fuzzing will run on Jenkins. Circle's timeout will likely stop us from using that and, since it isn't a main part of CI, I don't think that we need to worry about contributors seeing the results.
The actual fuzzer would essentially do the following.
flux-fuzz
. Continue running for 10 minutes.I'm a bit unclear on the last few steps which is mostly ensuring that we continue to build a corpus. My preference isn't to run it against master to build the corpus since each of the PRs will help build it up. But, I'm also not really sure how to get the artifacts over. So maybe we just rerun the fuzzer on master and only commit those? That isn't my preference though.
This issue has had no recent activity and will be closed soon.
@mark-rushakoff I know you did something around fuzzing Flux a while back, Is there anything public we can reference to incorporate?