Add support for token dictionaries

[disconnect3d]

Hey,

We have had an internship project in Trail of Bits to improve go-fuzz recently which was done by @vfsrfs.

We are aware of the ongoing official work on native fuzzing support but since we still rely on go-fuzz we went ahead to improve its pain points and so that's why we propose this PR. Feel free to drop it if you feel it is too much or you do not want to introduce any changes in go-fuzz.

Below I am pasting the description from the original PR merged to our fork of go-fuzz (https://github.com/trailofbits/go-fuzz/pull/2).

---

This PR adds support for dictionaries containing interesting keywords (tokens) that are useful for the mutation of inputs while fuzzing, particularly, when fuzzing syntax-aware programs (#174). This modification allows to provide the -dict flag to go-fuzz, so that the user can provide a dictionary file with useful tokens for the fuzzing campaign. E.g.:

-dict /path/dictionary.dict

The tokens parsed from the dictionary are stored in ROData.strLits, as those are the string literals that are used by the mutator engine when generating new fuzzing inputs.

The dictionary format that is accepted by the -dict flag is the same that is used by AFL/Libfuzzer (see https://github.com/google/AFL/tree/master/dictionaries).

This dictionary format defines that there is one token per line. Every line consists of a name followed by an equal sign and the token in quotes (e.g. name=”token”). It is also possible to define binary sequences by providing the values in hex (e.g. \xNN) within the token. To insert a backslash or a double quote within the token, it has to be escaped using a backslash (e.g. \\ or \”). \n and \t are recognized as well, since they might be useful for text-based protocols. Other problematic characters can be added by providing its hex value.

To make this implementation fully compatible with AFL/Libfuzzer’s dictionaries, token levels are supported. A level can be appended to every token, by appending @<num> to the keyword, e.g. keyword@1=”token”

These tokens will be loaded only, if the dictionary level is equal to or greater than the specified number. The default dictionary level is 0, but it can be increased by appending @<num> to the dictionary path. E.g.:

-dict /path/dictionary.dict@1

[josharian]

@thepudds I'm going to leave this to you to review, if you want.

Note that there is a similar outstanding PR at https://github.com/dvyukov/go-fuzz/pull/315. It might be worth looking at it, and at the comments there.

[CityOfLight77]

@thepudds @dvyukov