Errors running fuzzcheck

[teymour-aldridge]

I hope you're well :)

(relevant code at the bottom of this issue)

I'm trying to test a small parser with fuzzcheck, and I am running into a couple of errors.

The first error I found was (everything running on MacOS Catalina, x86_64):

teymouraldridge@Teymours-MBP tmp-XVrRgj % cargo +nightly fuzzcheck fuzz::test_fuzz_parser fuzz
error: failed to run `rustc` to learn about target-specific information

Caused by:
  process didn't exit successfully: `rustc - --crate-name ___ --print=file-names -Zinstrument-coverage=except-unused-functions -Zno-profiler-runtime --cfg fuzzing --target x86_64-apple-darwin --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=cfg` (exit status: 1)
  --- stderr
  error: unknown debugging option: `no-profiler-runtime`

I updated everything (rust toolchain – so running the latest nightly – and fuzzcheck) and that issue went away (yay!) but instead this issue appears:

teymouraldridge@Teymours-MBP tmp-XVrRgj % cargo +nightly fuzzcheck fuzz::test_fuzz_parser fuzz
error: no library targets found in package `tmp-xvrrgj`

My code is as follows:

use std::mem;

fn main() {}

fn exec(mut instruction: String) -> Vec<u8> {
    let shuffles = Shuffles::parse(&mut instruction);
    let mut vec: Vec<u8> = vec![1, 2, 3, 4, 5, 6, 7, 8];
    shuffles.execute(&mut vec)
}

/// Formal grammar:
/// ```
/// instructions ::= (instruction)+
/// instruction ::= 'b'
///                 'i'
///                 'o'
///                 '(' + instruction+ + ')'
///                 '0'..='9' + instruction
/// ```

#[derive(Debug)]
struct Shuffles(Vec<Shuffle>);

impl Shuffles {
    fn parse(input: &mut String) -> Self {
        let mut output = vec![];

        loop {
            if input.starts_with(')') || input.is_empty() {
                return Self(output);
            }
            output.push(Shuffle::parse(input))
        }
    }

    fn execute(&self, sequence: &mut Vec<u8>) -> Vec<u8> {
        for shuffle in &self.0 {
            shuffle.execute(sequence);
        }
        sequence.to_vec()
    }
}

#[derive(Debug)]
enum Shuffle {
    Break,
    InRiffle,
    OutRiffle,
    Group(Shuffles),
    Repeat(u8, Box<Shuffle>),
}

impl Shuffle {
    fn execute(&self, sequence: &mut Vec<u8>) {
        match self {
            Shuffle::Break => {
                let character = sequence.remove(0);
                sequence.push(character);
            }
            Shuffle::InRiffle => {
                let mut new = vec![];
                let top_half = &sequence[0..sequence.len() / 2];
                let bottom_half = &sequence[sequence.len() / 2..];
                assert_eq!(top_half.len(), bottom_half.len());
                for (top, bottom) in top_half.into_iter().zip(bottom_half) {
                    new.push(*bottom);
                    new.push(*top);
                }
                mem::swap(&mut new, sequence);
            }
            Shuffle::OutRiffle => {
                let mut new = vec![];
                let top_half = &sequence[0..sequence.len() / 2];
                let bottom_half = &sequence[sequence.len() / 2..];
                assert_eq!(top_half.len(), bottom_half.len());
                for (top, bottom) in top_half.into_iter().zip(bottom_half) {
                    new.push(*top);
                    new.push(*bottom);
                }
                mem::swap(&mut new, sequence);
            }
            Shuffle::Group(shuffles) => {
                shuffles.execute(sequence);
            }
            Shuffle::Repeat(n, shuffle) => {
                for _ in 0..*n {
                    shuffle.execute(sequence)
                }
            }
        }
    }

    fn parse(input: &mut String) -> Self {
        let character = input.remove(0);
        match character {
            'b' => Shuffle::Break,
            'i' => Shuffle::InRiffle,
            'o' => Shuffle::OutRiffle,
            '(' => {
                let shuffles = Shuffles::parse(input);
                assert_eq!(input.remove(0), ')');
                Shuffle::Group(shuffles)
            }
            '0'..='9' => {
                let mut string = String::from(character);
                loop {
                    let next = input.chars().next().unwrap();
                    match next {
                        '1'..='9' => {
                            string.push(next);
                            input.remove(0);
                        }
                        _ => {
                            break;
                        }
                    }
                }
                let n = string.parse::<u8>().unwrap();
                let shuffle = Shuffle::parse(input);
                Shuffle::Repeat(n, Box::new(shuffle))
            }
            _ => panic!(),
        }
    }
}

#[cfg(test)]
mod fuzz {
    use fuzzcheck::{
        alternation, concatenation, literal, mutators::grammar::grammar_based_string_mutator,
        recurse, recursive, repetition, FuzzerBuilder, SerdeSerializer,
    };

    use crate::Shuffles;

    #[test]
    fn test_fuzz_parser() {
        let grammar = repetition! {
            recursive! {
                g in alternation! {
                    literal!('b'),
                    literal!('i'),
                    literal!('o'),
                    concatenation! {
                        literal!('('),
                        recurse!(g),
                        literal!(')')
                    },
                    concatenation! {
                        repetition!(literal!('0'..='9'), 0..1000),
                        recurse!(g)
                    }
                }
            },
            0..1000
        };

        fn test(input: &String) {
            Shuffles::parse(&mut input.clone());
        }

        FuzzerBuilder::test(test)
            .mutator(grammar_based_string_mutator(grammar))
            .serializer(SerdeSerializer::default())
            .arguments_from_cargo_fuzzcheck()
            .observe_only_files_from_current_dir()
            .launch()
    }
}

[loiclec]

Hi Teymour! I'm terribly sorry, again, for the bugs. This one should be easy to fix, I'll get back to you later today or tomorrow :)

[loiclec]

in the meantime, if you move the tests to a lib.rs file instead of main.rs, it might fix it. There's an option in the latest cargo-fuzzcheck to fuzz-test non-library targets, but I haven't released that yet I think

[loiclec]

The previous version of cargo fuzzcheck would always call

cargo test --lib ... # other arguments

which would only compile the crate’s library. But if your fuzz tests are in an executable (main.rs) or integration tests (inside tests folder), then that wouldn't work.

In the newest cargo fuzzcheck (version 0.8.0, which I've just released), the command that is called instead is simply cargo test, which compiles everything. That's actually not great either, since it might compile other stuff that doesn't depend on fuzzcheck and then will complain that there is a missing runtime for -Zinstrument-coverage. I should come up with a proper solution for it, but for now, you can use a new option: --cargo-args which passes extra arbitrary options to the cargo test invocation.

So you can do:

cargo fuzzcheck tests::fuzz fuzz --cargo-args="--lib"

which will run cargo test --lib ... to specifically compile/test only the library target. Or:

cargo fuzzcheck tests::fuzz fuzz --cargo-args="--bins"

which will run cargo test --bins ... to compile/test only the executables.

[loiclec]

As an aside, I see that you are using grammar-based string mutators, which is nice! But you should be aware of a few drawbacks:

• you should always use finite ranges in repetition! because of a bug I'm hoping to fix soon
• if your grammar is overly ambiguous, left-recursive, or too complex, the fuzzer might become significantly slower or even crash. That's because fuzzcheck needs to parse each new generated value, but its parser is a basic recursive descent parser, which is prone to excessive backtracking.
• when using cargo fuzzcheck tests::fuzz tmin ... to minify a failing test case, you might enter an infinite loop where the fuzzer thinks it's making progress but is actually stuck. It will still minify the input, but not as much or as quickly as it should.

I've started writing a guide at https://fuzzcheck.neocities.org , which is in a pretty rough shape at the moment. But it already contains a section on how to write a grammar-based fuzz test on pulldown-cmark. Because the grammar for that test is too ambiguous, I choose instead to generate values of type (AST, String). That way, fuzzcheck’s parser is not involved and the grammar can be as complicated as you want.

And finally, I have unfortunately changed the API to launch a fuzz test again. Look at the final code at https://fuzzcheck.neocities.org/example2_fuzzing.html (or other examples) to see how to use it.

[teymour-aldridge]

No worries about the bugs – thank you for all your help resolving them :)

It works!

running 1 test
test fuzz::test_fuzz_parser ... START
NEW     1 high_cov_hits(1 sum: 654) uniq_cov(0 cov: 0.00% cplx: 0.00)   artifacts(0) iter/s: 273
NEW     1 high_cov_hits(0 sum: 0) uniq_cov(1 cov: 63.16% cplx: 676.00)   artifacts(0) iter/s: 227
NEW     1 high_cov_hits(0 sum: 0) uniq_cov(0 cov: 0.00% cplx: 0.00)   artifacts(0) iter/s: 213
NEW     1 high_cov_hits(0 sum: 0) uniq_cov(0 cov: 0.00% cplx: 0.00)   artifacts(0) iter/s: 197
RPLC 1  2 high_cov_hits(1 sum: 2146) uniq_cov(1 cov: 63.16% cplx: 676.00)   artifacts(0) iter/s: 102
RPLC 1  2 high_cov_hits(1 sum: 654) uniq_cov(1 cov: 63.16% cplx: 676.00)   artifacts(0) iter/s: 87
RPLC 1  3 high_cov_hits(1 sum: 2146) uniq_cov(1 cov: 63.16% cplx: 135.00)   artifacts(0) iter/s: 100
RPLC 1  3 high_cov_hits(1 sum: 2146) uniq_cov(1 cov: 63.16% cplx: 676.00)   artifacts(0) iter/s: 96
NEW     4 high_cov_hits(2 sum: 2150) uniq_cov(1 cov: 63.16% cplx: 135.00)   artifacts(0) iter/s: 89
NEW     4 high_cov_hits(1 sum: 2146) uniq_cov(2 cov: 84.21% cplx: 579.50)   artifacts(0) iter/s: 86
RPLC 1  4 high_cov_hits(1 sum: 2146) uniq_cov(1 cov: 63.16% cplx: 135.00)   artifacts(0) iter/s: 82
RPLC 1  6 high_cov_hits(2 sum: 2652) uniq_cov(2 cov: 84.21% cplx: 579.50)   artifacts(0) iter/s: 84
NEW     6 high_cov_hits(2 sum: 2150) uniq_cov(3 cov: 94.74% cplx: 1372.33)   artifacts(0) iter/s: 83
RPLC 1  6 high_cov_hits(2 sum: 2150) uniq_cov(2 cov: 84.21% cplx: 579.50)   artifacts(0) iter/s: 83
RPLC 1  6 high_cov_hits(2 sum: 2150) uniq_cov(2 cov: 84.21% cplx: 579.50)   artifacts(0) iter/s: 80
RPLC 1  9 high_cov_hits(2 sum: 2656) uniq_cov(3 cov: 94.74% cplx: 1372.33)   artifacts(0) iter/s: 103
RPLC 2  9 high_cov_hits(2 sum: 2652) uniq_cov(2 cov: 94.74% cplx: 547.50)   artifacts(0) iter/s: 102
RPLC 1  9 high_cov_hits(2 sum: 2652) uniq_cov(3 cov: 94.74% cplx: 1372.33)   artifacts(0) iter/s: 101
NEW     13 high_cov_hits(3 sum: 2748) uniq_cov(2 cov: 94.74% cplx: 547.50)   artifacts(0) iter/s: 119
RPLC 1  13 high_cov_hits(2 sum: 2656) uniq_cov(2 cov: 94.74% cplx: 547.50)   artifacts(0) iter/s: 116
NEW     15 high_cov_hits(4 sum: 2767) uniq_cov(2 cov: 94.74% cplx: 547.50)   artifacts(0) iter/s: 119
NEW     15 high_cov_hits(4 sum: 2767) uniq_cov(2 cov: 94.74% cplx: 547.50)   artifacts(1) iter/s: 115

The book is really helpful, thanks! And thank you for your guidance about the grammar-based mutator.

[loiclec]

amazing! thanks for reporting the bug :) let me know if you need any more help