search

graphql-rust / juniper

GraphQL server library for Rust
Other
5.63k stars 417 forks source link

Fuzz testing with cargo fuzz #642

Open LegNeato opened 4 years ago

LegNeato commented 4 years ago

We should fuzz test execute and execute_async to make sure no external input can cause the library to panic or crash.

trevyn commented 3 years ago

I set up a basic cargo fuzz for execute_sync here: https://github.com/trevyn/juniper/compare/master...trevyn:fuzz

It got up to 1.8M executions (~20 minutes) and then segfaulted:

#1842766        REDUCE cov: 2716 ft: 10646 corp: 1777/747Kb lim: 3205 exec/s: 1360 rss: 830Mb L: 17/3073 MS: 1 EraseBytes-
────────────────────────────────────────────────────────────────────────────────

Error: Fuzz target exited with signal: 11

Apparently there is a known bug in cargo fuzz where it doesn't save the test case, and one way to find it is to use afl.rs🤪: https://github.com/rust-fuzz/cargo-fuzz/issues/235#issuecomment-659747105

At any rate, it seems like what's really getting exercised is parse_document_source here: https://docs.rs/juniper/0.15.3/src/juniper/lib.rs.html#231

@LegNeato is that as intended? Also, I'm not sure if the async-flavored execute function is exercising anything different, and I'd have to figure out how to scaffold that.

trevyn commented 3 years ago

Ok, here's the AFL version: https://github.com/trevyn/juniper/compare/master...trevyn:fuzz-afl

Ran it for 30 minutes with no issues, will leave it running and report back later.

trevyn commented 3 years ago

1 crash found: crash-0000.min

I added a (failing) test to the trevyn:fuzz-afl branch.

trevyn commented 3 years ago

In addition to the stack overflow crash above, it's also easy to hit a Type not found in schema panic. That file has a lot of expect()s that look easy to hit, though, so I'm not sure what the intended behavior is.

LegNeato commented 2 years ago

Woah, totally missed this, great work! Is it ready to be integrated?

LegNeato commented 2 years ago

@trevyn , FWIW I was thinking about this due to https://github.com/graphql-rust/juniper/commit/17d474ed21dfd42c8e2c59a27e9d38620b82bdef.

I wonder if there is a way to give a schema and have a bunch of valid and invalid permutations thrown at us. Would be useful for all GraphQL servers to have a corpus of graphql queries with corresponding schema and if they should pass / not pass.

trevyn commented 2 years ago

@LegNeato Thanks! I’m not working with Juniper or GraphQL anymore, but feel free to use anything here!