testing: fuzzing in Envoy Mobile

[junr03]

This issue discusses avenues for leveraging fuzz testing in Envoy Mobile.

Potential Areas for Fuzzing:

1. Integration with the HCM: Envoy Mobile acts as a http codec interacting with the HCM -- sendXXX on the request path, and onXXX on the response path. Structure Aware fuzzing could help with detecting potential edge cases in API ordering, much like the Codec Fuzzer in upstream envoy. Some things to consider:
   • The fuzzer is not innately aware of illegal transitions, so initially there will be abundant false positives as illegal state transitions are found and pruned from the fuzzing path.
   • Corpus for input fuzzing could be kept in sync with the stable API call patterns done in the test/common/integration/dispatcher_integration_test.cc test cases.
   • BaseIntegrationTest (which is used by the dispatcher tests) is slow to fuzz, but google has a contractor working on performance.
   • Mocks are challenging in fuzzing because they might be lacking functionality that leads to false positives. They are also slow to initialize (static initialization might help with this).
2. Integration with Filter Manager: Envoy mobile integrates L7 filters using the Platform Bridge filter, which translates state between the platform specific API calls and upstream Envoy's Filter Manager. This surface is also conducive to structure aware fuzzing. We might be able to draw inspiration from the upstream Envoy filter fuzzer.
3. Utility Classes: Envoy uses static utility classes to provide translation of data from one language runtime to another. Self differential fuzzing could be used to ensure correct roundtrip translation. The Envoy Mobile team has not experienced many crashes (and only some memory leaks) in this space.

Project Ordering

1. Setup basic fuzzing infrastructure using a simple use case. For instance, fuzzing inputs to utility classes.
2. Setup structure aware fuzzing for the API surface that integrates with the HCM drawing on prior art from Envoy's Codec fuzzers.
3. Setup structure aware fuzzing between Platform L7 filters (written in C++) that use the Platform Bridge Filter.

Long term

1. Explore complete end-to-end fuzzing from swift/kotlin.

Additional Information

The following examples were generously provided by @asraa. Any mistakes are mine.

False positives issues (beware these, or be prepared to handle these sorts of issues):

• Part of the issue is that the codec fuzzer fuzzes BOTH H/1 and H/2. Examples:
  • H/1 requires connection close on stream reset so that was missing handling https://github.com/envoyproxy/envoy/pull/10643/files
  • H/1 doesn’t allow multiple streams https://github.com/envoyproxy/envoy/pull/10452
  • H/1 dispatches data after reset stream, another connection close/stream reset issue https://github.com/envoyproxy/envoy/pull/13014
• Parsers should already be rejecting some headers, caused issues in codec fuzzers https://github.com/envoyproxy/envoy/pull/10933

[junr03]

Thanks for all your time @asraa. I captured here my initial thoughts and all the notes (both yours and mine) from our offline meeting earlier today. Thanks again for your time, it was super helpful.

I will discuss internally with my team. But my initial thought on project ordering is as listed above.

Let me know what you think.

[goaway]

@junr03 I think targeting a simply initial use case is definitely the way to go. You mention inputs to utility classes as a possibility, and I'd propose we just pick one and go with that, both to start getting infrastructure in place and learn whatever else we don't know along the way.

Filter fuzzing also sounds compelling, and the PBF especially seems like a prime target for it.

We should hold off on dispatcher/client fuzzing work until https://github.com/envoyproxy/envoy-mobile/pull/1272 lands, but I actually think it will probably make it easier to fuzz this area once the split is in place, since dispatch won't necessarily need to be part of the mix.

[junr03]

We should hold off on dispatcher/client fuzzing work until #1272 lands, but I actually think it will probably make it easier to fuzz this area once the split is in place, since dispatch won't necessarily need to be part of the mix.

Agreed. I mostly used the historical name here just for ordering consistency. I totally agree that once the event concerns are out of the new Http::Client, it will be easier/cleaner to fuzz.

[asraa]

All sounds good to me!

BTW it looks like you are pulling in Envoy's bazel macros, so that will make things pretty easy! We pull in this dependency https://github.com/bazelbuild/rules_fuzzing. As long as you use envoy_cc_fuzz_macro, then the configs in .bazelrc added in this PR https://github.com/envoyproxy/envoy/pull/14834/files will enable a pretty seamless build for locally running fuzzers and building for OSS-Fuzz.