Integrate with google/oss-fuzz for continuous fuzz testing?

[nathaniel-brough]

Hey I'd like to suggest adding mimalloc to google/oss-fuzz. If you aren't familiar with fuzz testing, here is a bit of a run down (from Wikipedia);

In programming and software development, fuzzing or fuzz testing is an automated software testing technique that involves providing invalid, unexpected, or random data as inputs to a computer program. The program is then monitored for exceptions such as crashes, failing built-in code assertions, or potential memory leaks.

This in principle is kind of similar to the stress-test harness that you have. But a little different in that the execution of testing is guided by code-coverage i.e. test inputs that cover more code will be favoured and slightly mutated for the next round of fuzzing.

Google offers a free continuous fuzzing service called OSS-fuzz. If mimalloc is integrated into oss-fuzz, the (yet to be written) fuzz tests under mimalloc will be built and then run once a day, to search for bugs and vulnerabilities in mimalloc. This service can be integrated with the CI for mimalloc, so that the fuzz tests are run for 10min or so for every pull request, preventing buggy code from being merged.

Here is an example of another microsoft project msquic that has been integrated into oss-fuzz https://github.com/google/oss-fuzz/blob/master/projects/msquic/project.yaml

I've opened up a draft pull request to add a super basic fuzz-testing harness here https://github.com/microsoft/mimalloc/pull/931. It needs a little more polish, but thought I'd guage interest before continuing.

If you are keen on adding mimalloc to oss-fuzz I'd be happy to champion the integration :)

[daanx]

Oh, having automated fuzzing would be great! I have been surprised how many edge cases are there (especially with alignment etc.). Since mimalloc has many explicitly checked invariants in the debug builds, fuzzing may be quite effective in finding assertion failures.

• Are there any restrictions, in particular with licensing and copyright that we should consider?
• Before we automate this, is there a way to first run things manually so we can fix any initial bugs?
• Fuzzing may not work well with an allocator as various scenario's only play out in the interaction between threads, or more heavy allocation patterns with many segments being used and freed -- the scale may well be beyond any fuzzer?

I am keen on having more testing but also a tad hesitant to rely on an external service. There should be an easy way to "turn it off" for example.

[nathaniel-brough]

Are there any restrictions, in particular with licensing and copyright that we should consider?

I'm not a lawyer but I don't think there is anything that stands out to me as out of the ordinary licensing. From a licensing perspective this is roughly how it would look;

• The fuzzing harnesses would live under this repo and be licensed under the same MIT license. This I imagine is covered by microsofts CLAs.
• There are three files that I would contribute to https://github.com/google/oss-fuzz repository. These files would be licensed to Google under the Apache License Version 2.0.
  • A Dockerfile for setting up the build environment and fetching source code.
  • A build.sh script for building mimalloc, in the oss-fuzz environment
  • A project.yaml that configures which fuzzing engines and sanitizers to use.

Before we automate this, is there a way to first run things manually so we can fix any initial bugs?

Yeah I've been running the fuzzer in #931 locally for a few hours, and while there seem to be a few quirks I haven't seen any crashes/bugs so far. I'll try adding in some aligned allocs and enable debug assertions and see what comes of it. To run the fuzzer I'm using the following;

$ mkdir build
$ cd build
$ export CC=clang && export CXX=clang++
$ export CFLAGS=-fsanitize=fuzzer-no-link,address
$ export LDFLAGS=-fsanitize=address
$ cmake -GNinja ..
$ ninja -j$(nproc)
$ mkdir corpus
$ ./fuzz-random-alloc -max_len=10000 corpus

You should expect the fuzzer to run indefinetel or;

• Until it finds a bug and crashes,
• You manually stop it i.e. CTRL-C
• You set a timeout on the command line.

Obviously this is a little verbose, and as mentioned the PR needs a bit more polish especially regarding the cmake configs and some documentation is probably also needed.

Fuzzing may not work well with an allocator as various scenario's only play out in the interaction between threads, or more heavy allocation patterns with many segments being used and freed -- the scale may well be beyond any fuzzer?

Fuzzing has been used fairly often in hardening allocators, in fact the fuzzer I wrote in #931 is largely inspired by the fuzz harnesses in tcmalloc.

One of the goals of fuzzing is to trigger a crash that is deterministic so that it can easily be replicated. While it is possible to do multithreading with a fuzzing harness it generally runs counter to the goal of reproducibility as threads are not typically deterministically scheduled, so it's often avoided. There is some research going into creating a thread-scheduler that is deterministic and integrates well with fuzzing but it is still in it's infancy.

The fuzzer in #931 will allocate/reallocate/free as many allocation slots as you choose. Although I've found the best coverage happens when there are around ~500-1000 slots used.

There is also an option with libfuzzer to limit the maximum ram available to the fuzzer, so I've found that varying that parameter can be used fairly consistently to induce resource contention. e.g.

$ ./fuzz-random-alloc -max_len=10000 -rss_limit_mb=512 corpus

I am keen on having more testing but also a tad hesitant to rely on an external service. There should be an easy way to "turn it off" for example.

Yeah it's reasonable to be sceptical, it can sometimes end up being more trouble than it's worth integrating with an external service. It's worth noting that oss-fuzz itself is actually just an instance of the open-source clusterfuzz project so if oss-fuzz was to be discontinued it would be fairly straightforward to shift things over to a self-hosted instance. Regarding your specific example, "turning it off" would entail adding a PR to the oss-fuzz project removing the configuration files for mimalloc.

Happy to answer any further queries that you might have.