Maelstrom is a fast Rust, Go, and Python test runner that runs every test in its own container. Tests are either run locally or distributed to a clustered job runner.
We could parse out the test data for the fuzz tests and run each fuzz test as a separate job. This won't work when the fuzz test seeds are specified with F.Add.
We'll need to accept command-line-options or configuration values to set the minimum and maximum amount of time to spend on a fuzz test (or minimum and maximum number of iterations).
We need to figure out how instrumentation works. The go documentation says:
On supported platforms, 'go test' compiles the test executable with fuzzing coverage instrumentation. The fuzzing engine uses that instrumentation to find and cache inputs that expand coverage, increasing the likelihood of finding bugs.
We'll need a way to gather seeds for failure, report the failures they induced, and save the seeds.
At its most developed, one could imagine having the test coordinator run an arbitrary number of jobs, where each job did some amount of fuzzing based off of a seed. The test coordinator would look at the results it was getting back and create more jobs dynamically. Basically, wouldn't it be cool if you had a cluster of 1000 cores if you ask maelstrom to run a fuzz test for 10 seconds across the whole cluster, and get a meaningful amount of coverage?
We'd need to figure out how to run fuzz tests. This is a good example of the problem of having "open ended tests" of "test templates". When somebody just runs all of the tests, what do they want to happen with these templates? What values do they want? Anyway, that's a larger question. Maybe in the short term we'd just accept command-line options that would only run fuzz tests.
nfachan
commented
2 months ago
Some considerations include:
We could parse out the test data for the fuzz tests and run each fuzz test as a separate job. This won't work when the fuzz test seeds are specified with
F.Add.We'll need to accept command-line-options or configuration values to set the minimum and maximum amount of time to spend on a fuzz test (or minimum and maximum number of iterations).
We need to figure out how instrumentation works. The go documentation says:
We'll need a way to gather seeds for failure, report the failures they induced, and save the seeds.
At its most developed, one could imagine having the test coordinator run an arbitrary number of jobs, where each job did some amount of fuzzing based off of a seed. The test coordinator would look at the results it was getting back and create more jobs dynamically. Basically, wouldn't it be cool if you had a cluster of 1000 cores if you ask maelstrom to run a fuzz test for 10 seconds across the whole cluster, and get a meaningful amount of coverage?
We'd need to figure out how to run fuzz tests. This is a good example of the problem of having "open ended tests" of "test templates". When somebody just runs all of the tests, what do they want to happen with these templates? What values do they want? Anyway, that's a larger question. Maybe in the short term we'd just accept command-line options that would only run fuzz tests.
See: https://go.dev/doc/security/fuzz/ See: https://pkg.go.dev/testing#hdr-Fuzzing