Fuzz testing support

[jonesmz]

Creation of unit tests designed for use with fuzz testers, such as American Fuzzy Lop (http://lcamtuf.coredump.cx/afl/) will help catch bugs.

A good candidate where fuzz testing would be effective is for the unified shading language : http://docs.banshee3d.com/Native/bslfx.html

[jonesmz]

Another way that fuzz testing might be helpful is if a test harness was created that read in some kind of simplified instruction format to read and combine materials, meshes, textures, so on and so forth. The fuzz tester would generate instructions for this test harness, and additionally generate materials, meshes, textures, and so on in a psuedo-random (guided but random-ish) way. Tests that CRASH would be considered failures. Tests that detect things like invalid file formats, or impossible combinations, and gracefully exit, would be considered success.

Such a test harness would be testing a couple different things simultaneously.

• File format loaders
• Interactions between various object types like Materials, Textures, Meshes so on
• The memory model of objects living and dying based on the guided-randomization of the test harness instructions.

Since the engine uses std::shared_pointer, and handles, there's plenty of opportunity to let the guided randomization of fuzz testing do, basically, random things. The goal here would be to let the fuzz tester make the engine do absolutely ridiculous things, and render bizarre scenes, until it finds a combination that manages to make the engine crash, so that the crash can be fixed.

But this is the kind of testing that you let run in the background of a computer for weeks at a time. Not something to expect a result in the first 5 minutes.

[jonesmz]

Fuzz testers like this also greatly benefit from the compiler sanitizers, like the memory, address, and undefined behavior, sanitizers.

[BearishSun]

Looks interesting, I'll read up on it when I get a moment.